Skip to content

worker-large: Make worker consume standard git queue

Antoine R. Dumont requested to merge generated-differential-D6873-source into generated-differential-D6873-target

Related to T3825

Test Plan

currently applied in workers to check the load is ok, it is [1]

  • [1]

octo-diff:

$ $SWH_PUPPET_ENVIRONMENT_HOME/bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging worker18
Found host worker18.softwareheritage.org
...
I, [2022-01-03T12:37:48.377541 #177099]  INFO -- : Catalogs compiled for worker18.softwareheritage.org
I, [2022-01-03T12:37:48.659271 #177099]  INFO -- : Diffs computed for worker18.softwareheritage.org
diff origin/production/worker18.softwareheritage.org current/worker18.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/loader_git.yml] =>
   parameters =>
     content =>
      @@ -26,3 +26,4 @@
         task_queues:
         - oneshot3:swh.loader.git.tasks.UpdateGitRepository
      +  - swh.loader.git.tasks.UpdateGitRepository
      _
*******************************************
  File[/etc/systemd/system/swh-worker@loader_git.service.d/parameters.conf] =>
   parameters =>
     content =>
      @@ -3,5 +3,5 @@
      _
       [Service]
      -Environment=CONCURRENCY=7
      +Environment=CONCURRENCY=10
       Environment=MAX_TASKS_PER_CHILD=1
       Environment=LOGLEVEL=info
*******************************************
  File[/etc/systemd/system/swh-worker@loader_oneshot.service.d/parameters.conf] =>
   parameters =>
     content =>
      @@ -3,5 +3,5 @@
      _
       [Service]
      -Environment=CONCURRENCY=7
      +Environment=CONCURRENCY=5
       Environment=MAX_TASKS_PER_CHILD=1
       Environment=LOGLEVEL=info
*******************************************
  Systemd::Dropin_file[swh-worker@loader_git/parameters.conf] =>
   parameters =>
     content =>
      @@ -3,5 +3,5 @@
      _
       [Service]
      -Environment=CONCURRENCY=7
      +Environment=CONCURRENCY=10
       Environment=MAX_TASKS_PER_CHILD=1
       Environment=LOGLEVEL=info
*******************************************
  Systemd::Dropin_file[swh-worker@loader_oneshot/parameters.conf] =>
   parameters =>
     content =>
      @@ -3,5 +3,5 @@
      _
       [Service]
      -Environment=CONCURRENCY=7
      +Environment=CONCURRENCY=5
       Environment=MAX_TASKS_PER_CHILD=1
       Environment=LOGLEVEL=info
*******************************************
*** End octocatalog-diff on worker18.softwareheritage.org

$ $SWH_PUPPET_ENVIRONMENT_HOME/bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging worker17
...
I, [2022-01-03T12:37:59.397696 #178766]  INFO -- : Catalogs compiled for worker17.softwareheritage.org
I, [2022-01-03T12:37:59.678886 #178766]  INFO -- : Diffs computed for worker17.softwareheritage.org
diff origin/production/worker17.softwareheritage.org current/worker17.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/loader_git.yml] =>
   parameters =>
     content =>
      @@ -26,3 +26,4 @@
         task_queues:
         - oneshot3:swh.loader.git.tasks.UpdateGitRepository
      +  - swh.loader.git.tasks.UpdateGitRepository
      _
*******************************************
  File[/etc/systemd/system/swh-worker@loader_git.service.d/parameters.conf] =>
   parameters =>
     content =>
      @@ -3,5 +3,5 @@
      _
       [Service]
      -Environment=CONCURRENCY=10
      +Environment=CONCURRENCY=20
       Environment=MAX_TASKS_PER_CHILD=1
       Environment=LOGLEVEL=info
*******************************************
  File[/etc/systemd/system/swh-worker@loader_oneshot.service.d/parameters.conf] =>
   parameters =>
     content =>
      @@ -3,5 +3,5 @@
      _
       [Service]
      -Environment=CONCURRENCY=20
      +Environment=CONCURRENCY=10
       Environment=MAX_TASKS_PER_CHILD=1
       Environment=LOGLEVEL=info
*******************************************
  Systemd::Dropin_file[swh-worker@loader_git/parameters.conf] =>
   parameters =>
     content =>
      @@ -3,5 +3,5 @@
      _
       [Service]
      -Environment=CONCURRENCY=10
      +Environment=CONCURRENCY=20
       Environment=MAX_TASKS_PER_CHILD=1
       Environment=LOGLEVEL=info
*******************************************
  Systemd::Dropin_file[swh-worker@loader_oneshot/parameters.conf] =>
   parameters =>
     content =>
      @@ -3,5 +3,5 @@
      _
       [Service]
      -Environment=CONCURRENCY=20
      +Environment=CONCURRENCY=10
       Environment=MAX_TASKS_PER_CHILD=1
       Environment=LOGLEVEL=info
*******************************************
*** End octocatalog-diff on worker17.softwareheritage.org

Migrated from D6873 (view on Phabricator)

Merge request reports