Skip to content

Declare new service worker to consume save code now queues

Antoine R. Dumont requested to merge generated-differential-D5486-source into generated-differential-D5486-target

The scheduler runner will route the save code now queries to those queues. So we need to define a new worker with loaders subscribed to those new queues.

Related to T3084

Test Plan

octocatalog on workers (staging, prod):

$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging worker0.internal.staging.swh.network
...
*** Running octocatalog-diff on host worker0.internal.staging.swh.network
I, [2021-04-12T16:58:04.687667 #11853]  INFO -- : Catalogs compiled for worker0.internal.staging.swh.network
I, [2021-04-12T16:58:05.891259 #11853]  INFO -- : Diffs computed for worker0.internal.staging.swh.network
diff origin/production/worker0.internal.staging.swh.network current/worker0.internal.staging.swh.network
*******************************************
+ Concat_fragment[profile::cron::swh-worker-loader_high_priority-autorestart] =>
   parameters =>
      "order": "10"
      "tag": "profile::cron::swh-worker"
      "target": "profile::cron::swh-worker"
      "content": >>>
# Cron snippet swh-worker-loader_high_priority-autorestart
13-58/15 * * * * root chronic /usr/local/sbin/swh-worker-ping-restart loader_high_priority@worker0.internal.staging.swh.network loader_high_priority
<<<
*******************************************
+ File[/etc/softwareheritage/loader_high_priority.yml] =>
   parameters =>
      "ensure": "present"
      "group": "swhworker"
      "mode": "0644"
      "owner": "swhworker"
      "content": >>>
---
storage:
  cls: pipeline
  steps:
  - cls: buffer
    min_batch_size:
      content: 1000
      content_bytes: 52428800
      directory: 1000
      revision: 1000
      release: 1000
  - cls: filter
  - cls: retry
  - cls: remote
    args:
      url: http://storage1.internal.staging.swh.network:5002/
max_content_size: 104857600
celery:
  task_broker: amqp://swhconsumer:swh-deploy-worker-task_broker-password@scheduler0.internal.staging.swh.network:5672/%2f
  task_queues:
  - swh.loader.git.tasks.UpdateGitRepository
  - swh.loader.git.tasks.LoadDiskGitRepository
  - swh.loader.git.tasks.UncompressAndLoadDiskGitRepository
  - swh.loader.mercurial.tasks.LoadMercurial
  - swh.loader.mercurial.tasks.LoadArchiveMercurial
  - swh.loader.svn.tasks.LoadSvnRepository
  - swh.loader.svn.tasks.MountAndLoadSvnRepository
  - swh.loader.svn.tasks.DumpMountAndLoadSvnRepository
<<<
*******************************************
+ File[/etc/systemd/system/swh-worker@loader_high_priority.service.d/parameters.conf] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": ["Class[Systemd::Systemctl::Daemon_reload]"]
      "owner": "root"
      "show_diff": true
      "content": >>>
# Managed by puppet - modifications will be overwritten
# In defined class profile::swh::deploy::worker::instance

[Service]
Environment=CONCURRENCY=1
Environment=MAX_TASKS_PER_CHILD=100
Environment=LOGLEVEL=info
<<<
*******************************************
+ File[/etc/systemd/system/swh-worker@loader_high_priority.service.d] =>
   parameters =>
      "ensure": "directory"
      "group": "root"
      "owner": "root"
      "purge": true
      "recurse": true
*******************************************
+ Profile::Cron::D[swh-worker-loader_high_priority-autorestart] =>
   parameters =>
      "command": "chronic /usr/local/sbin/swh-worker-ping-restart loader_high_priority@worker0.internal.staging.swh.network loader_high_priority"
      "minute": "fqdn_rand/15"
      "target": "swh-worker"
      "unique_tag": "swh-worker-loader_high_priority-autorestart"
      "user": "root"
*******************************************
+ Profile::Swh::Deploy::Worker::Instance[loader_high_priority] =>
   parameters =>
      "ensure": "present"
      "instance_name": "loader_high_priority"
      "merge_policy": "deep"
      "sentry_name": "loader_high_priority"
*******************************************
+ Service[swh-worker@loader_high_priority] =>
   parameters =>
      "enable": true
*******************************************
+ Systemd::Dropin_file[swh-worker@loader_high_priority/parameters.conf] =>
   parameters =>
      "daemon_reload": "lazy"
      "ensure": "present"
      "filename": "parameters.conf"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "unit": "swh-worker@loader_high_priority.service"
      "content": >>>
# Managed by puppet - modifications will be overwritten
# In defined class profile::swh::deploy::worker::instance

[Service]
Environment=CONCURRENCY=1
Environment=MAX_TASKS_PER_CHILD=100
Environment=LOGLEVEL=info
<<<
*******************************************
*** End octocatalog-diff on worker0.internal.staging.swh.network
$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging worker01
...
*** Running octocatalog-diff on host worker01.softwareheritage.org
I, [2021-04-12T16:58:48.403207 #17232]  INFO -- : Catalogs compiled for worker01.softwareheritage.org
I, [2021-04-12T16:58:49.538970 #17232]  INFO -- : Diffs computed for worker01.softwareheritage.org
diff origin/production/worker01.softwareheritage.org current/worker01.softwareheritage.org
*******************************************
+ Concat_fragment[profile::cron::swh-worker-loader_high_priority-autorestart] =>
   parameters =>
      "order": "10"
      "tag": "profile::cron::swh-worker"
      "target": "profile::cron::swh-worker"
      "content": >>>
# Cron snippet swh-worker-loader_high_priority-autorestart
6-51/15 * * * * root chronic /usr/local/sbin/swh-worker-ping-restart loader_high_priority@worker01.internal.softwareheritage.org loader_high_priority
<<<
*******************************************
+ File[/etc/softwareheritage/loader_high_priority.yml] =>
   parameters =>
      "ensure": "present"
      "group": "swhworker"
      "mode": "0644"
      "owner": "swhworker"
      "content": >>>
---
storage:
  cls: pipeline
  steps:
  - cls: buffer
    min_batch_size:
      content: 1000
      content_bytes: 52428800
      directory: 1000
      revision: 1000
      release: 1000
  - cls: filter
  - cls: retry
  - cls: remote
    args:
      url: http://saam.internal.softwareheritage.org:5002/
max_content_size: 104857600
celery:
  task_broker: amqp://swhconsumer:swh-deploy-worker-task_broker-password@rabbitmq:5672/%2f
  task_queues:
  - swh.loader.git.tasks.UpdateGitRepository
  - swh.loader.git.tasks.LoadDiskGitRepository
  - swh.loader.git.tasks.UncompressAndLoadDiskGitRepository
  - swh.loader.mercurial.tasks.LoadMercurial
  - swh.loader.mercurial.tasks.LoadArchiveMercurial
  - swh.loader.svn.tasks.LoadSvnRepository
  - swh.loader.svn.tasks.MountAndLoadSvnRepository
  - swh.loader.svn.tasks.DumpMountAndLoadSvnRepository
<<<
*******************************************
+ File[/etc/systemd/system/swh-worker@loader_high_priority.service.d/parameters.conf] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": ["Class[Systemd::Systemctl::Daemon_reload]"]
      "owner": "root"
      "show_diff": true
      "content": >>>
# Managed by puppet - modifications will be overwritten
# In defined class profile::swh::deploy::worker::instance

[Service]
Environment=CONCURRENCY=1
Environment=MAX_TASKS_PER_CHILD=100
Environment=LOGLEVEL=info
<<<
*******************************************
+ File[/etc/systemd/system/swh-worker@loader_high_priority.service.d] =>
   parameters =>
      "ensure": "directory"
      "group": "root"
      "owner": "root"
      "purge": true
      "recurse": true
*******************************************
+ Profile::Cron::D[swh-worker-loader_high_priority-autorestart] =>
   parameters =>
      "command": "chronic /usr/local/sbin/swh-worker-ping-restart loader_high_priority@worker01.internal.softwareheritage.org loader_high_priority"
      "minute": "fqdn_rand/15"
      "target": "swh-worker"
      "unique_tag": "swh-worker-loader_high_priority-autorestart"
      "user": "root"
*******************************************
+ Profile::Swh::Deploy::Worker::Instance[loader_high_priority] =>
   parameters =>
      "ensure": "present"
      "instance_name": "loader_high_priority"
      "merge_policy": "deep"
      "sentry_name": "loader_high_priority"
*******************************************
+ Service[swh-worker@loader_high_priority] =>
   parameters =>
      "enable": true
*******************************************
+ Systemd::Dropin_file[swh-worker@loader_high_priority/parameters.conf] =>
   parameters =>
      "daemon_reload": "lazy"
      "ensure": "present"
      "filename": "parameters.conf"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "unit": "swh-worker@loader_high_priority.service"
      "content": >>>
# Managed by puppet - modifications will be overwritten
# In defined class profile::swh::deploy::worker::instance

[Service]
Environment=CONCURRENCY=1
Environment=MAX_TASKS_PER_CHILD=100
Environment=LOGLEVEL=info
<<<
*******************************************
*** End octocatalog-diff on worker01.softwareheritage.org

No change when diffing this on azurfe workers (indexer)

Migrated from D5486 (view on Phabricator)

Merge request reports