Declare new service worker to consume save code now queues (!343) · Merge requests · Platform / Infrastructure / Puppet / puppet-swh-site

Antoine R. Dumont requested to merge generated-differential-D5486-source into generated-differential-D5486-target Apr 12, 2021
The scheduler runner will route the save code now queries to those queues. So we need to define a new worker with loaders subscribed to those new queues.
Related to T3084
Test Plan

octocatalog on workers (staging, prod):
$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging worker0.internal.staging.swh.network
...
*** Running octocatalog-diff on host worker0.internal.staging.swh.network
I, [2021-04-12T16:58:04.687667 #11853]  INFO -- : Catalogs compiled for worker0.internal.staging.swh.network
I, [2021-04-12T16:58:05.891259 #11853]  INFO -- : Diffs computed for worker0.internal.staging.swh.network
diff origin/production/worker0.internal.staging.swh.network current/worker0.internal.staging.swh.network
*******************************************
+ Concat_fragment[profile::cron::swh-worker-loader_high_priority-autorestart] =>
   parameters =>
      "order": "10"
      "tag": "profile::cron::swh-worker"
      "target": "profile::cron::swh-worker"
      "content": >>>
# Cron snippet swh-worker-loader_high_priority-autorestart
13-58/15 * * * * root chronic /usr/local/sbin/swh-worker-ping-restart loader_high_priority@worker0.internal.staging.swh.network loader_high_priority
<<<
*******************************************
+ File[/etc/softwareheritage/loader_high_priority.yml] =>
   parameters =>
      "ensure": "present"
      "group": "swhworker"
      "mode": "0644"
      "owner": "swhworker"
      "content": >>>
---
storage:
  cls: pipeline
  steps:
  - cls: buffer
    min_batch_size:
      content: 1000
      content_bytes: 52428800
      directory: 1000
      revision: 1000
      release: 1000
  - cls: filter
  - cls: retry
  - cls: remote
    args:
      url: http://storage1.internal.staging.swh.network:5002/
max_content_size: 104857600
celery:
  task_broker: amqp://swhconsumer:swh-deploy-worker-task_broker-password@scheduler0.internal.staging.swh.network:5672/%2f
  task_queues:
  - swh.loader.git.tasks.UpdateGitRepository
  - swh.loader.git.tasks.LoadDiskGitRepository
  - swh.loader.git.tasks.UncompressAndLoadDiskGitRepository
  - swh.loader.mercurial.tasks.LoadMercurial
  - swh.loader.mercurial.tasks.LoadArchiveMercurial
  - swh.loader.svn.tasks.LoadSvnRepository
  - swh.loader.svn.tasks.MountAndLoadSvnRepository
  - swh.loader.svn.tasks.DumpMountAndLoadSvnRepository
<<<
*******************************************
+ File[/etc/systemd/system/swh-worker@loader_high_priority.service.d/parameters.conf] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": ["Class[Systemd::Systemctl::Daemon_reload]"]
      "owner": "root"
      "show_diff": true
      "content": >>>
# Managed by puppet - modifications will be overwritten
# In defined class profile::swh::deploy::worker::instance

[Service]
Environment=CONCURRENCY=1
Environment=MAX_TASKS_PER_CHILD=100
Environment=LOGLEVEL=info
<<<
*******************************************
+ File[/etc/systemd/system/swh-worker@loader_high_priority.service.d] =>
   parameters =>
      "ensure": "directory"
      "group": "root"
      "owner": "root"
      "purge": true
      "recurse": true
*******************************************
+ Profile::Cron::D[swh-worker-loader_high_priority-autorestart] =>
   parameters =>
      "command": "chronic /usr/local/sbin/swh-worker-ping-restart loader_high_priority@worker0.internal.staging.swh.network loader_high_priority"
      "minute": "fqdn_rand/15"
      "target": "swh-worker"
      "unique_tag": "swh-worker-loader_high_priority-autorestart"
      "user": "root"
*******************************************
+ Profile::Swh::Deploy::Worker::Instance[loader_high_priority] =>
   parameters =>
      "ensure": "present"
      "instance_name": "loader_high_priority"
      "merge_policy": "deep"
      "sentry_name": "loader_high_priority"
*******************************************
+ Service[swh-worker@loader_high_priority] =>
   parameters =>
      "enable": true
*******************************************
+ Systemd::Dropin_file[swh-worker@loader_high_priority/parameters.conf] =>
   parameters =>
      "daemon_reload": "lazy"
      "ensure": "present"
      "filename": "parameters.conf"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "unit": "swh-worker@loader_high_priority.service"
      "content": >>>
# Managed by puppet - modifications will be overwritten
# In defined class profile::swh::deploy::worker::instance

[Service]
Environment=CONCURRENCY=1
Environment=MAX_TASKS_PER_CHILD=100
Environment=LOGLEVEL=info
<<<
*******************************************
*** End octocatalog-diff on worker0.internal.staging.swh.network
$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging worker01
...
*** Running octocatalog-diff on host worker01.softwareheritage.org
I, [2021-04-12T16:58:48.403207 #17232]  INFO -- : Catalogs compiled for worker01.softwareheritage.org
I, [2021-04-12T16:58:49.538970 #17232]  INFO -- : Diffs computed for worker01.softwareheritage.org
diff origin/production/worker01.softwareheritage.org current/worker01.softwareheritage.org
*******************************************
+ Concat_fragment[profile::cron::swh-worker-loader_high_priority-autorestart] =>
   parameters =>
      "order": "10"
      "tag": "profile::cron::swh-worker"
      "target": "profile::cron::swh-worker"
      "content": >>>
# Cron snippet swh-worker-loader_high_priority-autorestart
6-51/15 * * * * root chronic /usr/local/sbin/swh-worker-ping-restart loader_high_priority@worker01.internal.softwareheritage.org loader_high_priority
<<<
*******************************************
+ File[/etc/softwareheritage/loader_high_priority.yml] =>
   parameters =>
      "ensure": "present"
      "group": "swhworker"
      "mode": "0644"
      "owner": "swhworker"
      "content": >>>
---
storage:
  cls: pipeline
  steps:
  - cls: buffer
    min_batch_size:
      content: 1000
      content_bytes: 52428800
      directory: 1000
      revision: 1000
      release: 1000
  - cls: filter
  - cls: retry
  - cls: remote
    args:
      url: http://saam.internal.softwareheritage.org:5002/
max_content_size: 104857600
celery:
  task_broker: amqp://swhconsumer:swh-deploy-worker-task_broker-password@rabbitmq:5672/%2f
  task_queues:
  - swh.loader.git.tasks.UpdateGitRepository
  - swh.loader.git.tasks.LoadDiskGitRepository
  - swh.loader.git.tasks.UncompressAndLoadDiskGitRepository
  - swh.loader.mercurial.tasks.LoadMercurial
  - swh.loader.mercurial.tasks.LoadArchiveMercurial
  - swh.loader.svn.tasks.LoadSvnRepository
  - swh.loader.svn.tasks.MountAndLoadSvnRepository
  - swh.loader.svn.tasks.DumpMountAndLoadSvnRepository
<<<
*******************************************
+ File[/etc/systemd/system/swh-worker@loader_high_priority.service.d/parameters.conf] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": ["Class[Systemd::Systemctl::Daemon_reload]"]
      "owner": "root"
      "show_diff": true
      "content": >>>
# Managed by puppet - modifications will be overwritten
# In defined class profile::swh::deploy::worker::instance

[Service]
Environment=CONCURRENCY=1
Environment=MAX_TASKS_PER_CHILD=100
Environment=LOGLEVEL=info
<<<
*******************************************
+ File[/etc/systemd/system/swh-worker@loader_high_priority.service.d] =>
   parameters =>
      "ensure": "directory"
      "group": "root"
      "owner": "root"
      "purge": true
      "recurse": true
*******************************************
+ Profile::Cron::D[swh-worker-loader_high_priority-autorestart] =>
   parameters =>
      "command": "chronic /usr/local/sbin/swh-worker-ping-restart loader_high_priority@worker01.internal.softwareheritage.org loader_high_priority"
      "minute": "fqdn_rand/15"
      "target": "swh-worker"
      "unique_tag": "swh-worker-loader_high_priority-autorestart"
      "user": "root"
*******************************************
+ Profile::Swh::Deploy::Worker::Instance[loader_high_priority] =>
   parameters =>
      "ensure": "present"
      "instance_name": "loader_high_priority"
      "merge_policy": "deep"
      "sentry_name": "loader_high_priority"
*******************************************
+ Service[swh-worker@loader_high_priority] =>
   parameters =>
      "enable": true
*******************************************
+ Systemd::Dropin_file[swh-worker@loader_high_priority/parameters.conf] =>
   parameters =>
      "daemon_reload": "lazy"
      "ensure": "present"
      "filename": "parameters.conf"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "unit": "swh-worker@loader_high_priority.service"
      "content": >>>
# Managed by puppet - modifications will be overwritten
# In defined class profile::swh::deploy::worker::instance

[Service]
Environment=CONCURRENCY=1
Environment=MAX_TASKS_PER_CHILD=100
Environment=LOGLEVEL=info
<<<
*******************************************
*** End octocatalog-diff on worker01.softwareheritage.org
No change when diffing this on azurfe workers (indexer)
Migrated from D5486 (view on Phabricator)
Declare new service worker to consume save code now queues

Test Plan

Merge request reports
