Skip to content

swh-lister: Merge lister configuration and services into one

Antoine R. Dumont requested to merge generated-differential-D1519-source into generated-differential-D1519-target
  • Instantiate only one swh-worker@lister service
  • Associate the lister's configuration for it to discuss with the right db (swh-lister on belvedere)
  • Drops the lister_<techno>.pp manifests
  • Removes each lister's configuration
  • Configure the rocquenquourt nodes to deploy swh-worker@lister

Note:

This does not do any cleanups. This will have to be done later as extra manual steps (after the puppet agent --test step).

Related #1772 (closed)

Test Plan

# worker01 runs on rocq which has the `lister` worker to deploy
$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to update_configuration worker01
Found host worker01.softwareheritage.org
Cloning into '/tmp/swh-ocd.6SyrsvvP/environments/production/data/private'...
done.
Cloning into '/tmp/swh-ocd.6SyrsvvP/environments/update_configuration/data/private'...
done.
*** Running octocatalog-diff on host worker01.softwareheritage.org
I, [2019-05-29T14:56:41.469951 #20238]  INFO -- : Catalogs compiled for worker01.softwareheritage.org
I, [2019-05-29T14:56:42.182587 #20238]  INFO -- : Diffs computed for worker01.softwareheritage.org
diff origin/production/worker01.softwareheritage.org current/worker01.softwareheritage.org
*******************************************
+ File[/etc/softwareheritage/lister.yml] =>
   parameters =>
      "ensure": "present"
      "group": "swhworker"
      "mode": "0644"
      "owner": "swhworker"
      "content": >>>
---
storage:
  cls: remote
  args:
    url: http://uffizi.internal.softwareheritage.org:5002/
scheduler:
  cls: remote
  args:
    url: http://saatchi.internal.softwareheritage.org:5008/
lister:
  cls: local
  args:
    db: postgresql://swh-lister:@belvedere.internal.softwareheritage.org:5434/swh-lister
celery:
  task_broker: amqp://swhconsumer:swh::deploy::worker::task_broker::password@rabbitmq:5672//
  task_modules:
  - swh.lister.debian.tasks
  - swh.lister.github.tasks
  - swh.lister.gitlab.tasks
  - swh.lister.npm.tasks
  - swh.lister.pypi.tasks
  task_queues:
  - swh.lister.debian.tasks.DebianListerTask
  - swh.lister.github.tasks.IncrementalGitHubLister
  - swh.lister.github.tasks.RangeGitHubLister
  - swh.lister.github.tasks.FullGitHubRelister
  - swh.lister.gitlab.tasks.IncrementalGitLabLister
  - swh.lister.gitlab.tasks.RangeGitLabLister
  - swh.lister.gitlab.tasks.FullGitLabRelister
  - swh.lister.npm.tasks.NpmListerTask
  - swh.lister.pypi.tasks.PyPIListerTask
credentials: ''
<<<
*******************************************
- File[/etc/softwareheritage/lister]
*******************************************
- File[/etc/softwareheritage/lister_debian.yml]
*******************************************
- File[/etc/softwareheritage/lister_github.yml]
*******************************************
- File[/etc/softwareheritage/lister_gitlab.yml]
*******************************************
- File[/etc/softwareheritage/lister_npm.yml]
*******************************************
- File[/etc/softwareheritage/lister_pypi.yml]
*******************************************
  File[/etc/softwareheritage/loader_debian.yml] =>
   parameters =>
     content =>
      @@ -5,5 +5,5 @@
           url: http://uffizi.internal.softwareheritage.org:5002/
       directory_packet_size: 100
      -lister_db_url: postgresql://lister-debian:swh::deploy::loader_debian::db::password@belvedere.internal.softwareheritage.org:5434/lister-debian
      +lister_db_url: postgresql://swh-lister:swh::deploy::loader_debian::db::password@belvedere.internal.softwareheritage.org:5434/swh-lister
       celery:
         task_broker: amqp://swhconsumer:swh::deploy::worker::task_broker::password@rabbitmq:5672//
*******************************************
+ File[/etc/systemd/system/swh-worker@lister.service.d/parameters.conf] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": "Class[Systemd::Systemctl::Daemon_reload]"
      "owner": "root"
      "content": >>>
# Managed by puppet - modifications will be overwritten
# In defined class profile::swh::deploy::worker::instance

[Service]
Environment=CONCURRENCY=5
Environment=MAX_TASKS_PER_CHILD=5
Environment=LOGLEVEL=warning
<<<
*******************************************
+ File[/etc/systemd/system/swh-worker@lister.service.d] =>
   parameters =>
      "ensure": "directory"
      "group": "root"
      "owner": "root"
*******************************************
- File[/etc/systemd/system/swh-worker@lister_debian.service.d/parameters.conf]
*******************************************
- File[/etc/systemd/system/swh-worker@lister_debian.service.d]
*******************************************
- File[/etc/systemd/system/swh-worker@lister_github.service.d/parameters.conf]
*******************************************
- File[/etc/systemd/system/swh-worker@lister_github.service.d]
*******************************************
- File[/etc/systemd/system/swh-worker@lister_gitlab.service.d/parameters.conf]
*******************************************
- File[/etc/systemd/system/swh-worker@lister_gitlab.service.d]
*******************************************
- File[/etc/systemd/system/swh-worker@lister_npm.service.d/parameters.conf]
*******************************************
- File[/etc/systemd/system/swh-worker@lister_npm.service.d]
*******************************************
- File[/etc/systemd/system/swh-worker@lister_pypi.service.d/parameters.conf]
*******************************************
- File[/etc/systemd/system/swh-worker@lister_pypi.service.d]
*******************************************
+ Profile::Swh::Deploy::Worker::Instance[lister] =>
   parameters =>
      "ensure": "present"
      "instance_name": "lister"
      "max_tasks_per_child": 5
*******************************************
- Profile::Swh::Deploy::Worker::Instance[lister_debian]
*******************************************
- Profile::Swh::Deploy::Worker::Instance[lister_github]
*******************************************
- Profile::Swh::Deploy::Worker::Instance[lister_gitlab]
*******************************************
- Profile::Swh::Deploy::Worker::Instance[lister_npm]
*******************************************
- Profile::Swh::Deploy::Worker::Instance[lister_pypi]
*******************************************
+ Service[swh-worker@lister] =>
   parameters =>
      "enable": true
*******************************************
- Service[swh-worker@lister_debian]
*******************************************
- Service[swh-worker@lister_github]
*******************************************
- Service[swh-worker@lister_gitlab]
*******************************************
- Service[swh-worker@lister_npm]
*******************************************
- Service[swh-worker@lister_pypi]
*******************************************
+ Systemd::Dropin_file[swh-worker@lister/parameters.conf] =>
   parameters =>
      "ensure": "present"
      "filename": "parameters.conf"
      "path": "/etc/systemd/system"
      "unit": "swh-worker@lister.service"
      "content": >>>
# Managed by puppet - modifications will be overwritten
# In defined class profile::swh::deploy::worker::instance

[Service]
Environment=CONCURRENCY=5
Environment=MAX_TASKS_PER_CHILD=5
Environment=LOGLEVEL=warning
<<<
*******************************************
- Systemd::Dropin_file[swh-worker@lister_debian/parameters.conf]
*******************************************
- Systemd::Dropin_file[swh-worker@lister_github/parameters.conf]
*******************************************
- Systemd::Dropin_file[swh-worker@lister_gitlab/parameters.conf]
*******************************************
- Systemd::Dropin_file[swh-worker@lister_npm/parameters.conf]
*******************************************
- Systemd::Dropin_file[swh-worker@lister_pypi/parameters.conf]
*******************************************
*** End octocatalog-diff on worker01.softwareheritage.org

Migrated from D1519 (view on Phabricator)

Merge request reports