Move indexer_journal_client to scheduler instances (!88) · Merge requests · Platform / Infrastructure / Puppet / puppet-swh-site

Antoine R. Dumont requested to merge generated-differential-D5059-source into generated-differential-D5059-target Feb 10, 2021
No need for those journal client to be on kafka nodes.
This is also an attempt to clarify where to deploy our backfill configuration.
In effect, this moves the service from:
prod: getty -> saatchi
staging: journal0 -> scheduler0
Deployment:
deactivate puppet on saatchi, scheduler0
stop old services on getty and journal0
(clean up)
puppet run on saatchi, scheduler0
Test Plan

octocatalog-diff:
$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging getty
...
I, [2021-02-10T17:09:55.240238 #14284]  INFO -- : Catalogs compiled for getty.internal.softwareheritage.org
I, [2021-02-10T17:09:56.576597 #14284]  INFO -- : Diffs computed for getty.internal.softwareheritage.org
diff origin/production/getty.internal.softwareheritage.org current/getty.internal.softwareheritage.org
*******************************************
- File[/etc/softwareheritage/indexer/journal_client.yml]
*******************************************
- File[/etc/softwareheritage/indexer]
*******************************************
- File[/etc/softwareheritage/journal]
*******************************************
- File[/etc/systemd/system/swh-indexer-journal-client.service]
*******************************************
- Package[python3-swh.indexer]
*******************************************
- Package[python3-swh.journal]
*******************************************
- Service[swh-indexer-journal-client]
*******************************************
- Systemd::Unit_file[swh-indexer-journal-client.service]
*******************************************
*** End octocatalog-diff on getty.internal.softwareheritage.org
$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging journal0.internal.staging.swh.network
...
*** Running octocatalog-diff on host journal0.internal.staging.swh.network
I, [2021-02-10T17:10:26.109041 #19467]  INFO -- : Catalogs compiled for journal0.internal.staging.swh.network
I, [2021-02-10T17:10:27.498780 #19467]  INFO -- : Diffs computed for journal0.internal.staging.swh.network
diff origin/production/journal0.internal.staging.swh.network current/journal0.internal.staging.swh.network
*******************************************
- File[/etc/softwareheritage/indexer/journal_client.yml]
*******************************************
- File[/etc/softwareheritage/indexer]
*******************************************
- File[/etc/softwareheritage/journal]
*******************************************
- File[/etc/systemd/system/swh-indexer-journal-client.service]
*******************************************
- Package[python3-swh.indexer]
*******************************************
- Package[python3-swh.journal]
*******************************************
- Service[swh-indexer-journal-client]
*******************************************
- Systemd::Unit_file[swh-indexer-journal-client.service]
*******************************************
*** End octocatalog-diff on journal0.internal.staging.swh.network
$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging scheduler0.internal.staging.swh.network
...
*** Running octocatalog-diff on host scheduler0.internal.staging.swh.network
I, [2021-02-10T17:10:57.460041 #24611]  INFO -- : Catalogs compiled for scheduler0.internal.staging.swh.network
I, [2021-02-10T17:10:59.194738 #24611]  INFO -- : Diffs computed for scheduler0.internal.staging.swh.network
diff origin/production/scheduler0.internal.staging.swh.network current/scheduler0.internal.staging.swh.network
*******************************************
+ File[/etc/softwareheritage/indexer/journal_client.yml] =>
   parameters =>
      "ensure": "present"
      "group": "swhdev"
      "mode": "0644"
      "notify": "Service[swh-indexer-journal-client]"
      "owner": "root"
      "content": >>>
---
journal:
  brokers:
  - journal0.internal.staging.swh.network
  group_id: swh.indexer.journal_client
  prefix: swh.journal.objects
scheduler:
  cls: remote
  args:
    url: http://scheduler0.internal.staging.swh.network:5008/
<<<
*******************************************
+ File[/etc/softwareheritage/indexer] =>
   parameters =>
      "ensure": "directory"
      "group": "swhworker"
      "mode": "0755"
      "owner": "swhworker"
*******************************************
+ File[/etc/systemd/system/swh-indexer-journal-client.service] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": "Class[Systemd::Systemctl::Daemon_reload]"
      "owner": "root"
      "show_diff": true
      "content": >>>
# Indexer Journal Client unit file
# Managed by puppet class profile::swh::deploy::indexer_journal_client
# Changes will be overwritten

[Unit]
Description=Software Heritage Indexer Journal Client
After=network.target

[Service]
User=swhstorage
Group=swhstorage
Type=simple
ExecStart=/usr/bin/swh indexer --config-file /etc/softwareheritage/indexer/journal_client.yml journal-client
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
<<<
*******************************************
+ Package[python3-swh.indexer] =>
   parameters =>
      "ensure": "present"
*******************************************
+ Service[swh-indexer-journal-client] =>
   parameters =>
      "enable": true
      "ensure": "running"
*******************************************
+ Systemd::Unit_file[swh-indexer-journal-client.service] =>
   parameters =>
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "notify": ["Service[swh-indexer-journal-client]"]
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "content": >>>
# Indexer Journal Client unit file
# Managed by puppet class profile::swh::deploy::indexer_journal_client
# Changes will be overwritten

[Unit]
Description=Software Heritage Indexer Journal Client
After=network.target

[Service]
User=swhstorage
Group=swhstorage
Type=simple
ExecStart=/usr/bin/swh indexer --config-file /etc/softwareheritage/indexer/journal_client.yml journal-client
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
<<<
*******************************************
*** End octocatalog-diff on scheduler0.internal.staging.swh.network
$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging saatchi
...
*** Running octocatalog-diff on host saatchi.internal.softwareheritage.org
I, [2021-02-10T17:15:05.562029 #30505]  INFO -- : Catalogs compiled for saatchi.internal.softwareheritage.org
I, [2021-02-10T17:15:07.186209 #30505]  INFO -- : Diffs computed for saatchi.internal.softwareheritage.org
diff origin/production/saatchi.internal.softwareheritage.org current/saatchi.internal.softwareheritage.org
*******************************************
+ File[/etc/softwareheritage/indexer/journal_client.yml] =>
   parameters =>
      "ensure": "present"
      "group": "swhdev"
      "mode": "0644"
      "notify": "Service[swh-indexer-journal-client]"
      "owner": "root"
      "content": >>>
---
journal:
  brokers:
  - kafka1.internal.softwareheritage.org
  - kafka2.internal.softwareheritage.org
  - kafka3.internal.softwareheritage.org
  - kafka4.internal.softwareheritage.org
  group_id: swh.indexer.journal_client
  prefix: swh.journal.objects
scheduler:
  cls: remote
  url: http://saatchi.internal.softwareheritage.org:5008/
<<<
*******************************************
+ File[/etc/softwareheritage/indexer] =>
   parameters =>
      "ensure": "directory"
      "group": "swhworker"
      "mode": "0755"
      "owner": "swhworker"
*******************************************
+ File[/etc/systemd/system/swh-indexer-journal-client.service] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": "Class[Systemd::Systemctl::Daemon_reload]"
      "owner": "root"
      "show_diff": true
      "content": >>>
# Indexer Journal Client unit file
# Managed by puppet class profile::swh::deploy::indexer_journal_client
# Changes will be overwritten

[Unit]
Description=Software Heritage Indexer Journal Client
After=network.target

[Service]
User=swhstorage
Group=swhstorage
Type=simple
ExecStart=/usr/bin/swh indexer --config-file /etc/softwareheritage/indexer/journal_client.yml journal-client
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
<<<
*******************************************
+ Package[python3-swh.indexer] =>
   parameters =>
      "ensure": "present"
*******************************************
+ Service[swh-indexer-journal-client] =>
   parameters =>
      "enable": true
      "ensure": "running"
*******************************************
+ Systemd::Unit_file[swh-indexer-journal-client.service] =>
   parameters =>
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "notify": ["Service[swh-indexer-journal-client]"]
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "content": >>>
# Indexer Journal Client unit file
# Managed by puppet class profile::swh::deploy::indexer_journal_client
# Changes will be overwritten

[Unit]
Description=Software Heritage Indexer Journal Client
After=network.target

[Service]
User=swhstorage
Group=swhstorage
Type=simple
ExecStart=/usr/bin/swh indexer --config-file /etc/softwareheritage/indexer/journal_client.yml journal-client
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
<<<
*******************************************
*** End octocatalog-diff on saatchi.internal.softwareheritage.org
Migrated from D5059 (view on Phabricator)
Move indexer_journal_client to scheduler instances

Test Plan

Merge request reports
