Skip to content

Deploy new origin intrinsic metadata journal client indexer

Antoine R. Dumont requested to merge generated-differential-D7928-source into generated-differential-D7928-target

Plan:

  • Push this commit only in staging for now (when accepted)
  • scheduler0.staging: Stop current journal client service
  • Run puppet on scheduler0.staging to redefine the journal client service (as this here)
  • Run this on staging workers to stop the old worker service (and declare the swh-journal-client service on workers)

This also adapts the instance.pp profile to clean up missing parts.

Related to T4282

Test Plan

octo-diff:

  • workers: purge old swh-worker@origin_intrinsic_metadata service and configuration files + install new indexer journal client service [1]

  • scheduler0.staging: Drop old indexer journal client service [2]

  • [1]

$SWH_PUPPET_ENVIRONMENT_HOME/bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging worker3.internal.staging.swh.network
Found host worker3.internal.staging.swh.network
WARN     -> Environment "staging-add-prometheus-metrics" contained non-word characters, correcting name to staging_add_prometheus_metrics
WARN     -> Environment "staging-bullseye-rabbitmq-plugin" contained non-word characters, correcting name to staging_bullseye_rabbitmq_plugin
WARN     -> Environment "staging-check-journal-client" contained non-word characters, correcting name to staging_check_journal_client
WARN     -> Environment "staging-check-journal-client-2nd-implementation" contained non-word characters, correcting name to staging_check_journal_client_2nd_implementation
WARN     -> Environment "staging-check-journal-client-first-implem" contained non-word characters, correcting name to staging_check_journal_client_first_implem
WARN     -> Environment "staging-pin" contained non-word characters, correcting name to staging_pin
Cloning into '/tmp/swh-ocd.EnSguLjO/environments/production/data/private'...
done.
Cloning into '/tmp/swh-ocd.EnSguLjO/environments/staging/data/private'...
done.
*** Running octocatalog-diff on host worker3.internal.staging.swh.network
I, [2022-06-01T11:15:26.983158 #372981]  INFO -- : Catalogs compiled for worker3.internal.staging.swh.network
I, [2022-06-01T11:15:27.429249 #372981]  INFO -- : Diffs computed for worker3.internal.staging.swh.network
diff origin/production/worker3.internal.staging.swh.network current/worker3.internal.staging.swh.network
*******************************************
- Concat_fragment[profile::cron::swh-worker-indexer_origin_intrinsic_metadata-autorestart]
*******************************************
+ File[/etc/softwareheritage/indexer/journal_client.yml] =>
   parameters =>
      "ensure": "present"
      "group": "swhstorage"
      "mode": "0640"
      "notify": "Service[swh-indexer-journal-client]"
      "owner": "root"
      "content": >>>
---
journal:
  brokers:
  - journal1.internal.staging.swh.network
  group_id: swh.indexer.journal_client
  prefix: swh.journal.objects
scheduler:
  cls: remote
  args:
    url: http://scheduler0.internal.staging.swh.network:5008/
<<<
*******************************************
  File[/etc/softwareheritage/indexer_origin_intrinsic_metadata.yml] =>
   parameters =>
     ensure =>
      - present
      + absent
*******************************************
+ File[/etc/softwareheritage/journal] =>
   parameters =>
      "ensure": "directory"
      "group": "swhworker"
      "mode": "0644"
      "owner": "swhworker"
*******************************************
+ File[/etc/systemd/system/swh-indexer-journal-client.service] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": "Class[Systemd::Systemctl::Daemon_reload]"
      "owner": "root"
      "show_diff": true
      "content": >>>
# Indexer Journal Client unit file
# Managed by puppet class profile::swh::deploy::indexer_journal_client
# Changes will be overwritten

[Unit]
Description=Software Heritage Indexer Journal Client
After=network.target

[Service]
Environment=SWH_SENTRY_DSN=https://swh::deploy::indexer::sentry_token@sentry.softwareheritage.org/5
Environment=SWH_SENTRY_ENVIRONMENT=staging
Environment=SWH_MAIN_PACKAGE=swh.indexer
User=swhstorage
Group=swhstorage
Type=simple
ExecStart=/usr/bin/swh indexer --config-file /etc/softwareheritage/indexer/journal_client.yml journal-client indexer origin-intrinsic-metadata
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
<<<
*******************************************
  File[/etc/systemd/system/swh-worker@indexer_origin_intrinsic_metadata.service.d/parameters.conf] =>
   parameters =>
     ensure =>
      - file
      + absent
*******************************************
- File[/etc/systemd/system/swh-worker@indexer_origin_intrinsic_metadata.service.d]
*******************************************
  Package[python3-swh.indexer] =>
   parameters =>
     notify =>
      - ["Profile::Swh::Deploy::Worker::Instance[indexer_content_mimetype]", "Profile::Swh::Deploy::Worker::Instance[indexer_fossology_license]", "Profile::Swh::Deploy::Worker::Instance[indexer_origin_intrinsic_metadata]"]
      + ["Profile::Swh::Deploy::Worker::Instance[indexer_content_mimetype]", "Profile::Swh::Deploy::Worker::Instance[indexer_fossology_license]"]
*******************************************
+ Package[python3-swh.journal] =>
   parameters =>
      "ensure": "installed"
*******************************************
- Profile::Cron::D[swh-worker-indexer_origin_intrinsic_metadata-autorestart]
*******************************************
  Profile::Swh::Deploy::Worker::Instance[indexer_origin_intrinsic_metadata] =>
   parameters =>
     ensure =>
      - present
      + absent
     send_task_events =>
      - true
      + false
     sentry_name =>
      - indexer
      + indexer_origin_intrinsic_metadata
*******************************************
+ Service[swh-indexer-journal-client] =>
   parameters =>
      "enable": true
      "ensure": "running"
*******************************************
  Service[swh-worker@indexer_origin_intrinsic_metadata] =>
   parameters =>
     enable =>
      - true
     ensure =>
      + absent
*******************************************
  Systemd::Dropin_file[swh-worker@indexer_origin_intrinsic_metadata/parameters.conf] =>
   parameters =>
     content =>
      - # Managed by puppet - modifications will be overwritten
# In defined class profile::swh::deploy::worker::instance

[Service]
Environment=CONCURRENCY=1
Environment=MAX_TASKS_PER_CHILD=5
Environment=LOGLEVEL=info
Environment=SWH_SENTRY_DSN=https://swh::deploy::indexer::sentry_token@sentry.softwareheritage.org/5
Environment=SWH_SENTRY_ENVIRONMENT=staging
Environment=SWH_MAIN_PACKAGE=swh.indexer


Environment=SWH_WORKER_CLI_EXTRA_ARGS=--events

     ensure =>
      - present
      + absent
*******************************************
+ Systemd::Unit_file[swh-indexer-journal-client.service] =>
   parameters =>
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "notify": ["Service[swh-indexer-journal-client]"]
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "content": >>>
# Indexer Journal Client unit file
# Managed by puppet class profile::swh::deploy::indexer_journal_client
# Changes will be overwritten

[Unit]
Description=Software Heritage Indexer Journal Client
After=network.target

[Service]
Environment=SWH_SENTRY_DSN=https://swh::deploy::indexer::sentry_token@sentry.softwareheritage.org/5
Environment=SWH_SENTRY_ENVIRONMENT=staging
Environment=SWH_MAIN_PACKAGE=swh.indexer
User=swhstorage
Group=swhstorage
Type=simple
ExecStart=/usr/bin/swh indexer --config-file /etc/softwareheritage/indexer/journal_client.yml journal-client indexer origin-intrinsic-metadata
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
<<<
*******************************************
*** End octocatalog-diff on worker3.internal.staging.swh.network
  • [2]
$SWH_PUPPET_ENVIRONMENT_HOME/bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging scheduler0.internal.staging.swh.network
Found host scheduler0.internal.staging.swh.network
WARN     -> Environment "staging-add-prometheus-metrics" contained non-word characters, correcting name to staging_add_prometheus_metrics
WARN     -> Environment "staging-bullseye-rabbitmq-plugin" contained non-word characters, correcting name to staging_bullseye_rabbitmq_plugin
WARN     -> Environment "staging-check-journal-client" contained non-word characters, correcting name to staging_check_journal_client
WARN     -> Environment "staging-check-journal-client-2nd-implementation" contained non-word characters, correcting name to staging_check_journal_client_2nd_implementation
WARN     -> Environment "staging-check-journal-client-first-implem" contained non-word characters, correcting name to staging_check_journal_client_first_implem
WARN     -> Environment "staging-pin" contained non-word characters, correcting name to staging_pin
Cloning into '/tmp/swh-ocd.pFC4u6sm/environments/production/data/private'...
done.
Cloning into '/tmp/swh-ocd.pFC4u6sm/environments/staging/data/private'...
done.
*** Running octocatalog-diff on host scheduler0.internal.staging.swh.network
I, [2022-06-01T11:19:07.109910 #378588]  INFO -- : Catalogs compiled for scheduler0.internal.staging.swh.network
I, [2022-06-01T11:19:07.514855 #378588]  INFO -- : Diffs computed for scheduler0.internal.staging.swh.network
diff origin/production/scheduler0.internal.staging.swh.network current/scheduler0.internal.staging.swh.network
*******************************************
- File[/etc/softwareheritage/indexer/journal_client.yml]
*******************************************
- File[/etc/softwareheritage/indexer]
*******************************************
- File[/etc/systemd/system/swh-indexer-journal-client.service]
*******************************************
- Package[python3-swh.indexer]
*******************************************
- Service[swh-indexer-journal-client]
*******************************************
- Systemd::Unit_file[swh-indexer-journal-client.service]
*******************************************
*** End octocatalog-diff on scheduler0.internal.staging.swh.network

Migrated from D7928 (view on Phabricator)

Merge request reports