Skip to content

Enable the journal-writer for the swh-idx-storage in production

Antoine R. Dumont requested to merge generated-differential-D5054-source into generated-differential-D5054-target

Related to T2780

Test Plan

bin/octocatalog-diff:

  • No change for storage02.euwest.azure (no journal-writer there)
  • No change for storage1.staging (journal-write already deployed there)
  • Update configuration file for saam's gunicorn-swh-indexer-storage (new journal-writer key)

Details:

$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging storage01.euwest.azure.internal.softwareheritage.org
...
*** Running octocatalog-diff on host storage01.euwest.azure.internal.softwareheritage.org
I, [2021-02-10T11:35:24.795334 #10336]  INFO -- : Catalogs compiled for storage01.euwest.azure.internal.softwareheritage.org
I, [2021-02-10T11:35:26.615042 #10336]  INFO -- : Diffs computed for storage01.euwest.azure.internal.softwareheritage.org
I, [2021-02-10T11:35:26.615089 #10336]  INFO -- : No differences
*** End octocatalog-diff on storage01.euwest.azure.internal.softwareheritage.org
$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging saam
...
*** Running octocatalog-diff on host saam.internal.softwareheritage.org
I, [2021-02-10T11:35:49.373625 #15404]  INFO -- : Catalogs compiled for saam.internal.softwareheritage.org
I, [2021-02-10T11:35:51.286994 #15404]  INFO -- : Diffs computed for saam.internal.softwareheritage.org
I, [2021-02-10T11:35:51.287042 #15404]  INFO -- : No differences
*** End octocatalog-diff on saam.internal.softwareheritage.org
$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging saam
...
*** Running octocatalog-diff on host saam.internal.softwareheritage.org
I, [2021-02-10T11:37:39.109683 #21074]  INFO -- : Catalogs compiled for saam.internal.softwareheritage.org
I, [2021-02-10T11:37:40.885348 #21074]  INFO -- : Diffs computed for saam.internal.softwareheritage.org
diff origin/production/saam.internal.softwareheritage.org current/saam.internal.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/storage/indexer.yml] =>
   parameters =>
     content =>
      @@ -4,3 +4,14 @@
         db: host=belvedere.internal.softwareheritage.org port=5432 user=swhstorage dbname=softwareheritage-indexer
           password=swh::deploy::indexer::storage::db::password
      +  journal_writer:
      +    cls: kafka
      +    brokers:
      +    - kafka1.internal.softwareheritage.org
      +    - kafka2.internal.softwareheritage.org
      +    - kafka3.internal.softwareheritage.org
      +    - kafka4.internal.softwareheritage.org
      +    prefix: swh.journal.indexed
      +    client_id: swh.idx_storage.journal_writer.saam
      +    producer_config:
      +      message.max.bytes: 1000000000
      _
*******************************************
*** End octocatalog-diff on saam.internal.softwareheritage.org

Migrated from D5054 (view on Phabricator)

Merge request reports