Skip to content

staging: configure idx-storage to write to kafka

Vincent Sellier requested to merge generated-differential-D4620-source into generated-differential-D4620-target

The topic name was temporary chosen for staging according to the T2780 suggestion. The partitioning, retention, and cleanup policy will have to be defined before landing this in production.

Related to T2816

Test Plan

octocatalog-diff

  • storage1.staging
diff origin/production/storage1.internal.staging.swh.network current/storage1.internal.staging.swh.network
*******************************************
  File[/etc/softwareheritage/storage/indexer.yml] =>
   parameters =>
     content =>
      @@ -4,3 +4,11 @@
         db: host=db1.internal.staging.swh.network port=5432 user=swh-indexer dbname=swh-indexer
           password=swh::deploy::indexer::storage::db::password
      +  journal_writer:
      +    cls: kafka
      +    brokers:
      +    - journal0.internal.staging.swh.network
      +    prefix: swh.journal.indexed
      +    client_id: swh.idx_storage.journal_writer.storage1
      +    producer_config:
      +      message.max.bytes: 1000000000
      _
*******************************************
*** End octocatalog-diff on storage1.internal.staging.swh.network
  • No changes in production :
*** Running octocatalog-diff on host saam.internal.softwareheritage.org
I, [2020-11-27T10:32:46.714932 #7376]  INFO -- : Catalogs compiled for saam.internal.softwareheritage.org
I, [2020-11-27T10:32:48.136707 #7376]  INFO -- : Diffs computed for saam.internal.softwareheritage.org
I, [2020-11-27T10:32:48.136747 #7376]  INFO -- : No differences
*** End octocatalog-diff on saam.internal.softwareheritage.org

Migrated from D4620 (view on Phabricator)

Merge request reports