Skip to content

Switch production over to the new kafka cluster

Nicolas Dandrimont requested to merge generated-differential-D3951-source into generated-differential-D3951-target
  • Use kafka[1-4].internal.softwareheritage.org as default kafka cluster
  • Anonymize messages sent to the production kafka cluster

Close T2596 Close T2597

Test Plan

octocatalog-diff on uffizi

*** Running octocatalog-diff on host uffizi.softwareheritage.org
I, [2020-09-15T11:29:26.724062 #36603]  INFO -- : Catalogs compiled for uffizi.softwareheritage.org
I, [2020-09-15T11:29:27.017178 #36603]  INFO -- : Diffs computed for uffizi.softwareheritage.org
diff origin/production/uffizi.softwareheritage.org current/uffizi.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/storage/storage.yml] =>
   parameters =>
     content =>
      @@ -95,9 +95,11 @@
             args:
               brokers:
      -        - esnode1.internal.softwareheritage.org
      -        - esnode2.internal.softwareheritage.org
      -        - esnode3.internal.softwareheritage.org
      +        - kafka1.internal.softwareheritage.org
      +        - kafka2.internal.softwareheritage.org
      +        - kafka3.internal.softwareheritage.org
      +        - kafka4.internal.softwareheritage.org
               prefix: swh.journal.objects
               client_id: swh.storage.journal_writer.uffizi
      +        anonymize: true
               producer_config:
                 message.max.bytes: 1000000000
*******************************************
*** End octocatalog-diff on uffizi.softwareheritage.org

octocatalog-diff on getty:

*** Running octocatalog-diff on host getty.internal.softwareheritage.org
I, [2020-09-15T11:31:50.224834 #38835]  INFO -- : Catalogs compiled for getty.internal.softwareheritage.org
I, [2020-09-15T11:31:50.427346 #38835]  INFO -- : Diffs computed for getty.internal.softwareheritage.org
diff origin/production/getty.internal.softwareheritage.org current/getty.internal.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/indexer/journal_client.yml] =>
   parameters =>
     content =>
      @@ -2,7 +2,8 @@
       journal:
         brokers:
      -  - esnode1.internal.softwareheritage.org
      -  - esnode2.internal.softwareheritage.org
      -  - esnode3.internal.softwareheritage.org
      +  - kafka1.internal.softwareheritage.org
      +  - kafka2.internal.softwareheritage.org
      +  - kafka3.internal.softwareheritage.org
      +  - kafka4.internal.softwareheritage.org
         group_id: swh.indexer.journal_client
       scheduler:
*******************************************
*** End octocatalog-diff on getty.internal.softwareheritage.org

Migrated from D3951 (view on Phabricator)

Merge request reports