Skip to content

extid topic is misconfigured in staging and production

The extid topic has the default configuration in staging and production:

  • staging:
/opt/kafka/bin/kafka-topics.sh --bootstrap-server journal0.internal.staging.swh.network:9092 --describe --topic swh.journal.objects.extid 
Topic: swh.journal.objects.extid	PartitionCount: 1	ReplicationFactor: 1	Configs: max.message.bytes=104857600
  • production:
/opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka1.internal.softwareheritage.org:9092 --describe --topic swh.journal.objects.extid 
Topic: swh.journal.objects.extid	PartitionCount: 1	ReplicationFactor: 1	Configs: max.message.bytes=104857600

The cleanup policy need to be configured to compact and the partition count to 64 in staging and 256 in production. The replication also needs to be increased to 2 in production

For staging:

/opt/kafka/bin/kafka-topics.sh --zookeeper $ZK  --alter --topic swh.journal.objects.extid --config cleanup.policy=compact --partition 64

For production:

/opt/kafka/bin/kafka-topics.sh --zookeeper $ZK  --alter --topic swh.journal.objects.extid --config cleanup.policy=compact --partition 256 --replication-factor 2

The content of the topic needs to be back filed so the previous content will be cleaned after the next compact


Migrated from T3485 (view on Phabricator)

Edited by Vincent Sellier