extid topic is misconfigured in staging and production
The extid topic has the default configuration in staging and production:
- staging:
/opt/kafka/bin/kafka-topics.sh --bootstrap-server journal0.internal.staging.swh.network:9092 --describe --topic swh.journal.objects.extid
Topic: swh.journal.objects.extid PartitionCount: 1 ReplicationFactor: 1 Configs: max.message.bytes=104857600
- production:
/opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka1.internal.softwareheritage.org:9092 --describe --topic swh.journal.objects.extid
Topic: swh.journal.objects.extid PartitionCount: 1 ReplicationFactor: 1 Configs: max.message.bytes=104857600
The cleanup policy need to be configured to compact and the partition count to 64 in staging and 256 in production. The replication also needs to be increased to 2 in production
For staging:
/opt/kafka/bin/kafka-topics.sh --zookeeper $ZK --alter --topic swh.journal.objects.extid --config cleanup.policy=compact --partition 64
For production:
/opt/kafka/bin/kafka-topics.sh --zookeeper $ZK --alter --topic swh.journal.objects.extid --config cleanup.policy=compact --partition 256 --replication-factor 2
The content of the topic needs to be back filed so the previous content will be cleaned after the next compact
Migrated from T3485 (view on Phabricator)
Edited by Vincent Sellier