swh-journal merge requestshttps://gitlab.softwareheritage.org/swh/devel/swh-journal/-/merge_requests2024-03-26T10:42:33Zhttps://gitlab.softwareheritage.org/swh/devel/swh-journal/-/merge_requests/271Fix on_eof behavior when the consumer group is rebalanced2024-03-26T10:42:33ZVincent SellierFix on_eof behavior when the consumer group is rebalancedWhen a CG is rebalanced, no messages are delivered to the consumer
until the stabilization so a timeout occurs in the consume() method.
The empty list is considered as we are at the end of the partition.
The result is all the consumers o...When a CG is rebalanced, no messages are delivered to the consumer
until the stabilization so a timeout occurs in the consume() method.
The empty list is considered as we are at the end of the partition.
The result is all the consumers of the group end at the same time
if the rebalancing takes more than 10s.
The fix wait until a list of partitions are assigned to the consumer
before testing if the end is reached.Vincent SellierVincent Sellierhttps://gitlab.softwareheritage.org/swh/devel/swh-journal/-/merge_requests/232Draft: Implement a separate kafka communication thread for journal clients2024-02-20T14:15:26ZNicolas Dandrimontolasd@softwareheritage.orgDraft: Implement a separate kafka communication thread for journal clientsThis communication thread is in charge of pulling the messages from
kafka and handing them off to a processing thread, as well as doing
regular polling of the rdkafka client (which in turn notifies the
brokers that the consumer is still ...This communication thread is in charge of pulling the messages from
kafka and handing them off to a processing thread, as well as doing
regular polling of the rdkafka client (which in turn notifies the
brokers that the consumer is still alive).
Doing this allows the kafka communication thread to pause the kafka
consumption explicitly when processing a batch of messages takes too
long. This can in turn avoid a lot of rebalance traffic on the kafka
brokers, and overall avoids a bunch of internal rdkafka timeouts.
## Test Plan
We've been using a variant of this on mmca (swh.provenance revision
journal client) for a while.
----
*Migrated from D8797 ([view on Phabricator](https://forge.softwareheritage.org/D8797))*