Skip to content

Draft: Implement a separate kafka communication thread for journal clients

Nicolas Dandrimont requested to merge generated-differential-D8797-source into master

This communication thread is in charge of pulling the messages from kafka and handing them off to a processing thread, as well as doing regular polling of the rdkafka client (which in turn notifies the brokers that the consumer is still alive).

Doing this allows the kafka communication thread to pause the kafka consumption explicitly when processing a batch of messages takes too long. This can in turn avoid a lot of rebalance traffic on the kafka brokers, and overall avoids a bunch of internal rdkafka timeouts.

Test Plan

We've been using a variant of this on mmca (swh.provenance revision journal client) for a while.

Migrated from D8797 (view on Phabricator)

Merge request reports