Skip to content

Use better kafka producer semantics in the journal writers

Nicolas Dandrimont requested to merge generated-differential-D2809-source into generated-differential-D2809-target

What we really want is for the broker to acknowledge all messages before we go on to the next step. That's accomplished by flushing the producer rather than enabling idempotence (which has other side-effects, such as only-once delivery, which we don't really care about as all our consumers are, in effect, idempotent).

Setting acks to all means that the broker acknowledges that all in-sync replicas have persisted the message, which is a stronger guarantee than what we had before.

Depends on !130 (closed)

Test Plan

This seems to have alleviated our broker memory consumption issues in production.

Migrated from D2809 (view on Phabricator)

Merge request reports