Skip to content

Handle BufferErrors in writer.kafka.Producer

Nicolas Dandrimont requested to merge generated-differential-D5144-source into generated-differential-D5144-target

When the local rdkafka producer queue is full, the producer would crash with a BufferError. While this condition is rare, it can happen when running a backfill operation.

When such a condition arises, retry sending the message a few times with an increasing backoff. When giving up, use the common KafkaDeliveryError mechanism rather than the plain BufferError.

Related to swh-scheduler#3000 (closed)

Test Plan

new tests added for retry and DeliveryError

Migrated from D5144 (view on Phabricator)

Merge request reports