Skip to content

Simplify the lo/high partition offset computation

David Douard requested to merge generated-differential-D6232-source into generated-differential-D6232-target

The computation of lo and high offsets used to be done in 2 steps:

  • first get the watermak offsets (thus the absolute min and max offsets of the whole partition)
  • then, as a "hook" in process(), retrieve the last committed offset for the partition and "push" these current offsets in the progress queue.

Instead, this simplifies a bit this process by quering the committed offsets while computing the hi/low offsets.

Also in this Diff:

  • Use proper signature for JournalClientOffsetRanges.process()

  • Explicitly close the temporary kafka consumer in get_offsets

    used to retrieve partitions and lo/hi offets.

    It could cause some dead-lock/long timeout kind of situation sometime (especially in the developper docker environment).

Migrated from D6232 (view on Phabricator)

Merge request reports