Simplify the lo/high partition offset computation
The computation of lo and high offsets used to be done in 2 steps:
- first get the watermak offsets (thus the absolute min and max offsets of the whole partition)
- then, as a "hook" in
process()
, retrieve the last committed offset for the partition and "push" these current offsets in the progress queue.
Instead, this simplifies a bit this process by quering the committed offsets while computing the hi/low offsets.
Also in this Diff:
-
Use proper signature for JournalClientOffsetRanges.process()
-
Explicitly close the temporary kafka consumer in
get_offsets
used to retrieve partitions and lo/hi offets.
It could cause some dead-lock/long timeout kind of situation sometime (especially in the developper docker environment).
Migrated from D6232 (view on Phabricator)