Skip to content
Snippets Groups Projects
Commit 6a6476f3 authored by vlorentz's avatar vlorentz Committed by vlorentz
Browse files

Add support for resuming from interrupted export

This changes from the monolithic CLI function to luigi tasks (now called by the CLI),
one to write the offsets and one to export each topic.

This avoids needing to start a new export from scratch in case of error, so only
the last topic needs to be re-run.

Additionally, this can run topic exporters in parallel, which can save some hours
of run time by still doing work while the exporter of an other topic is not
maximizing CPU usage because it consumed most partitions
parent a2c74d52
No related branches found
No related tags found
1 merge request!139Add support for resuming from interrupted export
Pipeline #14442 passed
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment