Co-authored-by: vlorentz <vlorentz@softwareheritage.org>

vlorentz authored 3 weeks ago and vlorentz committed 3 weeks ago

This changes from the monolithic CLI function to luigi tasks (now called by the CLI),
one to write the offsets and one to export each topic.

This avoids needing to start a new export from scratch in case of error, so only
the last topic needs to be re-run.

Additionally, this can run topic exporters in parallel, which can save some hours
of run time by still doing work while the exporter of an other topic is not
maximizing CPU usage because it consumed most partitions

This removes all the warnings related to comparing a ModelObjectType
enum with a string literal.

Use importlib.metadata instead.

Bump development tools: mypy, codespell, isort, ...

Move all tools configuration in pyproject.toml.

Remove no longer needed mypy overrides.

vlorentz authored 8 months ago and vlorentz committed 2 months ago

All offsets are now exported before all contents, so it makes more sense to log
the topic being exported on each line, now.

vlorentz authored 1 year ago and vlorentz committed 8 months ago

They allow looking for high cardinality values without decompressing most stripes
not containing any (equality) match.