Skip to content
Snippets Groups Projects

Add a backfiller cli command

Closed Add a backfiller cli command
generated-differential-D8964-source into master
Closed David Douard requested to merge generated-differential-D8964-source into master
  1. Jan 06, 2023
    • David Douard's avatar
      Add a backfiller cli command · bd7f58f0
      David Douard authored 
      This command allowd to backfill a kafka journal from an existing
Postgresql provenance storage.

The command will run a given number of workers in parallel. The state of
the backfilling process is saved in a leveldb store, so interrupting and
restarting a backfilling process is possible, with limitations: it won't
work properly if the range generation is modified.
      bd7f58f0
    • David Douard's avatar
      Extract the journal writer part from the ProvenanceStorageJournal class · 4b530e38
      David Douard authored 
      This allows to use the journal writing part independently from the
ProvenanceStorage proxy class, eg. for the backfiller mechanism.
      4b530e38