Skip to content

lister.core: Remove origin creation from lister behavior

Prior to this commit, lister did create origins as well in the archive (swh-storage's db). Now, we plan to only schedule those origins for ingestion (and we let the loaders do the actual creation).

Test Plan

tox

Then swh-docker-dev through multiple runs.

First run:

  • checkout origin/master (without this diff)
  • Add dummy/witness new task type loader-gnu (for the listing to actually succeed, implementation detail of the current lister/loader's status)
  • Run gnu lister through scheduler
  • Check swh-scheduler db (new tasks added)
  • Check swh-storage db (new origins)
  • Check swh-lister db (filled-in cache)

Second run:

  • checkout this diff
  • Add dummy/witness new task type loader-gnu
  • Run gnu lister through scheduler
  • Check swh-scheduler db (new tasks added)
  • Check swh-storage db (no new origins as this behavior got removed here so \m/)
  • Check swh-lister db (still filled-in cache)

Migrated from D1579 (view on Phabricator)

Merge request reports