Make GitHub lister use celery tasks instead of the command-line
The re-engineering of the GitHub lister implies using celery tasks for the "incremental" and "full" operations, so they can be integrated in the scheduler.
Specifically for the GitHub lister, three kinds of tasks are implied, divided in two queues (the incremental operation needs to be able to bypass the queue for the full operation):
-
swh_lister_github_incremental
queue - incremental task maps to the current
ghlister catchup
operation -
swh_lister_github_full
queue - full "meta"-task that schedules range updates
- update-range task that maps to the current
ghlister list <start>-<stop>
operation which takes a range of ids, and can be parallelized.
Migrated from T587 (view on Phabricator)
Edited by Phabricator Migration user