Implement task runner to schedule oneshot tasks with high priority for listed origins
This is a sub-task for implementing a MVP for the Bulk On-demand Archival feature.
The new save-bulk
lister for the Bulk On-demand Archival feature will record ListedOrigin
objects in scheduler database from a list of origin URLs and their visit types submitted by a user (swh-lister#4709 (closed)).
Those will be recurrently visited by loaders but nevertheless the first visits must be scheduled with high priority to give user quick feedback about the loading statuses of the origins he submitted.
A dedicated task runner for loading origins listed by the save-bulk
lister should be implemented to do so:
- the runner should grab
ListedOrigin
objects by filtering on lister ids corresponding to thesave-bulk
lister type - if origins were not visited since the first listing date, celery tasks must be created in dedicated RabbitMQ queues (different from the ones used for recurrent visits and save code now)
Related to swh/meta#5091.
Edited by Antoine Lambert