swh-cron: manifest-based scheduler for recurring tasks
We need a cron-like service to periodically reschedule (= submit relevant celery tasks) recurrent tasks.
Use cases:
- ghlister daily updates
- ghlister complete re-listing (monthly?)
- git cloning/loading of new github repos
- git fetching/loading of already known github repos
- ...
As an architecture, we want the service to read a list of "manifests", that map cron-like periods to swh.core.scheduling.Task objects + their configuration parameters.
As a technological building block we might want to use croniter.
Open design questions:
- cron-like (i.e., will //not// catch up with past tasks, e.g., if the daemon was down), or anacron-like (with catchup)
Migrated from T52 (view on Phabricator)