Skip to content

swh-cron: manifest-based scheduler for recurring tasks

We need a cron-like service to periodically reschedule (= submit relevant celery tasks) recurrent tasks.

Use cases:

  • ghlister daily updates
  • ghlister complete re-listing (monthly?)
  • git cloning/loading of new github repos
  • git fetching/loading of already known github repos
  • ...

As an architecture, we want the service to read a list of "manifests", that map cron-like periods to swh.core.scheduling.Task objects + their configuration parameters.

As a technological building block we might want to use croniter.

Open design questions:

  • cron-like (i.e., will //not// catch up with past tasks, e.g., if the daemon was down), or anacron-like (with catchup)

Migrated from T52 (view on Phabricator)

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information