General improvments of the indexer: Schedule indexer tasks

As of today, the indexer tasks are scheduled through a volatile scheduler [1] to:

1. avoid cluttering the (scheduling) database (softwareheritage-indexer) with message too long (swh/devel/swh-indexer#991 (closed))

2. check multiple queues prior to scheduling new ones

The indexer stack has been developed prior to the scheduler. Now the orchestrator part seems redundant and induces quite some complexity. If this is replaced by the actual scheduling part, it seems the point 2. can be addressed simultaneously.

So to achieve the indexer task scheduling goal, address 1. and 2.

• [1] https://forge.softwareheritage.org/source/snippets/browse/master/ardumont/schedule_with_queue_length_check.py

---

Migrated from T1227 (view on Phabricator)