Orchestrate origins scheduling according to scheduler metrics feedback

Currently the next-gen scheduler runners [1] for origins are triggered on each scheduler machine per environment [2]. So far, they are managed manually through a cli call for each type of visits (and eventually, once in a while per lister or forge, ...).

We need to replace those with a new module which does not exist yet, the orchestrator. That module would be in charge of recurringly schedule origins of a given type according to the scheduler metrics available. Deployment wise, that'd be a systemd service that would launch it.

Its scheduling algorithm would be something like:

for each visit type:
  - "check queue status":
      if below threshold for that queue:
        goto "fill-in the void state"
      else
        continue: "next visit type"
      end

  - "fill-in the void state":
    for each (randomly picked?) policy (according to ratio [3]) for that visit type:
      if room in the queue and origins to schedule for that policy:
        schedule origins: push messages in the queue
      elif room in the queue and no more origins to schedule for that policy:
        continue: "next policy"
      else ("no more room in the queue"):
        continue: "next visit type"
      end
    end for
end for

• [1] scheduler runners are process in charge of recurringly schedule origins per visit type (and more if need be, e.g. lister-uuid, ...)

• [2] root tmux session in:

• prod: saatchi

• staging: scheduler0

• [3] Ratio could be:

|----------------------------------------------------------------+-----------------------------------+-------|
| visit_type                                                     | scheduling policies               | ratio |
|----------------------------------------------------------------+-----------------------------------+-------|
| package-loader: archive, cran, debian, npm, nixguix, pypi, ... | already_visited_order_by_lag      |    50 |
|                                                                | never_visited_oldest_update_first |    50 |
|----------------------------------------------------------------+-----------------------------------+-------|
| git, svn, hg                                                   | already_visited_order_by_lag      |    49 |
|                                                                | never_visited_oldest_update_first |    49 |
|                                                                | origins_without_last_update       |     2 |
|----------------------------------------------------------------+-----------------------------------+-------|

---

Migrated from T3667 (view on Phabricator)