Add support for running metadata fetchers without a VCS/package loaders

This will allow updating metadata of repositories known to be inactive in terms of source-code, but may have changes in metadata (eg. watchers, stargazers, ...)

---

Migrated from T4394 (view on Phabricator)

assigned to @vlorentz

marked this issue as related to #4252

added Extrinsic metadata priority:Normal labels

We decided to add recurring fetches, so it will take care both of backfilling now, and visiting from time to time in the future. We're going to assume 3 months for now, as it seems reasonable to not exhaust rate limits.

The plan:

1. add metrics for token usage, as we need that to make decisions
2. add a swh-scheduler component (CLI called from a cron?) that reads from the DB and fills a Kafka topic
3. a journal client that reads from the topic and fetches the metadata

For now, we are only going to use a single topic, which means the whole thing will block when we exhaust a token's rate-limit; but the aim is to not hit rate-limits at all. We can revisit the question later, anyway.

changed milestone to %Collect and index forge metadata [Roadmap - Collect]

added activity::Implementation label

mentioned in merge request swh/devel/swh-storage!1015 (merged)

mentioned in merge request swh/devel/swh-loader-metadata!6 (merged)

done in swh/devel/swh-loader-metadata!6 (merged)

closed