Project 'infra/sysadm-environment' was moved to 'swh/infra/sysadm-environment'. Please update any links and bookmarks that may still have the old path.
Make bitbucket origins ingestion concurrent
The current execution is running slowly for now. It's a simple loop over the mapping file (one line is one origin) to trigger the ingestion per origin.
Make it go faster using the actual loader_oneshot.
For this, we need to iterate over the mapping file and send tasks
swh.loader.mercurial.tasks.LoadMercurial
with the proper parameters.
That also have the following pros:
-
no need to deploy another special worker
-
have the systemctl logs pushed to elk, which make it browsable more easily (at least for staff) [1]
-
[1] http://kibana0.internal.softwareheritage.org:5601/goto/23e431d304850e45e602752e849a3f28
Migrated from T3455 (view on Phabricator)