Update large workers to consume standard git queue
Those workers are currently used to dedicate some work load to decrease lag on gitlab.com origins. The lag has subsided and those workers are currently less loaded.
So we can dedicate some work load to our standard git queue. Notably, the github origins are lagging in term of consumption, making those workers consume from that queue should help.
This is the task used to track the work to do so.
Plan:
-
infra/puppet/puppet-swh-site!465: make worker[17-18] swh-worker@loader_git consume from the standard git queue -
infra/puppet/puppet-swh-site!465: (if load ok) increase concurrency for that service -
infra/puppet/puppet-swh-site!465: decrease concurrency for swh-worker@loader_oneshot (which is less sensitive currently) -
#3825 (closed): potentially increase the max_queue_length, if the consumption becomes fast enough to drain the current 20k configured. -
[1] Prior to this, the workers were used to ingest bitbucket mercurial and sourceforge origins.
Migrated from T3825 (view on Phabricator)