Skip to content
GitLab
Open
Issue created by Antoine R. Dumont@ardumontMaintainer1 of 4 checklist items completed1/4 checklist items

Stabilize loader git

Recent runs actually showed that the loader git:

  • #3625 (closed): uses too much memory
  • #3635: does too much work (regarding forks, it does not deduplicate computations for in the end noops)
  • swh/infra/sysadm-environment#3640: is too slow to finish which renders maintenance a pain
  • #3652: Does not properly handle some origins due to a packfile too large

So for example, given too much concurrency, the first point actually make it unstable. Another example, too many forks (at the same time or not) actually make the loader git overall slow.

This task is actually a parent task to track the work done to actually fix those points, make the loader git:

  • uses less memory
  • do less unneeded work
  • stops fast when required to
  • actually able to ingest "larger" origins

The perimeter of this task is frozen to this 4 points (it's not a neverending task ;)

Migrated from T3653 (view on Phabricator)

Edited by Phabricator Migration user