Reduce git loader memory footprint
It's currently using a huge amount of memory especially on large repositories. Which is currently a blocking point so we need to decrease their concurrency so not everything fails on oom.
Checking a bit the code, currently the loader retrieves the full packfile and then parse it multiple times to ingest the dag objects. We believe it's possible to retrieve packfiles incrementally.
This increases the communication with the server but that should be reducing the memory usage.
Related to swh/infra/sysadm-environment#3025
Migrated from T3625 (view on Phabricator)
Edited by Phabricator Migration user