Reduce git loader memory footprint
It's currently using a huge amount of memory especially on large repositories. Which is currently a blocking point so we need to decrease their concurrency so not everything fails on oom.
Checking a bit the code, currently the loader retrieves the full packfile and then parse it multiple times to ingest the dag objects. We believe it's possible to retrieve packfiles incrementally.
This increases the communication with the server but that should be reducing the memory usage.
Related to swh/infra/sysadm-environment#3025
Migrated from T3625 (view on Phabricator)
Activity
mentioned in commit b74e2108
mentioned in issue #3627
mentioned in merge request !76 (closed)
mentioned in merge request !83 (closed)
mentioned in merge request swh-storage!731 (closed)
mentioned in merge request swh-storage!733 (closed)
mentioned in merge request swh-loader-core!417 (merged)
mentioned in merge request !136 (closed)
mentioned in merge request !137 (closed)
added Git loader priority:High labels
!76 (closed) actually increased the memory footprint to the point of getting ingestion killed fast. So closed!
!136 (closed) (requires swh-loader-core!417 (merged) in the loader-core) is another attempt which tries to fetch packfiles to ingest instead of one big packfile. The implementation is not entirely satisfactory as it's not deterministic. It really depends on the connected nature of the git repository graph we are trying to ingest (and the order of the refs we are ingestion
- [3]). If it's fully connected, we will have a full packfile immediately (thus rendering ^ moot).
We tried to have a look at the depth parameter of the fetch_pack internally used by the loader [2]. But that part is still fuzzy. Notably, how to determine what a good depth factor could be.
-
[1] https://www.dulwich.io/docs/api/dulwich.client.html#dulwich.client.AbstractHttpGitClient.fetch_pack
-
[2] @anlambert also suggested to have a look at it later on, see !136 (closed).
-
[3] Another idea that was only discussed would be to make certain we first start by ingesting in order tag references (under the assumption that we will then ingest mostly in natural order the repository). Then focus on the remaining references (because mostly there is a high probability that if we start with HEAD and/or master/main at first, we will end up with mostly the overall repository in one one round).
-
[3] Another idea that was only discussed would be to make certain we first start by ingesting in order tag references (under the assumption that we will then ingest mostly in natural order the repository). Then focus on the remaining references (because mostly there is a high probability that if we start with HEAD and/or master at firstz, we will end up with the overall repository).
swh-storage!989 (closed) (tests are fine locally).
-
All runs done from medium to large repositories. No diverging hash and consistently the loader-git ran with the patched version uses less memory.
|---------+-----------------+-------+-------------------------+-------------------------+---------------------+--------------------------------------------------------------| | Machine | Origins | Refs | Snapshot | Memory (max RSS kbytes) | Time (h:)mm:ss(.ms) | Note | |---------+-----------------+-------+-------------------------+-------------------------+---------------------+--------------------------------------------------------------| | staging | torvalds/linux | 1496 | \xc2847...3fb4 | 1361324 | 6:59:16 | | | prod | // | // | \xc2847...3fb4 | 3080408 | 24:13:11 | | |---------+-----------------+-------+-------------------------+-------------------------+---------------------+--------------------------------------------------------------| | staging | CocoaPods/Specs | 14036 | X (hash mismatched) [1] | 5789344 | 23:10:48 | unrelated error: hash mismatched error | | prod | // | // | X (killed) [2] | 14280284 | 10:09:09 | (would have been the same error if not killed first) | |---------+-----------------+-------+-------------------------+-------------------------+---------------------+--------------------------------------------------------------| | staging | keybase/client | 19867 | \xf780...c5a7 | 278568 | 35:26.36 | (loaded recently, visit start from same snapshot as prod) | | | // | | \xf780...c5a7 | 489256 | 4:41:04 | Run from scratch (which not so long ago would have crashed) | | prod | // | // | \xf780...c5a7 | 1381324 | 1:29:02 | (loaded recently, visit start from same snapshot as staging) | |---------+-----------------+-------+-------------------------+-------------------------+---------------------+--------------------------------------------------------------| | staging | cozy/cozy-stack | 3128 | \x58f9a...8edaf | 172404 | 3:35.43 | | | prod | // | // | \x58f9a...8edaf | 1096400 | 5:11.26 | | |---------+-----------------+-------+-------------------------+-------------------------+---------------------+--------------------------------------------------------------| | staging | git/git | 1903 | \x2df8...848e | 464200 | 1:16:17 | unrelated error: hash mismatched error, partial snap | | prod | // | // | X | 2059192 | 2:31:38 | // (exactly the same hash mismatch) | |---------+-----------------+-------+-------------------------+-------------------------+---------------------+--------------------------------------------------------------| | staging | rust-lang/rust | 48615 | \xc59a...9ff3 | 1171708 | 22:23:37 | 1st run, divergent hash (new commit in between prod/staging) | | staging | // | // | \x0abd...3d3d | 899700 | 1:15:27 | 2nd run | | prod | // | // | \x4035...9ecb | 3397172 | 3:05:09 | 1st run, divergent hash (new commit in between prod/staging) | | prod | // | // | \x0abd...3d3d | 3190956 | 2:02:09 | 2nd run | |---------+-----------------+-------+-------------------------+-------------------------+---------------------+--------------------------------------------------------------|Full log details and snapshot extraction in swh/meta$1184
-
One good question got raised on the mailing list thread both by @douardda and @stsp.
Do we already know for sure what is actually causing the memory bloat?
I'm actually not sure yet. So i gave a spin to [1] triggering a run on the most heavy and problematic origins with it (on production nodes keybase/client and CocoaPods/Specs). Let's see what result that gives (ongoing).
-
I'm actually not sure yet. So i gave a spin to [1] triggering a run on the most heavy and problematic origins with it (on production nodes keybase/client and CocoaPods/Specs). Let's see what result that gives (ongoing).
That did not bare result as the script [2] used to profile was using subprocess call. The flamegraph stops at that call. so moot.
Runs on worker17 so production on standard loader-git flamegraph output [3] [4].
-
[2] swh/meta$1189
-
[3] CocoaPods/Specs:
-
[4] keybase/client
-
-
Trigger other runs with memory-profiler instead. [1] It's not perfect though. I cannot find the proper way to actually have the legends as they described in their documentation. [2]
kubernetes/kubernetes, graph with memory over time:
Information with actual memory usage per line of code (corresponding to the graph): worker1-staging-kubernetes-kubernetes-prof-run.txt
We do not see much yet and i'm also missing functions that i required to be profiled, not showed for some reason.