Skip to content

git.directory: Fix hash computations and make cloning step faster

Antoine R. Dumont requested to merge directory-loader-fix-hash-tree-computation into master
  • 1st commit: Fix hash computations to drop empty folders and the /.git folder
  • 2nd commit: Use treeless clone [1] to make the cloning step faster (without impacting the tree checkouted)

Inspired from the pip cloning step [2]. They are using a blobless clone (all commits but no blobs except for the head). There is another approach treeless (all commits but no trees except for the head). That's faster so i've adapted accordingly. What's interesting is that the next checkout to any references will then just pulls whatever is necessary to have the tree at the right git reference (e.g. commit, tag, branch).

The distinctive approaches are summarized visually in this article [3]. And i have an ongoing comparative clone in samples. The full clone [5], blobless [6] and treeless [4] triggered at the same time on the same machine. Treeless finished first, then blobless and finally full (the current deployed version).

[3] https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/#

[2] https://github.com/pypa/pip/blob/main/src/pip/_internal/vcs/git.py#L279

[1] man git rev-list explained the various filter to use to reach such behavior:

--filter=<filter-spec>
           Only useful with one of the --objects*; omits objects (usually blobs) from the list of printed objects. The <filter-spec> may be one of the following:

           The form --filter=blob:none omits all blobs.

           The form --filter=blob:limit=<n>[kmg] omits blobs larger than n bytes or units. n may be zero. The suffixes k, m, and g can be used to name units in KiB, MiB, or GiB. For example, blob:limit=1k is the same as
           blob:limit=1024.

           The form --filter=sparse:oid=<blob-ish> uses a sparse-checkout specification contained in the blob (or blob-expression) <blob-ish> to omit blobs that would not be not required for a sparse checkout on the requested refs.

           The form --filter=tree:<depth> omits all blobs and trees whose depth from the root tree is >= <depth> (minimum depth if an object is located at multiple depths in the commits traversed). <depth>=0 will not include any
           trees or blobs unless included explicitly in the command-line (or standard input when --stdin is used). <depth>=1 will include only the tree and blobs which are referenced directly by a commit reachable from <commit> or an
           explicitly-given object. <depth>=2 is like <depth>=1 while also including trees and blobs one more level removed from an explicitly-given commit or tree.

           Note that the form --filter=sparse:path=<path> that wants to read from an arbitrary path on the filesystem has been dropped for security reasons.

           Multiple --filter= flags can be specified to combine filters. Only objects which are accepted by every filter are included.

           The form --filter=combine:<filter1>+<filter2>+...<filterN> can also be used to combined several filters, but this is harder than just repeating the --filter flag and is usually not necessary. Filters are joined by + and
           individual filters are %-encoded (i.e. URL-encoded). Besides the + and % characters, the following characters are reserved and also must be encoded: ~!@#$^&*()[]{}\;",<>?'` as well as all characters with ASCII code <=
           0x20, which includes space and newline.

           Other arbitrary characters can also be encoded. For instance, combine:tree:3+blob:none and combine:tree%3A3+blob%3Anone are equivalent.

Refs. swh/meta#3781 (closed)

[1] Depends on swh-loader-core!489 (merged)

[4] treeless (finished first)

$ /usr/bin/time git clone https://github.com/chromium/chromium /tmp/git-tryouts/treeless-chromium --filter=tree:0
Cloning into '/tmp/git-tryouts/treeless-chromium'...
remote: Enumerating objects: 1509089, done.
remote: Counting objects: 100% (1444/1444), done.
remote: Compressing objects: 100% (1139/1139), done.
remote: Total 1509089 (delta 524), reused 1206 (delta 305), pack-reused 1507645
Receiving objects: 100% (1509089/1509089), 648.74 MiB | 17.08 MiB/s, done.
Resolving deltas: 100% (236271/236271), done.
remote: Enumerating objects: 34599, done.
remote: Counting objects: 100% (14170/14170), done.
remote: Compressing objects: 100% (13457/13457), done.
remote: Total 34599 (delta 654), reused 3108 (delta 328), pack-reused 20429
Receiving objects: 100% (34599/34599), 13.68 MiB | 19.57 MiB/s, done.
Resolving deltas: 100% (1636/1636), done.
remote: Enumerating objects: 398885, done.
remote: Counting objects: 100% (175840/175840), done.
remote: Compressing objects: 100% (138854/138854), done.
remote: Total 398885 (delta 50509), reused 42043 (delta 36807), pack-reused 223045
Receiving objects: 100% (398885/398885), 1.14 GiB | 12.66 MiB/s, done.
Resolving deltas: 100% (122281/122281), done.
Updating files: 100% (431087/431087), done.
338.11user 26.48system 8:24.51elapsed 72%CPU (0avgtext+0avgdata 1490196maxresident)k
2375598inputs+13461656outputs (8869major+387018minor)pagefaults 0swaps
$ cd /tmp/git-tryouts/treeless-chromium && /usr/bin/time git switch --detach 10.0.623.0
remote: Enumerating objects: 23266, done.
remote: Counting objects: 100% (12883/12883), done.
remote: Compressing objects: 100% (11790/11790), done.
remote: Total 23266 (delta 2114), reused 1093 (delta 1093), pack-reused 10383
Receiving objects: 100% (23266/23266), 133.84 MiB | 7.96 MiB/s, done.
Resolving deltas: 100% (3905/3905), done.
Updating files: 100% (451286/451286), done.
HEAD is now at 736e72fd911 Publish DEPS for Chromium 10.0.623.0
13.10user 8.24system 0:37.65elapsed 56%CPU (0avgtext+0avgdata 430776maxresident)k
21042inputs+1289120outputs (115major+110113minor)pagefaults 0swaps
$ time swh nar -x -f hex -H sha256 /tmp/git-tryouts/treeless-chromium 2>/dev/null
e1ffc33d9951df5c00696ba6ce1f6087e4aaad32285a94d8654998fef1418374
swh nar -x -f hex -H sha256 /tmp/git-tryouts/treeless-chromium 2> /dev/null  4.00s user 0.76s system 76% cpu 6.243 total

[5] full clone (still, still, still, ... ongoing)

$ /usr/bin/time git clone https://github.com/chromium/chromium /tmp/git-tryouts/full-chromium
Cloning into '/tmp/git-tryouts/chromium'...
remote: Enumerating objects: 21024672, done.
remote: Counting objects: 100% (22150/22150), done.
remote: Compressing objects: 100% (10889/10889), done.
remote: Total 21024672 (delta 11674), reused 20826 (delta 10441), pack-reused 21002522
Receiving objects: 100% (21024672/21024672), 37.75 GiB | 5.93 MiB/s, done.
Resolving deltas: 100% (16644048/16644048), done.
Checking objects: 100% (67108864/67108864), done.
Updating files: 100% (431087/431087), done.
5623.38user 411.97system 2:11:51elapsed 76%CPU (0avgtext+0avgdata 3598844maxresident)k
73741978inputs+90312192outputs (17270major+6544217minor)pagefaults 0swaps
$ /usr/bin/time git switch --detach 10.0.623.0
Updating files: 100% (451286/451286), done.
HEAD is now at 736e72fd91184 Publish DEPS for Chromium 10.0.623.0
5.95user 6.84system 0:13.83elapsed 92%CPU (0avgtext+0avgdata 1652792maxresident)k
2328110inputs+1009808outputs (17329major+78417minor)pagefaults 0swaps
$ time swh nar -x -f hex -H sha256 /tmp/git-tryouts/full-chromium 2>/dev/null
e1ffc33d9951df5c00696ba6ce1f6087e4aaad32285a94d8654998fef1418374
swh nar -x -f hex -H sha256 /tmp/git-tryouts/full-chromium 2> /dev/null  3.69s user 0.45s system 90% cpu 4.582 total

[6] blobless clone (finished 2nd)

$ /usr/bin/time git clone https://github.com/chromium/chromium /tmp/git-tryouts/blobless-chromium --filter=blob:none
Cloning into '/tmp/git-tryouts/blobless-chromium'...
remote: Enumerating objects: 12092775, done.
remote: Counting objects: 100% (12547/12547), done.
remote: Compressing objects: 100% (4849/4849), done.
remote: Total 12092775 (delta 7954), reused 11521 (delta 6978), pack-reused 12080228
Receiving objects: 100% (12092775/12092775), 2.94 GiB | 9.87 MiB/s, done.
Resolving deltas: 100% (9282343/9282343), done.
remote: Enumerating objects: 398885, done.
remote: Counting objects: 100% (175779/175779), done.
remote: Compressing objects: 100% (138805/138805), done.
remote: Total 398885 (delta 50499), reused 41986 (delta 36795), pack-reused 223106
Receiving objects: 100% (398885/398885), 1.14 GiB | 10.35 MiB/s, done.
Resolving deltas: 100% (122324/122324), done.
Updating files: 100% (431087/431087), done.
625.35user 66.95system 15:05.70elapsed 76%CPU (0avgtext+0avgdata 1996696maxresident)k
2544439inputs+19021568outputs (2383major+763468minor)pagefaults 0swaps
$ cd blobless-chromium
$ /usr/bin/time git switch --detach 10.0.623.0
remote: Enumerating objects: 23266, done.
remote: Counting objects: 100% (12883/12883), done.
remote: Compressing objects: 100% (11790/11790), done.
remote: Total 23266 (delta 2114), reused 1093 (delta 1093), pack-reused 10383
Receiving objects: 100% (23266/23266), 133.84 MiB | 6.18 MiB/s, done.
Resolving deltas: 100% (3905/3905), done.
Updating files: 100% (451286/451286), done.
HEAD is now at 736e72fd9118 Publish DEPS for Chromium 10.0.623.0
11.10user 8.45system 0:42.17elapsed 46%CPU (0avgtext+0avgdata 799752maxresident)k
7594inputs+1289120outputs (68major+112405minor)pagefaults 0swaps
$ time swh nar -x -f hex -H sha256 /tmp/git-tryouts/blobless-chromium 2>/dev/null
e1ffc33d9951df5c00696ba6ce1f6087e4aaad32285a94d8654998fef1418374
swh nar -x -f hex -H sha256 /tmp/git-tryouts/blobless-chromium 2> /dev/null  3.83s user 0.40s system 99% cpu 4.235 total

As a summary ("switch" as in "git switch"):

|----------------------+---------+--------+----------+------+---------+---------+-------------+----------+----------+-------+---------|
| Steps                |    user | system |  elapsed | %CPU | avgtext | avgdata | maxresident |   inputs |  outputs | major |   minor |
|----------------------+---------+--------+----------+------+---------+---------+-------------+----------+----------+-------+---------|
| treeless clone       |  338.11 |  26.48 |  8:24.51 |   72 |       0 |       0 |     1490196 |  2375598 | 13461656 |  8869 |  387018 |
| treeless switch      |   13.10 |   8.24 |  0:37.65 |   56 |       0 |       0 |      430776 |    21042 |  1289120 |   115 |  110113 |
|----------------------+---------+--------+----------+------+---------+---------+-------------+----------+----------+-------+---------|
| blobless clone       |  625.35 |  66.95 | 15:05.70 |   76 |       0 |       0 |     1996696 |  2544439 | 19021568 |  2383 |  763468 |
| blobless switch      |   11.10 |   8.45 |  0:42.17 |   46 |       0 |       0 |      799752 |     7594 |  1289120 |    68 |  112405 |
|----------------------+---------+--------+----------+------+---------+---------+-------------+----------+----------+-------+---------|
| full clone (ongoing) | 5623.38 | 411.97 |  2:11:51 |   76 |       0 |       0 |     3598844 | 73741978 | 90312192 | 17270 | 6544217 |
| full switch          |    5.95 |   6.84 |  0:13.83 |   92 |       0 |       0 |     1652792 |  2328110 |  1009808 | 17329 |   78417 |
|----------------------+---------+--------+----------+------+---------+---------+-------------+----------+----------+-------+---------|

conclusion: Treeless is indeed faster ^

Edited by Antoine R. Dumont

Merge request reports