ValueError: Checksum mismatched on <git://git.linux-nfs.org/projects/bfields/nfs4-acl-tools.git>: {'sha256': ...
Sentry Issue: SWH-LOADER-GIT-1N2
ValueError: Checksum mismatched on <git://git.linux-nfs.org/projects/bfields/nfs4-acl-tools.git>: {'sha256': 'd665d94efccddaefa75b09416fa0488f8700211a350bca652c3d42365507cd21'} != {'sha256': '31c0d1a8a485332da20d42517aa44dafee2e48477aa390f47bdaf7a955eb0953'}
File "swh/loader/core/loader.py", line 433, in load
more_data_to_fetch = self.fetch_data()
File "swh/loader/core/loader.py", line 811, in fetch_data
raise errors[0]Refs. swh/meta#4979 (closed)
Activity
moved from swh/infra/sysadm-environment#4928 (moved)
mentioned in issue swh/meta#4979 (closed)
It seems there exists difference what's listed in the upstream manifests and what's git 'checkout'ed ('switch'ed). [1]
The code and the clis (including guix's) are in agreement (last 2 cli calls) and diverge from the hash listed in the upstream manifest (what's passed as argument to the loader from the lister listing the upstream manifest).
$ git clone https://github.com/daattali/jquery-colourpicker --filter=tree:0 Cloning into 'jquery-colourpicker'... ... $ cd jquery-colourpicker $ git switch --detach 27c2a266d51e18a9fe6d7542264152b27c7d34e0 ... HEAD is now at 27c2a26 remove old version text from js $ swh nar -x -f hex -H sha256 /var/tmp/git-tryouts/jquery-colourpicker 2>/dev/null 0b473e867ae80f408ccac98448c62ffb3449337c38a3210d11d7c08e6055e851 $ guix hash -x -S nar -f hex -H sha256 /var/tmp/git-tryouts/jquery-colourpicker 0b473e867ae80f408ccac98448c62ffb3449337c38a3210d11d7c08e6055e851Edited by Antoine R. Dumont-
Hello,
from sentry which i don't recall whether you have access or not [1] (public one [2]).
Checksum mismatched on <https://github.com/daattali/jquery-colourpicker>: {'sha256': 'fa59934adf625fd8c342ead7c024e5530dd46fb11aa34c6b6aafaea5c2f6b0c8'} != {'sha256': '0b473e867ae80f408ccac98448c62ffb3449337c38a3210d11d7c08e6055e851'}[2] https://sentry.softwareheritage.org/share/issue/01a45a4b7dc049c5aedf60a3f896d428/
-
ah but the public sentry url just shows the last instance of such issue... (nvm then)
Edited by Antoine R. Dumont -
I do not have access to sentry; I have just sent a request. :-)
i've added you (i think?!)
Ah... I do not know where this hash fa599... could come from. Well, the commit is from 2016 and the repository seems "simple". Therefore, I do not know what could be done differently in order to get fa599... instead of 0b473....
I'd expect this was what the lister compulsed out of the guix manifest (at the time).
-
It seems to be the source of such issues. As i can reproduce it time and again (hash mismatch from upstream [1]) (out of parameters [2] [3]).
[1]
$ url=https://github.com/dimkr/loksh; git clone --filter=tree:0 $url Cloning into 'loksh'... ... Receiving objects: 100% (51/51), 250.44 KiB | 5.82 MiB/s, done. $ cd loksh $ git switch --detach 6.9 remote: Enumerating objects: 4, done. Receiving objects: 100% (11/11), 114.34 KiB | 3.57 MiB/s, done. HEAD is now at 85a4de6 Fix release name # what's computed from the clone+switch operations happened $ swh nar -x -f hex -H sha256 /var/tmp/git-tryouts/loksh 2>/dev/null be23318f56b3ebe1ff224484892f77b481f07ceb231d35fbeb6aecf9b124dd9c # this one, just to ensure it's not our implementation which is off $ guix hash -x -S nar -f hex -H sha256 /var/tmp/git-tryouts/loksh be23318f56b3ebe1ff224484892f77b481f07ceb231d35fbeb6aecf9b124dd9c # what's expected $ echo fd829fa2abc0bc3348b58ffa70b068e85d583e63af45fca000a240883bbd6374 fd829fa2abc0bc3348b58ffa70b068e85d583e63af45fca000a240883bbd6374[2] http://kibana0.internal.softwareheritage.org:5601/goto/4921978447138c8d2f556ae2bd14b203
[3] lister's generated parameter (out of the upstream manifest) fed to the loader)
{'url': 'https://github.com/dimkr/loksh', 'lister_name': 'nixguix', 'lister_instance_name': 'guix.gnu.org', 'ref': '6.9', 'checksums': {'sha256': 'fd829fa2abc0bc3348b58ffa70b068e85d583e63af45fca000a240883bbd6374'}, 'fallback_urls': [], 'checksum_layout': 'nar'} -
Yes, another check up to the manifest [1] shows something is off between the listing and the actual checkout. Hence the loader not being too happy about it (this issue).
The manifests lists a nar hash <7638c91fe9a49e639f95e5498529022ea0886842e1da08d9644ff174d66e3ed6> to check but it's not what's currently checkouted <0cf5945a7158e25af4ab40589fda5ada81217ae01675df6248b9a52b88e48ea8> [2].
[1] Manifest extract:
$ curl -k curl -k https://guix.gnu.org/sources.json > /var/tmp/guix-tryout/$(today)-sources.json $ file /var/tmp/guix-tryout/20230822-1534-sources.json /var/tmp/guix-tryout/20230822-1534-sources.json: ASCII text, with very long lines, with no line terminators $ ipython In [1]: filepath = '/var/tmp/guix-tryout/20230822-1534-sources.json' In [3]: import json In [4]: with open(filepath, 'r') as f: guix_data = json.loads(f.read())['sources'] In [5]: origins_git = [origin for origin in guix_data if origin['type'] == 'git' ] In [8]: loksh_origin = [o for o in origins_git if o['git_url'] == 'https://github.com/dimkr/loksh'][0] In [9]: loksh_origin Out[9]: {'type': 'git', 'git_url': 'https://github.com/dimkr/loksh', 'integrity': 'sha256-djjJH+mknmOfleVJhSkCLqCIaELh2gjZZE/xdNZuPtY=', 'outputHashAlgo': 'sha256', 'outputHashMode': 'recursive', 'git_ref': '7.3'} In [10]: loksh_origin['integrity'] Out[10]: 'sha256-djjJH+mknmOfleVJhSkCLqCIaELh2gjZZE/xdNZuPtY=' In [11]: def decode(integrity: str): ...: """Compute checksum hash out of an integrity field. ...: ...: """ ...: checksum_algo, chksum_b64 = integrity.split("-") ...: checksum = base64.decodebytes(chksum_b64.encode()).hex() ...: return checksum ...: In [13]: import base64 In [14]: decode(loksh_origin['integrity']) Out[14]: '7638c91fe9a49e639f95e5498529022ea0886842e1da08d9644ff174d66e3ed6'[2] Local checkout (from previous comment we already had a local clone so we only switch to the reference 7.3)
$ # In loksh repository $ git switch --detach 7.3 Previous HEAD position was 85a4de6 Fix release name HEAD is now at e88658f Update lolibc $ swh nar -x -f hex -H sha256 /var/tmp/git-tryouts/loksh 2>/dev/null 0cf5945a7158e25af4ab40589fda5ada81217ae01675df6248b9a52b88e48ea8 $ guix hash -x -S nar -f hex -H sha256 /var/tmp/git-tryouts/loksh 0cf5945a7158e25af4ab40589fda5ada81217ae01675df6248b9a52b88e48ea8Edited by Antoine R. Dumont 
Hi @ardumont
I think the issue for
lokshcomes from the submodule.The Guix definition reads,
(version "7.3") (source (origin (method git-fetch) (uri (git-reference (url "https://github.com/dimkr/loksh") (commit version) ;; Include the ‘lolibc’ submodule, a static compatibility library ;; created for and currently used only by loksh. (recursive? #t))) (file-name (git-file-name name version)) (sha256 (base32 "1miydvb79wagckchinp189l8i81f08lqajg5jngn77m4x4gwjf3n"))))As you can see,
$ git clone https://github.com/dimkr/loksh Cloning into 'loksh'... remote: Enumerating objects: 817, done. remote: Counting objects: 100% (116/116), done. remote: Compressing objects: 100% (61/61), done. remote: Total 817 (delta 53), reused 98 (delta 38), pack-reused 701 Receiving objects: 100% (817/817), 771.04 KiB | 358.00 KiB/s, done. Resolving deltas: 100% (531/531), done. $ builtin cd loksh $ git switch --detach 7.3 HEAD is now at e88658f Update lolibc $ guix hash -rx . 1a4fwj42p9dr91idyx8nw1x230fsbbd9yn20mgs5mqjqf5d99x8cand the hash mismatches indeed. Now, let update the Git submodule and rehash:
$ git submodule init Submodule 'subprojects/lolibc' (https://github.com/dimkr/lolibc) registered for\ path 'subprojects/lolibc' $ git submodule update Cloning into '/tmp/loksh/subprojects/lolibc'... Submodule path 'subprojects/lolibc': checked out 'f6e9da78ba0ca66a8711a7e8cc64e\ 418e68eaa00' $ guix hash -rx . 1miydvb79wagckchinp189l8i81f08lqajg5jngn77m4x4gwjf3n $ guix hash -x -S nar -f base64 -H sha256 . djjJH+mknmOfleVJhSkCLqCIaELh2gjZZE/xdNZuPtY= $ guix hash -x -S nar -f hex -H sha256 . 7638c91fe9a49e639f95e5498529022ea0886842e1da08d9644ff174d66e3ed6It matches. :-)
-
@civodul @zimoun just so you know, i had speed issues with the git ingestion of git tree checkout which got overcome by using treeless cloning (and some folder filtering too) [1]
But "now" i'm facing a hash discrepancy issue [2] between what's listed in the guix manifest and what's actually computed on disk (both the guix and the swh tool agree on the nar hash computed so there might be either an issue with the tool that compulse the manifest or in the tool that computes the nar hash).
Any thoughts on the matter?
[1] !162 (merged)
[2] tbf, it was already there but the loader was so slow... it was not that urgent a fix.
Edited by Antoine R. Dumont -
Hi @ardumont
About
<git://git.linux-nfs.org/projects/bfields/nfs4-acl-tools.git>, the hash that is expected is31c0d1a8a485332da20d42517aa44dafee2e48477aa390f47bdaf7a955eb0953and I do not know what is the other one. Could you provide some context where this hash is computed?Cheers, simon
-
jsyk, there are more than 3 repo in failures
Context is that the the lister nixguix (deployed in staging) is listing more kind of origins including git tree checkouts (to store their nar in swh alongside their dag if the hash match). But currently, there remains some hash mismatch which lead to this opened issue.
Thx to you, i've got one more insight at least for the submodule part (the loader does nothing regarding submodule, no --recurse-submodule or submodule init step is done currently when cloning).
for the origin git://git.linux-nfs.org... [1], it's an old issue (the one i opened from sentry at the time). I don't have anything more than the following expected hash (i don't see the git reference required in the issue).
When cloning it locally though, i still have the same 31c0... nar hash from the current master head.
Checksum mismatched on <git://git.linux-nfs.org/projects/bfields/nfs4-acl-tools.git>: {'sha256': 'd665d94efccddaefa75b09416fa0488f8700211a350bca652c3d42365507cd21'} != {'sha256': '31c0d1a8a485332da20d42517aa44dafee2e48477aa390f47bdaf7a955eb0953'}Although the latest guix manifest lists:
In [11]: nfs4_origin Out[11]: {'type': 'git', 'git_url': 'git://git.linux-nfs.org/projects/bfields/nfs4-acl-tools.git', 'integrity': 'sha256-McDRqKSFMy2iDUJReqRNr+4uSEd6o5D0e9r3qVXrCVM=', 'outputHashAlgo': 'sha256', 'outputHashMode': 'recursive', 'git_ref': 'nfs4-acl-tools-0.3.7'}But no, that mismatches nonetheless
$ git switch --detach nfs4-acl-tools-0.3.7 HEAD is now at 8892483 nfs4-acl-tools 0.3.7 $ guix hash -x -S nar -f hex -H sha256 . 31c0d1a8a485332da20d42517aa44dafee2e48477aa390f47bdaf7a955eb0953 $ git submodule init $ guix hash -x -S nar -f hex -H sha256 . 31c0d1a8a485332da20d42517aa44dafee2e48477aa390f47bdaf7a955eb0953 $ guix hash -rx . 0lq9xdaskxysggs918vs8x42xvmg9nj7lla21ni2scw5ljld3h1i(^ still not McDRqKSFMy2iDUJReqRNr+4uSEd6o5D0e9r3qVXrCVM=)
Edited by Antoine R. Dumont -
Could you provide some context where this hash is computed?
Git directory loader class [1] is in charge of loading such repository. Which inherits the behavior from the base class BaseDirectoryLoader [2]. Which also inherits the hash computation behavior from its base class NodeLoader [3].
The unfortunate inheritance design stems from the initial core loader which we cannot overcome easily. So it's still class inheriting from class loop hell to stay DRY (but not KISS ¯\(ツ)/¯).
Edited by Antoine R. Dumont -
Well, IIUC, the main logic is:
for artifact_path in self.fetch_artifact(): [...] nar.serialize(artifact_path) actual_checksums = nar.hexdigest() if actual_checksums != self.checksums: errors.append(And assuming that
.serialize()is somehow validated by cross-comparison with the both CLIswh narandguix hash, then it would mean that the hash mismatch would come from.fetch_artifact(). The method is maybe not doing the same asgit clone ... && git swicth --detach. Well, do you have access torepo_pathfrom:def fetch_artifact(self) -> Iterator[Path]: with raise_not_found_repository(): with tempfile.TemporaryDirectory() as tmpdir: repo_path = clone_repository( self.origin.url, self.git_ref, target=Path(tmpdir) ) yield repo_path? I mean, is it possible to run something like
swh nar -x -f hex -H sha256 <repo_path>where<repo_path>is the actual path on disk? If it is possible, it would help to reduce the scope of the potential issue, no?Cheers, simon
-
Well, IIUC, the main logic is:
yes, that's it.
See
clone_repositoryfunction is within the same swh.loader.git.directory module [4].I mean, is it possible to run something like swh nar -x -f hex -H sha256 <repo_path> where <repo_path> is the actual path on disk?
It should not be necessary. That's already what's the Nar code you pinpointed does.
nar = Nar(list(self.checksums.keys()), exclude_vcs=True) nar.serialize(artifact_path)If it is possible, it would help to reduce the scope of the potential issue, no?
(talking about origins without any git submodule...)
For me, 'swh nar' and the actual loader renders the same hash (out of the local checkout) but it does not correspond to what's fed to the loader hence the issue (again except for the origins you spotted have submodules).
And assuming that .serialize() is somehow validated by cross-comparison with the both CLI swh nar and guix hash, then
It's actually doing the computation exactly like the 'swh nar' does (what you initially provided me with a while back). Then the hexdigest call is simply rendering the hash results into a dict of {hash_algo: hash_result_in_hex_format, ...} with hash in {'sha256', ...}. That's how hash is computed in python world (so we kept the Nar object compliant to this scheme).
it would mean that the hash mismatch would come from .fetch_artifact().
Fetch artifact (as per your 2nd quote) is just cloning the repository (see [1] for the actual implementation) and provide the repository's local checkout path for hashing (and when it needs to, the hash is actually a nar one, that depends on the lister's output which is the loader's input ;).
The method is maybe not doing the same as git clone ... && git swicth --detach.
For the submodule you identified, yes, it's missing
--recurse-submodule(or whatever the git flag to do that is [4]). But for the rest, it's ok.Note that no all origins are failing. You only see in sentry the ones failing.
Well, do you have access to repo_path from:
Yes, that's actually the artifact_path you read from [5] (the loop is described in the [3] link in my last comment ;).
Rereading, everything again. I guess, you'd like to be able to add something like...
def fetch_artifact(self) -> Iterator[Path]: with raise_not_found_repository(): with tempfile.TemporaryDirectory() as tmpdir: repo_path = clone_repository( self.origin.url, self.git_ref, target=Path(tmpdir) ) guix_cmd = [ "guix", "hash", "-x", "-S", "nar", "-f", "hex", "-H", "sha256", repo_path ] check_output(guix_cmd) yield repo_path?
-
(talking about origins without any git submodule...)
For me, 'swh nar' and the actual loader renders the same hash (out of the local checkout) but it does not correspond to what's fed to the loader hence the issue
What does it mean "it does not correspond to what's fed to the loader hence the issue"? I mean, the file
sources.jsoncontains hashes that are the same as the local checkout. Well, from my understanding.IIUC, one mismatch is:
ValueError: Checksum mismatched on <git://git.linux-nfs.org/projects/bfields/nfs4-acl-tools.git>: {'sha256': 'd665d94efccddaefa75b09416fa0488f8700211a350bca652c3d42365507cd21'} != {'sha256': '31c0d1a8a485332da20d42517aa44dafee2e48477aa390f47bdaf7a955eb0953'}```then I get,
$ wget https://guix.gnu.org/sources.json $ cat sources.json | jq | grep -C 5 nfs4-acl-tools "outputHashAlgo": "sha256", "outputHashMode": "flat" }, { "type": "git", "git_url": "git://git.linux-nfs.org/projects/bfields/nfs4-acl-tools.git", "integrity": "sha256-McDRqKSFMy2iDUJReqRNr+4uSEd6o5D0e9r3qVXrCVM=", "outputHashAlgo": "sha256", "outputHashMode": "recursive", "git_ref": "nfs4-acl-tools-0.3.7" }, { "type": "url", "urls": [ "https://nickle.org/release/nickle-2.90.tar.gz", $ git clone git://git.linux-nfs.org/projects/bfields/nfs4-acl-tools.git Cloning into 'nfs4-acl-tools'... remote: Counting objects: 443, done. remote: Compressing objects: 100% (184/184), done. remote: Total 443 (delta 257), reused 443 (delta 257) Receiving objects: 100% (443/443), 272.93 KiB | 257.00 KiB/s, done. Resolving deltas: 100% (257/257), done. $ cd nfs4-acl-tools $ git switch --detach nfs4-acl-tools-0.3.7 HEAD is now at 8892483 nfs4-acl-tools 0.3 $ guix hash -x -S nar -f base64 -H sha256 . McDRqKSFMy2iDUJReqRNr+4uSEd6o5D0e9r3qVXrCVM= $ guix hash -x -S nar -f hex -H sha256 . 31c0d1a8a485332da20d42517aa44dafee2e48477aa390f47bdaf7a955eb0953Therefore, it appears to me that the error is the computation of
d665d94efccddaefa75b09416fa0488f8700211a350bca652c3d42365507cd21by the loader. Do I miss something?Somehow, assuming that the code
.serialize()is correct, the mismatch would come from that the loader is hashing a different content.Therefore, yes it would appear to me worth to be able to double-check that the content living at
repo_pathis the one that is expected. Well, yes run some Guix command (orswh nar) onrepo_pathas the re-reading code appear to me helpful for debugging the mismatch. Even, if it is possible, instead of runningguix hash, I would run kind ofcopy(repo_path, "/var/tmp/" + repo_path)or whatever other location. Then, having access to/var/tmp/xxxxwherexxxxis therepo_pathof thisnfs4-acl-toolsproblematic repo, this access would help, IMHO. For instance runswh nar /var/tmp/xxxxand compare the hash. Thengit cloneand run somediffbetween the two repositories. Etc.Well, I do not know if it makes sense.
Cheers, simon
-
"it does not correspond to what's fed to the loader hence the issue"?
i excluded the issue from the (swh) nar computation [1] here so i thought this was something mislisted (in the upstream manifest).
[1] because i had a different hash too from my own local checkout
That part
$ guix hash -rx . 0lq9xdaskxysggs918vs8x42xvmg9nj7lla21ni2scw5ljld3h1i(^ still not McDRqKSFMy2iDUJReqRNr+4uSEd6o5D0e9r3qVXrCVM=)
But that was me misusing the guix hash as you just showed me.
$ guix hash -x -S nar -f base64 -H sha256 . McDRqKSFMy2iDUJReqRNr+4uSEd6o5D0e9r3qVXrCVM=I mean, the file sources.json contains hashes that are the same as the local checkout. Well, from my understanding.
yes, i was having some doubt (because of my guix hash misuse).
Therefore, it appears to me that the error is the computation of d665d94efccddaefa75b09416fa0488f8700211a350bca652c3d42365507cd21 by the loader. Do I miss something?
yes.
Somehow, assuming that the code .serialize() is correct, the mismatch would come from that the loader is hashing a different content.
hashing with both command guix (correctly!) and swh nar should make sure whether swh nar is correct or not.
Guix command (or swh nar)
I'd say both (
swh narshould have the shortcomings the loader has if any).... Well, I do not know if it makes sense.
totally makes sense ;)
Thanks. I'll see what i can do.
Edited by Antoine R. Dumont -
Hi,
Yeah all the hashing parameters can be confusing. :-)
- The recipe of Guix packages stores
-S nar -H sha256 -f nix-base32which is a modified Base-32 format. Well, thus, this is the default of the Guix world. - The file
sources.jsonlists-S nar -H sha256 -f base64which is just the format conversion of the latter. - The loader seems using
-S nar -H sha256 -f hex.
Well, since the file
sources.jsonis generated specifically for SWH and since the format is "versioned", it would be possible to change the hash format frombase64tohexand update the format version.Let me know how I can help for fixing this checksum mismatch.
Cheers, simon
- The recipe of Guix packages stores
mentioned in commit 6be839dd
mentioned in merge request !163 (merged)
mentioned in commit 32c9ddf7
-
Related to swh/meta#4979 (comment 148022)
mentioned in commit anlambert/swh-loader-core@47b212e1
mentioned in merge request swh-loader-core!493 (merged)
mentioned in commit anlambert/swh-loader-core@3305d60f
mentioned in merge request !164 (merged)
@ardumont @zimoun I took some time to analyze that issue by proceeding the following way:
-
I extracted the list of git origin URLs from the associated Sentry issue using a script of mine.
-
I used the following script to execute the git directory loading on these origin URLs by getting the loader parameters from the guix_sources.json and dumping loading errors to a file (requires swh-loader-core!49 (closed):
import base64 import json from collections import defaultdict from subprocess import run def get_nar_hex_checksum(nar_hash): algo, b64_checksum = nar_hash.split("-") return json.dumps({algo: base64.b64decode(b64_checksum).hex()}) with open("guix_sources.json", "r") as guix_sources_file: guix_sources = json.load(guix_sources_file) git_sources = defaultdict(list) for guix_source in guix_sources["sources"]: if guix_source["type"] != "git" or guix_source["outputHashMode"] != "recursive": continue git_sources[guix_source["git_url"]].append( { "ref": f"'{guix_source['git_ref']}'", "checksum_layout": "nar", "checksums": get_nar_hex_checksum(guix_source["integrity"]), } ) for origin_url in open("origin-urls-nar-mismatch", "r"): origin_url = origin_url.rstrip("\n") if origin_url not in git_sources: continue for params in git_sources[origin_url]: swh_command = [ "swh", "-l", "DEBUG", "loader", "run", "git-checkout", origin_url, ] + [f"{k}={v}" for k, v in params.items()] completed_process = run(swh_command, capture_output=True) if completed_process.returncode != 0: # to ensure command can be executed in terminal after copy paste print(" ".join(swh_command).replace("{", "'{").replace("}", "}'")) print(completed_process.stderr.decode())After analyzing the loading errors, almost all of them were related to submodules handling and I noticed two issues:
-
The NAR hash computation code implemented in
swh-loader-corewas not excluding VCS directories or special paths in a recursive way. This could result in different directory hash computation as git creates a.gitfile in each submodule folder and it should not be taken into account when computing the hash (as guix does). The fix for this can be found in swh-loader-core!493 (merged). -
The loader was erroneously always fetching submodules but it turned out guix needs them only for a couple of packages so this was resulting in quite a lot of hash mismatches in the loader. I refined the way submodules are handled in !164 (merged).
After applying both fixes, almost all origin URLs from the list I extracted could be successfully loaded. Nevertheless, it remains some cases where there is still hashes mismatch, I will give some examples in another comment.
Edited by Antoine Lambert-
-
Below you can find some remaining origins with hashes mismatch.
First let's take an example where we can successfully recompute the NAR hash computed by guix: the
python-send2trashpackage. It is defined as follow in guix package specification:(define-public python-send2trash (package (name "python-send2trash") (version "1.8.0") (source (origin (method git-fetch) ;; Source tarball on PyPI doesn't include tests. (uri (git-reference (url "https://github.com/arsenetar/send2trash") (commit version))) (file-name (git-file-name name version)) (sha256 (base32 "1k7dfypaaq4f36fbciaasv72j6wgjihw8d88axmz9c329bz8v5qx")))) (build-system python-build-system) (arguments '(#:phases (modify-phases %standard-phases (add-before 'check 'pre-check (lambda _ (setenv "HOME" "/tmp"))) (replace 'check (lambda* (#:key tests? #:allow-other-keys) (when tests? (invoke "pytest" "-vv"))))))) (native-inputs (list python-pytest)) (home-page "https://github.com/arsenetar/send2trash") (synopsis "Send files to the user's @file{~/Trash} directory") (description "This package provides a Python library to send files to the user's @file{~/Trash} directory.") (license license:bsd-3)))and its related entry in the
guix_sources.jsonfile is:{ "type": "git", "git_url": "https://github.com/arsenetar/send2trash", "integrity": "sha256-HZeN/kpisPRrVwg1xGGUjxspztZKRbacGY5gpa537cw=", "outputHashAlgo": "sha256", "outputHashMode": "recursive", "git_ref": "1.8.0" }We can compute and check the NAR hash using the following:
$ git clone https://github.com/arsenetar/send2trash $ cd send2trash/ $ git checkout 1.8.0 $ $ guix hash -x -S nar -H sha256 -f nix-base32 . 1k7dfypaaq4f36fbciaasv72j6wgjihw8d88axmz9c329bz8v5qx $ guix hash -x -S nar -H sha256 -f base64 . HZeN/kpisPRrVwg1xGGUjxspztZKRbacGY5gpa537cw= $ swh nar -x -H sha256 -f base64 . HZeN/kpisPRrVwg1xGGUjxspztZKRbacGY5gpa537cw=We can see the computed hashes are the same as in package specification and JSON data.
However for some packages, the computed hash does not match the one provided by guix. I paste below the JSON metadata for those I found while working on that issue:
{ "type": "git", "git_url": "https://github.com/BinaryAnalysisPlatform/bap", "integrity": "sha256-hSP1vrKE+eLiA2jWfzg0xJGrnHOKTYcxAkxr28G9ibs=", "outputHashAlgo": "sha256", "outputHashMode": "recursive", "git_ref": "v2.5.0-alpha" }{ "type": "git", "git_url": "https://github.com/apache/groovy", "integrity": "sha256-1aZTzpwfRmLNyqL+LQ63NZk2+AfWeAf7UMOyKjs2pQA=", "outputHashAlgo": "sha256", "outputHashMode": "recursive", "git_ref": "GROOVY_3_0_5" }{ "type": "git", "git_url": "http://git.bouncycastle.org/repositories/bc-java", "integrity": "sha256-30al+VVvK4LAH2uvJV7FnnGmHsxMUrwADnoA6N3BiZA=", "outputHashAlgo": "sha256", "outputHashMode": "recursive", "git_ref": "r1rv67" }{ "type": "git", "git_url": "https://github.com/moderngl/glcontext", "integrity": "sha256-TGkVDZbxxvOOal+rLHeCNUoyOzvg9wQsAMan8LDn938=", "outputHashAlgo": "sha256", "outputHashMode": "recursive", "git_ref": "2.4.0" }{ "type": "git", "git_url": "https://github.com/s-andrews/FastQC", "integrity": "sha256-E6O3mLszo6KuM/nhRxeS0FaJgtbaI+o64v3OBWpuyQM=", "outputHashAlgo": "sha256", "outputHashMode": "recursive", "git_ref": "v0.11.9" }{ "type": "git", "git_url": "https://github.com/kitao/pyxel", "integrity": "sha256-qbKoFg6Bpvy3W7OkUO7yxeqfztiGAGdZOkQX7Mt6mi8=", "outputHashAlgo": "sha256", "outputHashMode": "recursive", "git_ref": "v1.4.3" }My guess is there is a side effect, likely due to my environment, that prevents to compute the same hash as guix. I dug into guix code to check if some kind of special processing was done when fetching the code from git but did not find anything relevant. @zimoun maybe this could ring a bell to you ?
-
Hi @anlambert
First, sorry to have missed the recursive case. And thanks for the fix. :-)
{ "type": "git", "git_url": "https://github.com/BinaryAnalysisPlatform/bap", "integrity": "sha256-hSP1vrKE+eLiA2jWfzg0xJGrnHOKTYcxAkxr28G9ibs=", "outputHashAlgo": "sha256", "outputHashMode": "recursive", "git_ref": "v2.5.0-alpha" }It seems an upstream inplace replacement.
I read Guix commit
404df667e3b06f1e9a416c956e53c03ca3642140from Jun 28, 2022. Then, if I read correctly, the tagv2.5.0-alphais rewritten upstream on Jul 9, 2022.SWH says:
a972f8a419294dfb21847db5172ba58c5d7767ebauthored by Ivan Gotovchits on 08 June 2022, 23:57:08 UTC then probably replaced by4d019ce6f26764cc39b4252eb866908094d2ca10authored by Ivan Gotovchits on 30 June 2022, 12:54:55 UTC then probably replaced...$ guix hash -S nar -f base64 -H sha256 -x $(guix build -S bap) hSP1vrKE+eLiA2jWfzg0xJGrnHOKTYcxAkxr28G9ibs= $ guix hash -S nar -f nix-base32 -H sha256 -x $(guix build -S bap) 1fw9pp0xnssc08qqfkcafffap4f46hw7zmk80gif5yc4nazga8w5 $ guix build -S bap --check The following derivation will be built: /gnu/store/931x0yxqr800alglvc3gkqaqkfrgwh1b-bap-2.5.0-alpha-checkout.drv building /gnu/store/931x0yxqr800alglvc3gkqaqkfrgwh1b-bap-2.5.0-alpha-checkout.drv... guile: warning: failed to install locale environment variable `PATH' set to `/gnu/store/y3vdq2pdkljrw63xxnc2vb6lz07ycar6-git-minimal-2.41.0/bin:/gnu/store/78rf44kf4xf6lc01jjy9ci5905j2344l-gzip-1.12/bin:/gnu/store/z45rrcnvcw31h1wbl1zh8hh79jkwv0gs-tar-1.34/bin' Initialized empty Git repository in /gnu/store/axrfwjm357glpn02dil5zfis8ma51z6w-bap-2.5.0-alpha-checkout/.git/ From https://github.com/BinaryAnalysisPlatform/bap * tag v2.5.0-alpha -> FETCH_HEAD Note: switching to 'FETCH_HEAD'. [...] HEAD is now at baa9022 prevents knowledge conflicts on mangled names (#1535) r:sha256 hash mismatch for /gnu/store/axrfwjm357glpn02dil5zfis8ma51z6w-bap-2.5.0-alpha-checkout: expected hash: 1fw9pp0xnssc08qqfkcafffap4f46hw7zmk80gif5yc4nazga8w5 actual hash: 0lywz6ld9jmbryf8m4fnq793ylwh34kf5qws6w60ibkhg8wkjhlz hash mismatch for store item '/gnu/store/axrfwjm357glpn02dil5zfis8ma51z6w-bap-2.5.0-alpha-checkout' build of /gnu/store/931x0yxqr800alglvc3gkqaqkfrgwh1b-bap-2.5.0-alpha-checkout.drv failed View build log at '/var/log/guix/drvs/93/1x0yxqr800alglvc3gkqaqkfrgwh1b-bap-2.5.0-alpha-checkout.drv.gz'. guix build: error: build of `/gnu/store/931x0yxqr800alglvc3gkqaqkfrgwh1b-bap-2.5.0-alpha-checkout.drv' failedLet me check the other ones.
Cheers, simon
-
About
groovy, my guess is about.gitattributesand the normalization of CRLF. IIUC, Guix is applying this file before hashing and SWH do not. I do not know, we discussed a discrepency from that normalization inswh-develon March.Is that possible?
-
If the assumption about
groovyand CRLF is the correct one, I guess same cause leads to same effect withbc-java. The file.gitattributesreads:# Set default behaviour, in case users don't have core.autocrlf set. * text=autoEdited by zimoun -
About
FastQC, hum I do not know. I need to investigate... it is weird.Currently Guix stores in its source the hash
00y9drm0bkpxw8xfl8ysss18jmnhj8blgqgr6fpa58rkpfcbg8qkand this has not changed since February (Guix revisionb6a4fbb488f1f539ae45ed7924c9af8905fa0d8b).This hash matches the one in
sources.json.$ guix hash -S nar -f nix-base32 -H sha256 /gnu/store/5dyvjgylwiixhp8z9a08rnddh53h4cdx-fastqc-0.11.9-checkout 00y9drm0bkpxw8xfl8ysss18jmnhj8blgqgr6fpa58rkpfcbg8qk $ guix hash -S nar -f base64 -H sha256 /gnu/store/5dyvjgylwiixhp8z9a08rnddh53h4cdx-fastqc-0.11.9-checkout E6O3mLszo6KuM/nhRxeS0FaJgtbaI+o64v3OBWpuyQM=The weird part is about the option
-xbecause it is omitted. Hum?!$ git clone https://github.com/s-andrews/FastQC Cloning into 'FastQC'... remote: Enumerating objects: 901, done. remote: Counting objects: 100% (295/295), done. remote: Compressing objects: 100% (164/164), done. remote: Total 901 (delta 157), reused 187 (delta 71), pack-reused 606 Receiving objects: 100% (901/901), 11.23 MiB | 4.13 MiB/s, done. Resolving deltas: 100% (387/387), done. $ git -C FastQC checkout v0.11.9 Note: switching to 'v0.11.9'. $ guix hash -S nar -f base64 -H sha256 -x FastQC 6ZBuIfVO79U9iJGKEzoWQUuusk1dYmPb4MJo8yZI00s= $ mv FastQC/.git bye-bye $ guix hash -S nar -f base64 -H sha256 -x FastQC 6ZBuIfVO79U9iJGKEzoWQUuusk1dYmPb4MJo8yZI00s= $ guix hash -S nar -f base64 -H sha256 FastQC E6O3mLszo6KuM/nhRxeS0FaJgtbaI+o64v3OBWpuyQM=Hum, let me open a bug report on Guix side. :-)
-
About
pyxel, it seems a change of tag (mistake on Guix side or upstream retag). Arf, when people will decide to switch to intrinsic identifier for good. :-)HEAD is now at be75b72 Merge branch 'develop' r:sha256 hash mismatch for /gnu/store/m79d72fh3k4vypcqrsikrwikrscwqd6m-python-pyxel-1.4.3-checkout: expected hash: 0bwsgb5yq5s479cnf046v379zsn5ybp5195kbfvzr9l11qbaicm9 actual hash: 03ch79cmh9fxvq6c2f3zc2snzczhqi2n01f254lsigckc7d5wz08 hash mismatch for store item '/gnu/store/m79d72fh3k4vypcqrsikrwikrscwqd6m-python-pyxel-1.4.3-checkout' -
Summary:
| Package | SWH side | Guix side | |-----------+---------------+-------------| | bap | NotIssue | ToFix | | groovy | normalization | CRLF? | | bc-java | normalization | CRLF? | | glcontext | NotIssue | ToFix | | FastQC | ? | ?bug#65979? | | pyxel | NotIssue | ToFix |Guix side: bug#65979.
Let me know if there is other URLs that need investigations.
@ardumont @anlambert @civodul Therefore, to conclude, we have to discuss about the normalization and CRLF, no? :-)
-
Here the updated table:
| Package | SWH side | Guix side | |-----------+---------------+-------------| | bap | NotIssue | patch#66013 | | glcontext | NotIssue | patch#66013 | | pyxel | NotIssue | bug#66015 | | FastQC | ? | bug#65979 | | bc-java | normalization | CRLF? | | groovy | normalization | CRLF? |Tracked Guix side: patch#66013, bug#66015, bug#65979.
Thanks for spotting that out.
If you notice other discrepancies, let me know. :-)
-
Hi, an update:
| Package | SWH side | Guix side | |-----------+---------------+-------------| | bap | NotIssue | Fixed | | glcontext | NotIssue | Fixed | | pyxel | NotIssue | Fixed | | FastQC | ? | bug#65979 | | bc-java | normalization | CRLF? | | groovy | normalization | CRLF? |Tracked Guix side:
patch#66013,bug#66015, bug#65979.The file
sources.jsonwill be updated soon, the time that all propagate. -
Hi,
About
FastQC, the culprit is.svnfolders as explained in bug#65979. :-)$ ls -a1 FastQC | grep git .git .gitignore $ tree -ad -L 2 FastQC/Help FastQC/Help ├── 1 Introduction │ └── .svn ├── 2 Basic Operations │ └── .svn └── 3 Analysis Modules └── .svn 7 directoriesAnd there is a discrepancy between command-line option
-xinguix hashand what the fixed-output derivation computes. IIUC, only the top-level.git,.svn,.bzr, etc. should be excluded for matching the fixed-output derivation checksum hash. Well, let confirm on Guix side. :-)Cheers, simon
-
@ardumont, @zimoun I have submitted swh-loader-core!496 (merged) fixing nar hash computation on the SWH side for the
FastQClike cases.There is still a discrepancy between the nar hash currently computed by
guix hashbut this should also be fixed on the guix side if I understood the dicussion in https://issues.guix.gnu.org/65979#6 correctly ?$ guix hash -S nar -x -H sha256 -f base64 FastQC/ 6ZBuIfVO79U9iJGKEzoWQUuusk1dYmPb4MJo8yZI00s= -
Hi @anlambert
Cool!
swh-loader-core!496 (merged) looks good to me.
It reminds me a question and I have lost the answer. :-) Some packages use what Guix calls "
recursive? #true" asorigin, which concretely means Git submodules. And all the submodules are part of the hash.{ "type": "git", "git_url": "https://github.com/gnuradio/volk", "integrity": "sha256-kI4IuO6TLplo5lLAGIPWQWtePcjIEWB9XaJDA6WlqSg=", "outputHashAlgo": "sha256", "outputHashMode": "recursive", "git_ref": "v3.0.0" },# Without submodule $ git clone https://github.com/gnuradio/volk $ git -C volk checkout v3.0.0 $ guix hash -S nar -f base64 -H sha256 --exclude-vcs volk hKPucQ2Az+Yh8WzcY91N749N3eefSkRkKUSBGstY/TA=# With submodule $ git -C volk submodule init $ git -C volk submodule update Submodule path 'cpu_features': checked out '188d0d3c383689cdb6bb70dc6da2469faec84f61' $ guix hash -S nar -f base64 -H sha256 --exclude-vcs volk kI4IuO6TLplo5lLAGIPWQWtePcjIEWB9XaJDA6WlqSg=# Manual exclude-vcs $ mv volk/.git bye-bye $ mv volk/cpu_features/.git tchao-tchao $ guix hash -S nar -f base64 -H sha256 volk kI4IuO6TLplo5lLAGIPWQWtePcjIEWB9XaJDA6WlqSg=I am raising the case just to be sure we have discussed it. :-)
There is still a discrepancy between the nar hash currently computed by
guix hashYes but
guix hashis an helper and not used at all for producingsources.json. Therefore, it does not matter for the loader. Fixing it is a bonus. :-) Thanks for pointing the bug.Cheers, simon
-
I added a workaround in f9c18e78 to handle the submodules case.
Nevertheless, if it is possible from your side to add a new boolean field in the JSON file (
submodules?) indicating that submodules should be fetched to compute the hash, this will simplify the processing from our side. -
I added a workaround in f9c18e78 to handle the submodules case.
Thanks! Aah, now I remember. :-) Thank you for fixing this case I have initially missed.
Nevertheless, if it is possible from your side to add a new boolean field in the JSON file (
submodules?) indicating that submodules should be fetched to compute the hash, this will simplify the processing from our side.Yes, for sure. Tracked by which keyword?
submodule? Like that,"type": "git", "git_url": "https://github.com/gnuradio/volk", "integrity": "sha256-kI4IuO6TLplo5lLAGIPWQWtePcjIEWB9XaJDA6WlqSg=", "outputHashAlgo": "sha256", "outputHashMode": "recursive", "git_ref": "v3.0.0", "submodule": "true"? Or another way?
Do you want that this new keyword will be for all
"type": "git", mostly false, and true only for the cases using Git submodules?Or, do you want that this new keyword will be there only for the cases using Git submodules and absent else?
-
Do you want that this new keyword will be for all
"type": "git", mostly false, and true only for the cases using Git submodules?Or, do you want that this new keyword will be there only for the cases using Git submodules and absent else?
I think the
submodulekeyword could only be added when fetching submodules is required, this way the size of the JSON file will not grow too much. -
@anlambert About adding
submodulekeyword, done with #66491.It will be deployed next week.
-
@anlambert Patch #66491 is installed. So the new
sources.jsonsupportingsubmoduleshould be deployed at the end of the afternoon or tomorrow.Note that now,
https://guix.gnu.org/sources.jsonis compressed although the extension has not changed.Cheers, simon
-
@zimoun, I (finally) implemented the exploitation of the new
submoduleinfo insources.json: !173 (merged), swh-lister!517 (merged).
mentioned in commit anlambert/swh-loader-core@18c5cfb6