Skip to content
Snippets Groups Projects

Mercurial checkouts origins contain '.hg' directory

    • Closed Issue created by Ludovic Courtès @civodul

    Mercurial checkouts origins are ingested with the .hg folder. It's excluded from the nar hash computation but it remains ingested with its tree and it should not.

    Original issue:

    Hello!

The tarball at https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:218d95849f10fc0691d7dfa80999ce5061e654ef/raw/ contains a `.hg` metadata directory, which I think is unintended, right?

    Plan:

    ✓ 4 of 4 checklist items completed · Edited

    Designs

      Child items ...

      2. Activity

      • Antoine Lambert
        Antoine Lambert @anlambert ·
        Maintainer

        I confirm this is a bug on our side, we filtered out the .hg directory when computing the NAR hash but not on the archived directory, sigh ...

      • Antoine R. Dumont mentioned in issue #5222 (closed)

        mentioned in issue #5222 (closed)

      • Antoine Lambert mentioned in commit anlambert/swh-loader-core@abe53369

        mentioned in commit anlambert/swh-loader-core@abe53369

      • Antoine Lambert mentioned in merge request swh/devel/swh-loader-core!517 (merged)

        mentioned in merge request swh/devel/swh-loader-core!517 (merged)

      • Antoine Lambert mentioned in commit anlambert/swh-loader-mercurial@4aefe6e1

        mentioned in commit anlambert/swh-loader-mercurial@4aefe6e1

      • Antoine Lambert mentioned in merge request swh/devel/swh-loader-mercurial!144 (merged)

        mentioned in merge request swh/devel/swh-loader-mercurial!144 (merged)

      • Antoine Lambert mentioned in commit anlambert/swh-loader-mercurial@d1500a40

        mentioned in commit anlambert/swh-loader-mercurial@d1500a40

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Owner

        Will be solved once the next release #5261 (closed) is deployed.

      • Antoine R. Dumont mentioned in issue #5261 (closed)

        mentioned in issue #5261 (closed)

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Owner

        I've looked into the 16 origins of type hg-checkout in the archive [1]

        Only one has a remaining .hg directory [2]

        Note that this origin has also a visit type hg.

        Funnily enough, for that origin, I noticed that the hg visit type rendering tree listing does not show the .hg folder while the hg-checkout type rendering does...

        Ah but it seems to be the case for all hg visit type rendering tree ui, the .hg is not listed.

        @anlambert might be interested to know this either way ^

        [1] https://archive.softwareheritage.org/browse/search/?q=&with_visit=true&with_content=true&visit_type=hg-checkout

        [2] http://hg.openjdk.java.net/openjfx/8u-dev/rt

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Owner

        For that origin, its last visit ended up in a not_found [1]. The loader did not find anything. Hence why the api is still displaying the old '.hg' at its root folder. It's the previous successful visit's snapshot directory. (We were not able to revisit with the new deployed version yet).

        The main site seems to have trouble too [2]. So nothing we can do about it currently.

        [1] https://archive.softwareheritage.org/api/1/origin/http://hg.openjdk.java.net/openjfx/8u-dev/rt/visits/

            {
        "date": "2024-02-16T10:01:44.972263+00:00",
        "metadata": {},
        "origin": "http://hg.openjdk.java.net/openjfx/8u-dev/rt",
        "origin_visit_url": "https://archive.softwareheritage.org/api/1/origin/http://hg.openjdk.java.net/openjfx/8u-dev/rt/visit/5527/",
        "snapshot": null,
        "snapshot_url": null,
        "status": "not_found",
        "type": "hg-checkout",
        "visit": 5527
    },
    {
        "date": "2024-02-15T21:54:29.632829+00:00",
        "metadata": {},
        "origin": "http://hg.openjdk.java.net/openjfx/8u-dev/rt",
        "origin_visit_url": "https://archive.softwareheritage.org/api/1/origin/http://hg.openjdk.java.net/openjfx/8u-dev/rt/visit/5526/",
        "snapshot": "217fc524ad3a79c735d8c739aeaa2409bfdb9a32",
        "snapshot_url": "https://archive.softwareheritage.org/api/1/snapshot/217fc524ad3a79c735d8c739aeaa2409bfdb9a32/",
        "status": "full",
        "type": "hg-checkout",
        "visit": 5526
    },

        [2] https://hg.openjdk.org/openjfx/8u-dev/rt

        Edited by Antoine R. Dumont
      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Owner

        The site went up again. I've scheduled a visit:

        swh@swh-toolbox-55f7fcf794-m7rwf:~$ export SWH_CONFIG_FILENAME=/etc/swh/config-scheduler.yml
swh@swh-toolbox-55f7fcf794-m7rwf:~$ swh scheduler -C $SWH_CONFIG_FILENAME task add -p oneshot load-hg-checkout url=http://hg.openjdk.java.net/openjfx/8u-dev/rt ref=8u202-ga checksum_layout=nar checksums='{"sha256": "2e15215de59feb86687a3da9d02f8902ed38ba01d3249b36635cef787945e379"}'
Created 1 tasks

Task 416364570
  Next run: today (2024-02-26T16:55:58.037010+00:00)
  Interval: 1 day, 0:00:00
  Type: load-hg-checkout
  Policy: oneshot
  Args:
  Keyword args:
    checksum_layout: 'nar'
    checksums: {'sha256': '2e15215de59feb86687a3da9d02f8902ed38ba01d3249b36635cef787945e379'}

        @vsellier nvm about not being able to pass deep parameter to the cli, we actually can in the end. @anlambert proved me wrong (thx ;)

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Owner

        Well, nope, it could not go the all way... It just failed [1].

        Another one bites the dust [2]

        [2]

        [
    {
        "date": "2024-02-26T17:04:53.244287+00:00",
        "metadata": {},
        "origin": "http://hg.openjdk.java.net/openjfx/8u-dev/rt",
        "origin_visit_url": "https://archive.softwareheritage.org/api/1/origin/http://hg.openjdk.java.net/openjfx/8u-dev/rt/visit/5528/",
        "snapshot": null,
        "snapshot_url": null,
        "status": "not_found",
        "type": "hg-checkout",
        "visit": 5528
    },

        [1]

        swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]: {"asctime": "2024-02-26 16:56:39,000", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/loader.py", "lineno": 414, "funcName": "load", "task_name": null, "task_id": null, "name": "swh.loader.mercurial.directory.HgCheckoutLoader", "levelname": "INFO", "message": "Load origin 'http://hg.openjdk.java.net/openjfx/8u-dev/rt' with type 'hg-checkout'"}
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]: real URL is https://hg.openjdk.org/openjfx/8u-dev/rt
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]: worker: Warm shutdown (MainProcess)
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]: worker: Warm shutdown (MainProcess)
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]: adding changesets
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]: transaction abort!
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]: rollback completed
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]: Process Process-1:1:
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]: Traceback (most recent call last):
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/billiard/process.py", line 323, in _bootstrap
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     self.run()
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/billiard/process.py", line 110, in run
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     self._target(*self._args, **self._kwargs)
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/utils.py", line 76, in _clone_task
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     raise e
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/utils.py", line 71, in _clone_task
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     clone_func()
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/hg.py", line 1016, in clone
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     exchange.pull(
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/exchange.py", line 1713, in pull
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     _fullpullbundle2(repo, pullop)
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/exchange.py", line 1564, in _fullpullbundle2
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     _pullbundle2(pullop)
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/exchange.py", line 1923, in _pullbundle2
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     bundle2.processbundle(
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py", line 508, in processbundle
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     processparts(repo, op, unbundler)
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py", line 514, in processparts
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     with partiterator(repo, op, unbundler) as parts:
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py", line 457, in __exit__
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     raise exc
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py", line 516, in processparts
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     _processpart(op, part)
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py", line 594, in _processpart
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     handler(op, part)
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py", line 2092, in handlechangegroup
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     ret = _processchangegroup(
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py", line 524, in _processchangegroup
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     ret = cg.apply(op.repo, tr, source, url, **kwargs)
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/changegroup.py", line 547, in apply
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     if not cl.addgroup(
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/revlog.py", line 3561, in addgroup
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     for data in deltas:
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/changegroup.py", line 779, in deltaiter
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     for chunkdata in iter(lambda: self.deltachunk(chain), {}):
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/changegroup.py", line 779, in <lambda>
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     for chunkdata in iter(lambda: self.deltachunk(chain), {}):
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/changegroup.py", line 365, in deltachunk
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     delta = readexactly(self._stream, l - self.deltaheadersize)
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/util.py", line 3174, in readexactly
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     s = stream.read(n)
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py", line 1481, in read
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     data = self._payloadstream.read(size)
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/util.py", line 2713, in read
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     for chunk in self.iter:
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/util.py", line 2684, in splitbig
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     for chunk in chunks:
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:   File "/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py", line 1342, in decodepayloadchunks
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]:     raise error.Abort(
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]: mercurial.error.Abort: stream ended unexpectedly  (got 29127 bytes, expected 32768)
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]: {"asctime": "2024-02-26 17:04:53,240", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/loader.py", "lineno": 513, "funcName": "load", "task_name": null, "task_id": null, "name": "swh.loader.mercurial.directory.HgCheckoutLoader", "levelname": "ERROR", "message": "Loading failure, updating to `not_found` status", "exc_info": "Traceback (most recent call last):\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/loader/mercurial/utils.py\", line 40, in raise_not_found_repository\n    yield\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/loader/mercurial/directory.py\", line 40, in clone_repository\n    clone(repo_url, str(local_clone_dir), rev=hg_changeset)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/loader/mercurial/hgutil.py\", line 132, in clone\n    clone_with_timeout(src, dest, closure, timeout)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/utils.py\", line 112, in clone_with_timeout\n    raise CloneFailure(src, dest, errors.get())\nswh.loader.core.utils.CloneFailure: ('http://hg.openjdk.java.net/openjfx/8u-dev/rt', '/tmp/tmpmo7w5w0_-2024-02-26T16:56:39.393668/rt', 'Traceback (most recent call last):\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/utils.py\", line 71, in _clone_task\\n    clone_func()\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/hg.py\", line 1016, in clone\\n    exchange.pull(\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/exchange.py\", line 1713, in pull\\n    _fullpullbundle2(repo, pullop)\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/exchange.py\", line 1564, in _fullpullbundle2\\n    _pullbundle2(pullop)\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/exchange.py\", line 1923, in _pullbundle2\\n    bundle2.processbundle(\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py\", line 508, in processbundle\\n    processparts(repo, op, unbundler)\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py\", line 514, in processparts\\n    with partiterator(repo, op, unbundler) as parts:\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py\", line 457, in __exit__\\n    raise exc\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py\", line 516, in processparts\\n    _processpart(op, part)\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py\", line 594, in _processpart\\n    handler(op, part)\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py\", line 2092, in handlechangegroup\\n    ret = _processchangegroup(\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py\", line 524, in _processchangegroup\\n    ret = cg.apply(op.repo, tr, source, url, **kwargs)\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/changegroup.py\", line 547, in apply\\n    if not cl.addgroup(\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/revlog.py\", line 3561, in addgroup\\n    for data in deltas:\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/changegroup.py\", line 779, in deltaiter\\n    for chunkdata in iter(lambda: self.deltachunk(chain), {}):\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/changegroup.py\", line 779, in <lambda>\\n    for chunkdata in iter(lambda: self.deltachunk(chain), {}):\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/changegroup.py\", line 365, in deltachunk\\n    delta = readexactly(self._stream, l - self.deltaheadersize)\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/util.py\", line 3174, in readexactly\\n    s = stream.read(n)\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py\", line 1481, in read\\n    data = self._payloadstream.read(size)\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/util.py\", line 2713, in read\\n    for chunk in self.iter:\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/util.py\", line 2684, in splitbig\\n    for chunk in chunks:\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py\", line 1342, in decodepayloadchunks\\n    raise error.Abort(\\nmercurial.error.Abort: stream ended unexpectedly  (got 29127 bytes, expected 32768)\\n')\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/loader.py\", line 451, in load\n    more_data_to_fetch = self.fetch_data()\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/loader.py\", line 799, in fetch_data\n    for artifact_path in self.fetch_artifact():\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/loader/mercurial/directory.py\", line 81, in fetch_artifact\n    repo = clone_repository(\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/loader/mercurial/directory.py\", line 39, in clone_repository\n    with raise_not_found_repository():\n  File \"/usr/local/lib/python3.10/contextlib.py\", line 153, in __exit__\n    self.gen.throw(typ, value, traceback)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/loader/mercurial/utils.py\", line 42, in raise_not_found_repository\n    raise NotFound(e)\nswh.loader.exception.NotFound: ('http://hg.openjdk.java.net/openjfx/8u-dev/rt', '/tmp/tmpmo7w5w0_-2024-02-26T16:56:39.393668/rt', 'Traceback (most recent call last):\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/utils.py\", line 71, in _clone_task\\n    clone_func()\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/hg.py\", line 1016, in clone\\n    exchange.pull(\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/exchange.py\", line 1713, in pull\\n    _fullpullbundle2(repo, pullop)\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/exchange.py\", line 1564, in _fullpullbundle2\\n    _pullbundle2(pullop)\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/exchange.py\", line 1923, in _pullbundle2\\n    bundle2.processbundle(\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py\", line 508, in processbundle\\n    processparts(repo, op, unbundler)\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py\", line 514, in processparts\\n    with partiterator(repo, op, unbundler) as parts:\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py\", line 457, in __exit__\\n    raise exc\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py\", line 516, in processparts\\n    _processpart(op, part)\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py\", line 594, in _processpart\\n    handler(op, part)\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py\", line 2092, in handlechangegroup\\n    ret = _processchangegroup(\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py\", line 524, in _processchangegroup\\n    ret = cg.apply(op.repo, tr, source, url, **kwargs)\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/changegroup.py\", line 547, in apply\\n    if not cl.addgroup(\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/revlog.py\", line 3561, in addgroup\\n    for data in deltas:\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/changegroup.py\", line 779, in deltaiter\\n    for chunkdata in iter(lambda: self.deltachunk(chain), {}):\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/changegroup.py\", line 779, in <lambda>\\n    for chunkdata in iter(lambda: self.deltachunk(chain), {}):\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/changegroup.py\", line 365, in deltachunk\\n    delta = readexactly(self._stream, l - self.deltaheadersize)\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/util.py\", line 3174, in readexactly\\n    s = stream.read(n)\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py\", line 1481, in read\\n    data = self._payloadstream.read(size)\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/util.py\", line 2713, in read\\n    for chunk in self.iter:\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/util.py\", line 2684, in splitbig\\n    for chunk in chunks:\\n  File \"/opt/swh/.local/lib/python3.10/site-packages/mercurial/bundle2.py\", line 1342, in decodepayloadchunks\\n    raise error.Abort(\\nmercurial.error.Abort: stream ended unexpectedly  (got 29127 bytes, expected 32768)\\n')", "swh_task_args": [], "swh_task_kwargs": {"origin": "http://hg.openjdk.java.net/openjfx/8u-dev/rt", "lister_name": null, "lister_instance_name": null}}
swh/loader-hg-checkout-5cc5cb874-6fwns[loaders]: {"asctime": "2024-02-26 17:04:53,310", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/app/trace.py", "lineno": 131, "funcName": "info", "task_name": null, "task_id": null, "name": "celery.app.trace", "levelname": "INFO", "message": "Task swh.loader.mercurial.tasks.LoadMercurialCheckout[269fab60-7404-4c80-a3d0-04b937fdebd2] succeeded in 494.68754841946065s: {'status': 'uneventful'}", "data": {"id": "269fab60-7404-4c80-a3d0-04b937fdebd2", "name": "swh.loader.mercurial.tasks.LoadMercurialCheckout", "return_value": "{'status': 'uneventful'}", "runtime": 494.68754841946065, "args": "()", "kwargs": "{'ref': '8u202-ga', 'url': 'http://hg.openjdk.java.net/openjfx/8u-dev/rt', 'checksums': {'sha256': '2e15215de59feb86687a3da9d02f8902ed38ba01d3249b36635cef787945e379'}, 'checksum_layout': 'nar'}"}}
        Edited by Antoine R. Dumont
        • Antoine R. Dumont
          Antoine R. Dumont @ardumont ·
          Owner

          It got scheduled again and it was successful this time [1]

          Browsing the origin is now ok, no more .hg in the tree listing.

          [2] https://archive.softwareheritage.org/browse/origin/directory/?origin_url=http://hg.openjdk.java.net/openjfx/8u-dev/rt&visit_type=hg-checkout

          [1]

          loaders {"asctime": "2024-02-27 14:49:36,991", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/loader.py", "lineno": 414
, "funcName": "load", "task_name": null, "task_id": null, "name": "swh.loader.mercurial.directory.HgCheckoutLoader", "levelname": "INFO", "message": "Load origin 'http://hg.o
penjdk.java.net/openjfx/8u-dev/rt' with type 'hg-checkout'"}
loaders real URL is https://hg.openjdk.org/openjfx/8u-dev/rt
loaders adding changesets
loaders adding manifests
loaders adding file changes
loaders added 10276 changesets with 154004 changes to 61959 files
loaders new changesets 3aa1b5351c26:85d09981ae0d
loaders updating to branch default
loaders 22058 files updated, 0 files merged, 0 files removed, 0 files unresolved
loaders {"asctime": "2024-02-27 14:51:28,061", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/app/trace.py", "lineno": 131, "fun
cName": "info", "task_name": null, "task_id": null, "name": "celery.app.trace", "levelname": "INFO", "message": "Task swh.loader.mercurial.tasks.LoadMercurialCheckout[55b7642
0-9280-4e3e-9836-2123a0477f9f] succeeded in 111.52736099157482s: {'status': 'eventful'}", "data": {"id": "55b76420-9280-4e3e-9836-2123a0477f9f", "name": "swh.loader.mercurial
.tasks.LoadMercurialCheckout", "return_value": "{'status': 'eventful'}", "runtime": 111.52736099157482, "args": "()", "kwargs": "{'ref': '8u202-ga', 'url': 'http://hg.openjdk
.java.net/openjfx/8u-dev/rt', 'checksums': {'sha256': '2e15215de59feb86687a3da9d02f8902ed38ba01d3249b36635cef787945e379'}, 'checksum_layout': 'nar'}"}}
        • Ludovic Courtès
          Ludovic Courtès @civodul ·
          Author

          Yay, thanks @ardumont! :tada:

          The one at https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:218d95849f10fc0691d7dfa80999ce5061e654ef/raw/ (from the original report) still has .hg but maybe that'll "fix itself" eventually?

        • Antoine R. Dumont
          Antoine R. Dumont @ardumont ·
          Owner

          Hello,

          What we did is fix the tree in the archive (with the next visit, the snapshot now targets a proper representation of the tree). That vault entry, since it's targetting a swhid, must be invalidated then cooked again. I'll see what i can do.

          Thanks for the heads up, I missed it.

        • Antoine R. Dumont
          Antoine R. Dumont @ardumont ·
          Owner

          #5274 (closed)

        • Antoine R. Dumont
          Antoine R. Dumont @ardumont ·
          Owner

          @civodul I was wrong, the issue I've just opened is invalid.

          This swhid cannot change, and, by transitivity, neither can its cooked representation.

          I've looked for the associated origin of the swhid you provided [1] [2]

          The new visit snapshots for that origin target the swhid directories [3] (from the release with the fix on mercurial checkout loader).

          As mentioned [5], we now need to look for extid mapping with extid_version 1.

          The old mapping maps the incorrect swhid (the one you mentioned).

          The new ones (with extid_version 1) are ok.

          [5] #5222 (comment 165337)

          [4] This is an old mapping (extid_version 0)

          2024-03-05 09:41:00 softwareheritage@albertina:5432 λ select * from extid where target='\x218d95849f10fc0691d7dfa80999ce5061e654ef' and extid_type in ('nar-sha256', 'checksum-sha256');
+------------+--------------------------------------------------------------------+-------------+--------------------------------------------+---------------+
| extid_type |                               extid                                | target_type |                   target                   | extid_version |
+------------+--------------------------------------------------------------------+-------------+--------------------------------------------+---------------+
| nar-sha256 | \xc98bd6991721d60b9a79600428bfbe8db0aaac3d383cd2df803ed7867c7cb63b | directory   | \x218d95849f10fc0691d7dfa80999ce5061e654ef |             0 |
+------------+--------------------------------------------------------------------+-------------+--------------------------------------------+---------------+
(1 row)

Time: 31.718 ms

          [5] This matches the v1.0.11, v1.0.7 versions of that origin (2 versions in the current guix manifest, see below [6])

          2024-03-05 10:15:26 softwareheritage@albertina:5432 λ select * from extid where target in ('\x2e6b9f2aeaef28607b57a348c90ca6438d5f57ed', '\x1f35a460673e2c8ccf948b8c18cb8c18d2ec24fe') and extid_type in ('nar-sha256');
+------------+--------------------------------------------------------------------+-------------+--------------------------------------------+---------------+
| extid_type |                               extid                                | target_type |                   target                   | extid_version |
+------------+--------------------------------------------------------------------+-------------+--------------------------------------------+---------------+
| nar-sha256 | \x27685e5c826179fd9e82b0546b0a7ae6bf151a4861ee59849f3caa0040e95cf0 | directory   | \x2e6b9f2aeaef28607b57a348c90ca6438d5f57ed |             1 |
| nar-sha256 | \xc98bd6991721d60b9a79600428bfbe8db0aaac3d383cd2df803ed7867c7cb63b | directory   | \x1f35a460673e2c8ccf948b8c18cb8c18d2ec24fe |             1 |
+------------+--------------------------------------------------------------------+-------------+--------------------------------------------+---------------+
(2 rows)

Time: 33.079 ms

          [1] https://hg.sr.ht/~arnebab/wisp

          [2] swh:1:dir:2e6b9f2aeaef28607b57a348c90ca6438d5f57ed - swh:1:dir:1f35a460673e2c8ccf948b8c18cb8c18d2ec24fe

          [6]

          In [1]: url='https://guix.gnu.org/sources.json'

In [3]: import requests

In [4]: import json

In [5]: guix_data=requests.get(url).json()['sources']

In [6]: [d for d in guix_data if d['type'] == 'hg' and 'https://hg.sr.ht/~arnebab/wisp' == d['hg_url']]
Out[6]:
[{'type': 'hg',
  'hg_url': 'https://hg.sr.ht/~arnebab/wisp',
  'integrity': 'sha256-J2heXIJhef2egrBUawp65r8VGkhh7lmEnzyqAEDpXPA=',
  'outputHashAlgo': 'sha256',
  'outputHashMode': 'recursive',
  'hg_changeset': 'v1.0.11'},
 {'type': 'hg',
  'hg_url': 'https://hg.sr.ht/~arnebab/wisp',
  'integrity': 'sha256-yYvWmRch1guaeWAEKL++jbCqrD04PNLfgD7Xhnx8tjs=',
  'outputHashAlgo': 'sha256',
  'outputHashMode': 'recursive',
  'hg_changeset': 'v1.0.7'}]

In [7]: import click
   ...: import base64
   ...:
   ...:
   ...: def encode_b64(checksum: str):
   ...:     return base64.encodebytes(checksum.encode()).decode()
   ...:
   ...:
   ...: def decode_b64(integrity: str):
   ...:   checksum_algo, chksum_b64 = integrity.split("-")
   ...:   return checksum_algo, base64.decodebytes(chksum_b64.encode()).hex()
   ...:
   ...:

In [8]: decode_b64('sha256-J2heXIJhef2egrBUawp65r8VGkhh7lmEnzyqAEDpXPA=')
Out[8]: ('sha256', '27685e5c826179fd9e82b0546b0a7ae6bf151a4861ee59849f3caa0040e95cf0')

In [9]: decode_b64('sha256-yYvWmRch1guaeWAEKL++jbCqrD04PNLfgD7Xhnx8tjs=')
Out[9]: ('sha256', 'c98bd6991721d60b9a79600428bfbe8db0aaac3d383cd2df803ed7867c7cb63b')
          Edited by Antoine R. Dumont
        • Please register or sign in to reply
      • Antoine R. Dumont closed

        closed

      • Antoine R. Dumont changed title from Mercurial checkouts obtained from the Vault contain '.hg' directory to Mercurial checkouts origins contain '.hg' directory

        changed title from Mercurial checkouts obtained from the Vault contain '.hg' directory to Mercurial checkouts origins contain '.hg' directory

      • Antoine R. Dumont changed the description

        changed the description

      • Antoine R. Dumont mentioned in issue #5150 (closed)

        mentioned in issue #5150 (closed)

      • Antoine R. Dumont marked this issue as related to #5274 (closed)

        marked this issue as related to #5274 (closed)

      • Antoine R. Dumont mentioned in issue #5274 (closed)

        mentioned in issue #5274 (closed)

      • Antoine Lambert mentioned in commit anlambert/swh-loader-core@c45625f6

        mentioned in commit anlambert/swh-loader-core@c45625f6

      Please register or sign in to reply