TarballDirectoryLoader: Ingest tarball like package loaders do

This now matches the tarball loader behavior (top-level directory included [1]). This also matches what's expected by the guix dataset.

As the nix hashes computed are done from the first directory included in the tarball though, we must also provide that directory. That way, the hashes checks done during ingestion can match appropriately. That was the initial implementation.

In terms of data, as this will change the visit snapshot and the extid mappings, the core loaders (NodeLoader, ...) now declares an extid_version bumped to 1 (it was 0 by default). Which means that all extid mappings will be recomputed.

[1] https://gitlab.softwareheritage.org/swh/devel/swh-loader-core/-/blob/master/swh/loader/package/loader.py?ref_type=heads#L829-837

Refs. swh/infra/sysadm-environment#5222 (closed)

mentioned in issue swh/infra/sysadm-environment#5222 (closed)

Jenkins job DLDBASE/gitlab-builds #273 failed .
See Console Output and Coverage Report for more details.

In terms of ui browsing, this won't change a thing since the webapp already has heuristic to naturally skip the unique top-level directory.

There is no such processing implemented in the webapp (see https://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://pypi.org/project/your-win/ for instance) but indeed it would be nice to have.

lol, i was convinced you implemented it for the deposit archives or something ;)

added 1 commit

• 00cc2dd8 - TarballDirectoryLoader: Ingest tarball as package loaders do

Compare with previous version

Jenkins job DLDBASE/gitlab-builds #274 failed .
See Console Output and Coverage Report for more details.

added 1 commit

• f09df8b6 - TarballDirectoryLoader: Ingest tarball as package loaders do

Compare with previous version

changed the description

Jenkins job DLDBASE/gitlab-builds #275 failed .
See Console Output and Coverage Report for more details.

added 1 commit

• de890c8d - TarballDirectoryLoader: Ingest tarball as package loaders do

Compare with previous version

Jenkins job DLDBASE/gitlab-builds #276 failed .
See Console Output and Coverage Report for more details.

marked this merge request as draft

added 1 commit

• 590db4b1 - TarballDirectoryLoader: Adapt tarball ingestions

Compare with previous version

Jenkins job DLDBASE/gitlab-builds #277 succeeded .
See Console Output and Coverage Report for more details.

I forgot to mention that you should also add a test when a NAR hash computed without the top level directory is expected.

added 1 commit

• 8e1e9213 - Apply 1 suggestion(s) to 1 file(s)

Compare with previous version