Franck Bret authored 2 years ago and Antoine Lambert committed 6 months ago

Add incremental support based on sha256 EXTID.

Use looseversion.LooseVersion2 to parse crate versions.

Manage release date for each versions of a package.

Fetch extrinsic metadata for a crate version from crates Web API.

Adapt test dataset and add incremental test cases.

Related to swh/meta#4104

This includes the string representation of the actual exception -- as error
message -- that prevented the loading to properly happen so that there is a
chance to show a comprehensive error to the Save-Code-Now end user.

Related to swh-web#4805

Latest tenacity release adds some internal changes that broke the
mocking of sleep calls in tests.

Fix it by directly mocking time.sleep (was not working previously).

SWH data model allows an origin to have multiple visit types so we must
ensure to retrieve the latest snapshot for the same visit type performed
by a loader.

Related to swh/meta#5092.

While the package version was properly uppercase encoded in the URL
for fetching package version info, it was not in the download URL
resulting in 404 and partial visit of the golang origin.

The oldest part of the scheduler API was updated to use model classes
(based on attr package) instead of dictionaries in order to improve
typing.

pkg_resources is now deprecated (as of setuptools 68) so use importlib
instead.

This is needed to make swh.loader.core not depend on swh.loader.package.

This used to contain the __version__ attribute as well as the
DEFAULT_PARAMS dict. These 2 have been moved to
swh/loader/core/__init__.py

The PackageLoader.get_loader_version() method has been fixed
accordingly.

Rather than having it defined in the conftest.py file, since it is not
recommended to import conftest directly.

This should help identify these classes full path independently from how their
module is loaded (eg. have a consistent logger name etc.), which can
make tests fail for no valid reason.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

Add an extra check in function fetch_extids_from_checksums to ensure
a NAR hash extid matches the NAR hash of the targeted archived object.

Related to swh/infra/sysadm-environment#5256.

Currently, those impacted loaders are only used through the nixguix stack but they are
not specific to nixguix.

Refs. swh/devel/swh-loader-core#4749

It has been decommissionned in production and replaced by a lister and various
listers (as per the README).

Refs. swh/meta#3781

This now matches the tarball loader behavior (top-level directory included
[1]). This also matches what's expected by the guix dataset.

As the nix hashes computed are done from the first directory included in the
tarball though, we must also provide that directory. That way, the hashes
checks done during ingestion can match appropriately. That was the initial
implementation.

In terms of data, as this will change the visit snapshot and the extid
mappings, the core loaders (NodeLoader, ...) now declares an extid_version
bumped to 1 (it was 0 by default). Which means that all extid mappings will be
recomputed.

[1] https://gitlab.softwareheritage.org/swh/devel/swh-loader-core/-/blob/master/swh/loader/package/loader.py?ref_type=heads#L829-837

Refs. swh/infra/sysadm-environment#5222

Related to swh/meta#5075.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

This is required after the changes introduced in 54d8c7df and it is also
more consistent with real data provided by nix or guix.

Moreover, this loader should be removed soon as a better lister have been
implemented for nixguix that no longer create loading tasks with nixguix
visit type.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

It exist cases where sha256 checksum for a source package is missing,
typically for legacy debian releases.

So ensure to not return a null extid in DebianPackageInfo class by
using sha1 or md5sum checksum instead.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

It might exist cases where multiple versions of a package target the same
release object.

For instance a rpm package has one specific version for each distribution
release but they can target the same intrinsic version and source package
contents are exactly the same.

So avoid downloading and processing a package version if the corresponding
extid has already been encountered during the current loading by maintaining
a mapping between extids and release swhids.

Antoine R. Dumont authored 1 year ago and Nicolas Dandrimont committed 1 year ago

This way also avoids checking the mountpoint permissions or the file owner, which
currently results in nar mismatch on (nixguix) origins with executables (for various
loaders e.g. tarball, git-checkout, ...) in our production setup.

Refs. swh/infra/sysadm-environment#5230

The referenced requests_mock bug has been fixed upstream

Retrying 5 times with an exponential backoff could turn asleep those
loaders for more than 2O minutes so reduce the number of retries to 3
to make the loaders sleep for only a couple of seconds instead.

Where the usage of async got dropped from the discovery protocol.

requests package set the value of the Accept-Encoding HTTP header to
"gzip, deflate" by default and some servers (https://download.ocamlcore.org/
for instance) will then send a compressed version of the artifact to download
with response header content-encoding usually set to gzip.

Nevertheless, this conflicts with the code checking if the response bytes
should be uncompressed as it should not when Content-Encoding header is equal
to gzip and Content-Type is equal to application/*gzip.

As artifacts to download are usually tarballs already compressed with gzip,
set the Accept-Encoding request header to identity in order to force the
server to send raw artifact bytes without the Content-Encoding header set.

It seems a better naming as shorter is better.

The NodeLoader class handles two checksums layout:

- standard: checksum is computed from the raw downloaded artifact bytes
- nar: checksum is a NAR hash, recursively computed from source code tree

Previously, only nar checksums were stored as ExtIDs while standard ones
were only used for integrity checks after downloads.

As mapping a tarball (resp. file) standard checksum to its corresponding
directory (resp. content) SWHID is of interest for Guix to check if SWH
archived this software artifact, ensure to also save standard checksums
as ExtIDs of type "checksum-<hash_algo>".

Commit 424540f1 broke the CLI launch of VCS loaders as those did not get
their swh.workers entrpoints renamed.

So revert it and restore previous swh.workers entrypoints names instead.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

Those located in swh/loader/tests were no longer collected.