It seems that the cpio files generated by rpm2cpio now have filenames
with a leading `./`. Add support for this change in the test fixture (it
seems that the actual extraction isn't affected by this change).

Bump development tools: mypy, codespell, isort, ...

Move all tools configuration in pyproject.toml.

Remove no longer needed mypy overrides.

If an error occurs when attempting to fetch and parse latest version
info, return the latest version in versions list instead.

Antoine Lambert authored 2 months ago and Antoine Lambert committed 2 months ago

A package manager can reference a package version that is no longer
available for download so ensure produced snapshot HEAD alias targets
a fetched package release.

When the deposit loader downloads a deposited tarball, it calls the raw
endpoint of deposit private HTTP API. In production, the implementation
of that endpoint downloads a set of tarballs from an azure blob storage
and return an aggregated version of them.

It has been observed some HTTP read timeouts while downloading tarballs
from azure. Those have been fixed in swh-deposit!453 by increasing
the read timeout used to read data from azure.

However, there is still some deposit loader tasks that end up with error
as the read timeout used to query the raw endpoint of deposit private API
is twice lesser than the read timeout used to download from azure.
So ensure those HTTP read timeouts have the same value of 120 seconds.

Instead of implementing the versions sorting in each package loader
prefer  to have a base implementation in swh.loader.package.PackageLoader
class through the get_sorted_versions method. It relies on the looseversion
module enabling to interact with heterogeneous version schemes which works
pretty well with a large majority of package loaders.

The get_default_version method of the PackageLoader class now also has a
base implementation returning the last element from the list returned by
the get_sorted_versions method. As a consequence, each snapshot produced
by a package loader contains a HEAD alias branch targeting the branch
for the highest version number of a package.

Both methods can be reimplemented in package loaders for special cases
like debian for instance.

Also remove the use of the packaging module to parse versions as it is
only dedicated to parse Python package versions.

Related to swh-lister#4711.

Franck Bret authored 2 years ago and Antoine Lambert committed 6 months ago

Add incremental support based on sha256 EXTID.

Use looseversion.LooseVersion2 to parse crate versions.

Manage release date for each versions of a package.

Fetch extrinsic metadata for a crate version from crates Web API.

Adapt test dataset and add incremental test cases.

Related to swh/meta#4104

This includes the string representation of the actual exception -- as error
message -- that prevented the loading to properly happen so that there is a
chance to show a comprehensive error to the Save-Code-Now end user.

Related to swh-web#4805

Latest tenacity release adds some internal changes that broke the
mocking of sleep calls in tests.

Fix it by directly mocking time.sleep (was not working previously).

SWH data model allows an origin to have multiple visit types so we must
ensure to retrieve the latest snapshot for the same visit type performed
by a loader.

Related to swh/meta#5092.

While the package version was properly uppercase encoded in the URL
for fetching package version info, it was not in the download URL
resulting in 404 and partial visit of the golang origin.

The oldest part of the scheduler API was updated to use model classes
(based on attr package) instead of dictionaries in order to improve
typing.

pkg_resources is now deprecated (as of setuptools 68) so use importlib
instead.

This is needed to make swh.loader.core not depend on swh.loader.package.

This used to contain the __version__ attribute as well as the
DEFAULT_PARAMS dict. These 2 have been moved to
swh/loader/core/__init__.py

The PackageLoader.get_loader_version() method has been fixed
accordingly.

Rather than having it defined in the conftest.py file, since it is not
recommended to import conftest directly.

This should help identify these classes full path independently from how their
module is loaded (eg. have a consistent logger name etc.), which can
make tests fail for no valid reason.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

Add an extra check in function fetch_extids_from_checksums to ensure
a NAR hash extid matches the NAR hash of the targeted archived object.

Related to swh/infra/sysadm-environment#5256.

Currently, those impacted loaders are only used through the nixguix stack but they are
not specific to nixguix.

Refs. #4749

It has been decommissionned in production and replaced by a lister and various
listers (as per the README).

Refs. swh/meta#3781

This now matches the tarball loader behavior (top-level directory included
[1]). This also matches what's expected by the guix dataset.

As the nix hashes computed are done from the first directory included in the
tarball though, we must also provide that directory. That way, the hashes
checks done during ingestion can match appropriately. That was the initial
implementation.

In terms of data, as this will change the visit snapshot and the extid
mappings, the core loaders (NodeLoader, ...) now declares an extid_version
bumped to 1 (it was 0 by default). Which means that all extid mappings will be
recomputed.

[1] https://gitlab.softwareheritage.org/swh/devel/swh-loader-core/-/blob/master/swh/loader/package/loader.py?ref_type=heads#L829-837

Refs. swh/infra/sysadm-environment#5222

Related to swh/meta#5075.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

This is required after the changes introduced in 54d8c7df and it is also
more consistent with real data provided by nix or guix.

Moreover, this loader should be removed soon as a better lister have been
implemented for nixguix that no longer create loading tasks with nixguix
visit type.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

It exist cases where sha256 checksum for a source package is missing,
typically for legacy debian releases.

So ensure to not return a null extid in DebianPackageInfo class by
using sha1 or md5sum checksum instead.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

It might exist cases where multiple versions of a package target the same
release object.

For instance a rpm package has one specific version for each distribution
release but they can target the same intrinsic version and source package
contents are exactly the same.

So avoid downloading and processing a package version if the corresponding
extid has already been encountered during the current loading by maintaining
a mapping between extids and release swhids.

Antoine R. Dumont authored 1 year ago and Nicolas Dandrimont committed 1 year ago

This way also avoids checking the mountpoint permissions or the file owner, which
currently results in nar mismatch on (nixguix) origins with executables (for various
loaders e.g. tarball, git-checkout, ...) in our production setup.

Refs. swh/infra/sysadm-environment#5230

The referenced requests_mock bug has been fixed upstream

Retrying 5 times with an exponential backoff could turn asleep those
loaders for more than 2O minutes so reduce the number of retries to 3
to make the loaders sleep for only a couple of seconds instead.