It exist cases when a package version is referenced in package versions
list but package info for that version is missing.

This type of error is not fixable from the SWH side so there is no need
to report these errors to sentry as it generates a lot of noise in the
sentry dashboards.

Fixes #4751.

Make it in sync with current state of the swh stack.

The visit is expected to be kept as an attribute of the 'run' cli
command, so make it be the case for package loaders as well.

This probably should be properly handled at interface level as well...

It was previously installed through a transient requirement that
removed its use so it must be explicitly listed in the requirements.

Antoine Lambert authored 1 month ago and Antoine Lambert committed 3 weeks ago

This was used at the time we were building debian packages for
swh components but we no longer do that.

Antoine Lambert authored 1 month ago and Antoine Lambert committed 3 weeks ago

Instead of using the deposit API raw private endpoint to get
aggregated version of tarballs uploaded with a deposit, let the
loader handle this possibly costly operation by using the new
deposit API upload-urls private endpoint returning URLs for
downloading uploaded tarballs.

Fixes swh-deposit#4657.

It seems that the cpio files generated by rpm2cpio now have filenames
with a leading `./`. Add support for this change in the test fixture (it
seems that the actual extraction isn't affected by this change).

Bump development tools: mypy, codespell, isort, ...

Move all tools configuration in pyproject.toml.

Remove no longer needed mypy overrides.

If an error occurs when attempting to fetch and parse latest version
info, return the latest version in versions list instead.

Antoine Lambert authored 2 months ago and Antoine Lambert committed 2 months ago

A package manager can reference a package version that is no longer
available for download so ensure produced snapshot HEAD alias targets
a fetched package release.

When the deposit loader downloads a deposited tarball, it calls the raw
endpoint of deposit private HTTP API. In production, the implementation
of that endpoint downloads a set of tarballs from an azure blob storage
and return an aggregated version of them.

It has been observed some HTTP read timeouts while downloading tarballs
from azure. Those have been fixed in swh-deposit!453 by increasing
the read timeout used to read data from azure.

However, there is still some deposit loader tasks that end up with error
as the read timeout used to query the raw endpoint of deposit private API
is twice lesser than the read timeout used to download from azure.
So ensure those HTTP read timeouts have the same value of 120 seconds.

Instead of implementing the versions sorting in each package loader
prefer  to have a base implementation in swh.loader.package.PackageLoader
class through the get_sorted_versions method. It relies on the looseversion
module enabling to interact with heterogeneous version schemes which works
pretty well with a large majority of package loaders.

The get_default_version method of the PackageLoader class now also has a
base implementation returning the last element from the list returned by
the get_sorted_versions method. As a consequence, each snapshot produced
by a package loader contains a HEAD alias branch targeting the branch
for the highest version number of a package.

Both methods can be reimplemented in package loaders for special cases
like debian for instance.

Also remove the use of the packaging module to parse versions as it is
only dedicated to parse Python package versions.

Related to swh-lister#4711.

Franck Bret authored 2 years ago and Antoine Lambert committed 6 months ago

Add incremental support based on sha256 EXTID.

Use looseversion.LooseVersion2 to parse crate versions.

Manage release date for each versions of a package.

Fetch extrinsic metadata for a crate version from crates Web API.

Adapt test dataset and add incremental test cases.

Related to swh/meta#4104

This includes the string representation of the actual exception -- as error
message -- that prevented the loading to properly happen so that there is a
chance to show a comprehensive error to the Save-Code-Now end user.

Related to swh-web#4805

Latest tenacity release adds some internal changes that broke the
mocking of sleep calls in tests.

Fix it by directly mocking time.sleep (was not working previously).

SWH data model allows an origin to have multiple visit types so we must
ensure to retrieve the latest snapshot for the same visit type performed
by a loader.

Related to swh/meta#5092.

While the package version was properly uppercase encoded in the URL
for fetching package version info, it was not in the download URL
resulting in 404 and partial visit of the golang origin.

The oldest part of the scheduler API was updated to use model classes
(based on attr package) instead of dictionaries in order to improve
typing.

pkg_resources is now deprecated (as of setuptools 68) so use importlib
instead.

This is needed to make swh.loader.core not depend on swh.loader.package.

This used to contain the __version__ attribute as well as the
DEFAULT_PARAMS dict. These 2 have been moved to
swh/loader/core/__init__.py

The PackageLoader.get_loader_version() method has been fixed
accordingly.

Rather than having it defined in the conftest.py file, since it is not
recommended to import conftest directly.

This should help identify these classes full path independently from how their
module is loaded (eg. have a consistent logger name etc.), which can
make tests fail for no valid reason.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

Add an extra check in function fetch_extids_from_checksums to ensure
a NAR hash extid matches the NAR hash of the targeted archived object.

Related to swh/infra/sysadm-environment#5256.

Currently, those impacted loaders are only used through the nixguix stack but they are
not specific to nixguix.

Refs. #4749

It has been decommissionned in production and replaced by a lister and various
listers (as per the README).

Refs. swh/meta#3781

This now matches the tarball loader behavior (top-level directory included
[1]). This also matches what's expected by the guix dataset.

As the nix hashes computed are done from the first directory included in the
tarball though, we must also provide that directory. That way, the hashes
checks done during ingestion can match appropriately. That was the initial
implementation.

In terms of data, as this will change the visit snapshot and the extid
mappings, the core loaders (NodeLoader, ...) now declares an extid_version
bumped to 1 (it was 0 by default). Which means that all extid mappings will be
recomputed.

[1] https://gitlab.softwareheritage.org/swh/devel/swh-loader-core/-/blob/master/swh/loader/package/loader.py?ref_type=heads#L829-837

Refs. swh/infra/sysadm-environment#5222

Related to swh/meta#5075.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

This is required after the changes introduced in 54d8c7df and it is also
more consistent with real data provided by nix or guix.

Moreover, this loader should be removed soon as a better lister have been
implemented for nixguix that no longer create loading tasks with nixguix
visit type.