Related to T3922

black is considered stable since release 22.1.0 and the version
we are currently using is quite outdated and not compatible with
click 8.1.0, so it is time to bump it to its latest stable release.

Please note that E501 pycodestyle warning related to line length
is replaced by B950 one from flake8-bugbear as recommended by black.
https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length

Related to T3922

pytest-postgresql 3.1.3 and pytest-redis 2.4.0 added support for
pytest >= 7 so we can now drop the pytest pinning.

The Crates lister retrieves crates package for Rust lang.

It basically fetches https://github.com/rust-lang/crates.io-index.git
to a temp directory and then walks through each file to get the
crate's info.

Due to test modules being copied in subdirectories of the
build directory by setuptools, it makes pytest fail by raising
ImportPathMismatchError exceptions when invoked from root
directory of the module.

So ignore the build folder to discover tests.

Commit 6a747955 modified the origin URLs for CVS projects
hosted on SourceForge but it also broke incremental listing
due to a no longer valid assertion, so fix that issue.

The decorator is dropped on `get_origins_from_page` as we cannot retry an iterator
consumption anyway.

Related to T3948

CVS projects are different from other VCS ones, they use the rsync
protocol, a list of modules needs to be fetched from an info page
and multiple origin URLs can be produced for a same project.

Related to T3789

as the scheduler is now able to deduplicate it when recording listed origins.

Related to T3945

Related to T3945

Prior to this commit, the listing could fail when either reading a page or the page of
results (lauchpad api raises RestfulError). This now retries when those kind of
exceptions happen. If the error persists (after multiple tryouts and exponential
backoff), the listing continues nonetheless (with warning logs).

Note that if the page ends up being empty, it's no longer accounted for.

This actually allows the listing to finish in case of issues.

Related to T3945

Related to T3945

Bazaar support was removed a long time ago and predates a lot of the new
mechanisms in place in the API. Unfortunately, it looks like a lot of
the URLs are offline now, but there are still a few projects that can be
listed, this is pretty low-effort.

To install the new hook:

  $ pre-commit install -t commit-msg

We need to avoid using naive datetime as this fails during conversion.

Related to T3746
Related to P1280

Related to T3916

And log the metadata dictionary.

Related to D6967

This also drops spurious copyright headers to those files if present.

Related to T3812

Now that we have packaged tenacity 6.2 for debian buster and use it
in production, we can remove the workarounds to support tenacity < 5.

I would like to use it as the metadata authority URI in the loader,
instead of '{p_url.scheme}://{p_url.netloc}/', which I do not think
is accurate, as it is possible to have multiple Maven instances at
the same netloc.

A debian package can have sources coming from multiple suites
so we need to ensure to update the last_update field in the
ListedOrigin model if the current processed suite has a greater
modification time for its sources index.

Related to T2400

Use the value of the "Last-Modified" header from the HTTP response
resulting of the debian sources index HTTP request.

It will prevent to create loading tasks for debian packages with no
changes since last listing.

Related to T2400

All debian suites do not necessarily have the same set of components.

So prefer to log that a component is missing for a suite instead of
raising an excption that will stop the listing.

Remove no longer used date parameter in extra_loader_arguments.

Related to T2400

For a given package, the debian lister generates a dictionary mapping
distribution and version to a list of files to be processed by the
debian loader.

For each file to process, the debian loader expects to find an URI
in order to download it and then use its content to ingest package
source code into the archive.

However, it turns out these URIs were not computed by the lister
in its current implementation making any debian loading task fail
due to these missing info.

So add the computation of these URIS and ensure they will be provided
in the debian loader input parameters.

Related to T2400

In some circumstances, GitHub will return two separate repos with the
same html_url in the same page. This makes the lister fail with a
cardinality error.

The Maven lister retrieves the maven central indexes, exports them in a
convenient text format, and parse them to identify all src archives and
pom files in the maven repository. Then the pom files are downloaded and
analysed to find and yield any scm reference.

Note: This is a new version of the maven lister diff D6133 which takes
into account the initial round of reviews.

Related to T1724

This fixes the master build [1]

[1] https://jenkins.softwareheritage.org/view/swh-draft/job/DLS/job/tests/1625/console

It will be used to create a synthetic release message that contains
the package's name, like the Debian loader does.

Related to T3645

CRAN origins must be loaded with the cran visit type and not the tar one.

Related to T3675