Related to D6967

This also drops spurious copyright headers to those files if present.

Related to T3812

Now that we have packaged tenacity 6.2 for debian buster and use it
in production, we can remove the workarounds to support tenacity < 5.

I would like to use it as the metadata authority URI in the loader,
instead of '{p_url.scheme}://{p_url.netloc}/', which I do not think
is accurate, as it is possible to have multiple Maven instances at
the same netloc.

A debian package can have sources coming from multiple suites
so we need to ensure to update the last_update field in the
ListedOrigin model if the current processed suite has a greater
modification time for its sources index.

Related to T2400

Use the value of the "Last-Modified" header from the HTTP response
resulting of the debian sources index HTTP request.

It will prevent to create loading tasks for debian packages with no
changes since last listing.

Related to T2400

All debian suites do not necessarily have the same set of components.

So prefer to log that a component is missing for a suite instead of
raising an excption that will stop the listing.

Remove no longer used date parameter in extra_loader_arguments.

Related to T2400

For a given package, the debian lister generates a dictionary mapping
distribution and version to a list of files to be processed by the
debian loader.

For each file to process, the debian loader expects to find an URI
in order to download it and then use its content to ingest package
source code into the archive.

However, it turns out these URIs were not computed by the lister
in its current implementation making any debian loading task fail
due to these missing info.

So add the computation of these URIS and ensure they will be provided
in the debian loader input parameters.

Related to T2400

In some circumstances, GitHub will return two separate repos with the
same html_url in the same page. This makes the lister fail with a
cardinality error.

The Maven lister retrieves the maven central indexes, exports them in a
convenient text format, and parse them to identify all src archives and
pom files in the maven repository. Then the pom files are downloaded and
analysed to find and yield any scm reference.

Note: This is a new version of the maven lister diff D6133 which takes
into account the initial round of reviews.

Related to T1724

This fixes the master build [1]

[1] https://jenkins.softwareheritage.org/view/swh-draft/job/DLS/job/tests/1625/console

It will be used to create a synthetic release message that contains
the package's name, like the Debian loader does.

Related to T3645

CRAN origins must be loaded with the cran visit type and not the tar one.

Related to T3675

Related to T3470

Related to T3629

This does not yet enter into the registration of a new lister.

Related to T3629

That avoids having multiple distinct opam root directories per opam lister instance. The
current opam commands used by the lister are actually listing specifically per instance.

Related to P1171

Any extra state initialization (outside the scheduler scope) is to happen in the
get_pages method.

Related to T3590

This matches how it's done for all other multi instances listers.

Related to T3590

We should avoid side-effects in the constructor as much as possible. That avoids
surprising behavior at object instantiation time. The state if needed must be
initialized into the `swh.lister.pattern.Lister.get_pages` method, as preconized in the
class docstring.

This also fixes the current test that actually bootstrap a real opam local "clone" in
/tmp.

Related to T3590

This will allow to dedicate the heptapod instances into its their own stats.

Related to T3581

Related to T3581#70593

This will allow to list the foss.heptapod.net instance for example.

Related to T3581

GitLab API can return errors 500 when listing projects
(see https://gitlab.com/gitlab-org/gitlab/-/issues/262629).

To avoid ending the listing prematurely, skip buggy URLs and move
to next pages.

Related to T3442

Increase number of origins per page to the maximum value allowed
by GitLab API (100) to send less requests.

Ask for simple responses to reduce size of JSON data.

Temporarily server failures can happen when listing a GitLab instance,
HTTP status codes 502, 503 or 520 are returned in that case.

So adapt lister requests retry policy to execute requests again when
such errors are encountered.

Related to T3442

This aligns the behavior with the opam loader

Related to T3358

Make the instance parameter of the base pattern lister optional and set
lister name to URL network location when not provided.

It simplifies lister creation when associated forge type have a lot of
instances in the wild (e.g. gitlab or cgit) while giving more details
about the listed forge instance.

Also process listers for forge with multiple instances (cgit, gitea,
gitlab, phabricator and tuleap) to ensure URL network location will be
used when instance parameter is not provided.

Related to T3403

This rewrote the current implementation to actually use pypi's xml-rpc api which allows
to be incremental. It also allows to fetch the last release date per package. This last
part actually make it possible to update the "last_update" entry in the ListedOrigin
model.

Related to T3399

See inline comment as to why.
This change also adds a Mercurial repo to the test data.

I previously forgot to add the `https://` prefix to the cloning URL.
Whoops.