In order to reduce http api call amount made by the loader, download a
crates.io database dump, and parse its csv files to get a last_update
value for each versions of a Crate.
Those values are sent to the loader through extra_loader_arguments
'crates_metadata'.

'artifacts' and 'crates_metadata' now uses "version" as key.

Related T4104, D8171

In that case, this fallbacks to use the "outputHash" which is an equivalent field of the
integrity one except it's for "recursive" outputHashMode. This adds the necessary
assertions around this case so correct data is sent to loaders as well.

Related to T3781

This actually includes all query param values as paths to check. When paths have
extensions, it then pattern matches against tarballs if any. When no extension is
detected, it's doing as before, fallbacks to head query the url to have more information
on the file.

Prior to this commit, this only looked over a hard-coded list of values (for hard-coded
keys: file, f, name, url) detected through docker runs. This way of doing it should
decrease future misdetections (when new unknown "keys" show up in the wild).

Related to T3781

The current content type detection was a bit off mostly for content which includes
charset. This commit fixes it.

Related to T3781

Without this, some git repositories are detected as file (due to upstream
misqualification too). This does some extra effort to detect those to avoid sending
noise to loaders.

This also refactors some common code to build vcs artifacts to avoid duplication.

Related to T3781

Without this, some tarballs hidden within query parameters are not detected. This does
some extra effort to detect those to avoid sending noise to loaders.

Related to T3781

Without this distinction the current directory or content loader will fail the download
as they currently expect the checksums to be about the tarball. When a recursive
"integrity" is provided, it's actually about the uncompressed tarball as per the
nix-store computation.

It's detailed within the code.

Related to T3294
Related to T3781

Related to T3294
Related to T3781

When that arises, we skip the origins.

Related to T3781

Related to T3781

When that arises, we skip the origins.

Related to T3781

Some origins are listed as urls while they are not. They are possibly vcs. So this
commit tries to detect and and deal with those if possible. If not possible, they are
skipped.

Related to T3781
Related to P1470

The end goal is to ingest sparsely the origins, that would avoid hitting the various
servers around the same time for colocated origins in the upstream manifest (especially
file or tarball).

Related to T3781

Related to T3781

Related to T3781

Related to T3781

In listers collecting artifacts for each package to load, add artifacts
checksums, when that info is available, in parameters sent to loaders
in order to check downloaded artifact integrity.

'artifacts' extra_loader_arguments should be a list

Related T1777

Prefer to execute lister through a celery task as it also enables to
catch possible issues with task implementation.

Also use docker compose v2 commands.

The base lister class now ensures the count of listed origins will
be accurate.

Previously, the run method was returning the total count of ListedOrigin
objects sent to scheduler database.

However, some listers can send multiple ListedOrigin objects for a given
origin URL during the listing process, for instance when an origin is
contained in multiple pages (e.g. gogs listing) or when the listing
is gathering multiple versions of an origin spread across multiple
pages (e.g. maven listing).

This changes ensures an accurate count of listed origins by maintaining
a set of origin URLs associated to the sent ListedOrigin objects.

Related T1718

The puppet lister retrieves origins from from https://forge.puppet.com/modules

Related T4519

Related T2833

Use http api point to get package names and build origin urls.

Related T4547

With the extension, the readme is included in the swh-docs build and fails. It's not
intended for the documentation build so renaming it keep it out of the doc build loop.

This fixes build [1].

[1] https://jenkins.softwareheritage.org/view/all/job/DDOC/job/dev/2395/

That HTTP header value will now contain the lister name but also a link
to our contact form in order for sysadmins to easily reach us if needed.

The following template is used to generate it:

"Software Heritage <lister_name> lister v<swh-lister version>
 (+https://www.softwareheritage.org/contact)"

Numerous listers were using the same page_request method or equivalent
in their implementation so prefer to deduplicate that code by adding
an http_request method in base lister class: swh.lister.pattern.Lister.

That method simply wraps a call to requests.Session.request and logs
some useful info for debugging and error reporting, also an HTTPError
will be raised if a request ends up with an error.

All listers using that new method now benefit of requests retry when
an HTTP error occurs thanks to the use of the http_retry decorator.

Instead of retrying HTTP requests only for 429 status code by default,
prefer to use the generic retry policy enabling to also retry for status
codes >= 500 but also on ConnectionError exceptions.

Rename throttling_retry decorator to http_retry to reflect this change.

For some forges, the default tab for a repository detail is not the
summary tab so the clone urls are not detected and the repository
is ignored

Related to T4544

This also affects the gitea lister

Align with other lister names by turning it to lowercase.

It ensures created temporary directories will be removed once they
are no longer needed.

This matches other lister verbosity.

Related to T4517

Use a value that matches good practice recommended by pub.dev REST API doc.

https://github.com/dart-lang/pub/blob/master/doc/repository-spec-v2.md

In order to get a last_update for each ListedOrigin sent to scheduler
database, send an extra HTTP request for each listed package to the
/api/packages/<package_name> endpoint of pub.dev API.

A pub.dev developer inform us that endpoint is heavily used and cached
so there is no particular issues to query that endpoint for each package
in a row periodically.

Use https://pub.dev/packages/<package_name> instead of
https://pub.dev/api/packages/<package_name>