This pushes the rather elementary logic within the lister's scope. This will simplify
and unify cli call between lister and scheduler clis. This will also allow to reduce
erroneous operations which can happen for example in the add-forge-now.

With the following, we will only have to provide the type and the instance, then
everything will be scheduled properly.

Refs. #4693

```
$ swh lister run
Traceback (most recent call last):
  File "/home/dev/.local/bin/swh", line 33, in <module>
    sys.exit(load_entry_point('swh.core', 'console_scripts', 'swh')())
  File "/home/dev/swh-environment/swh-core/swh/core/cli/__init__.py", line 144, in main
    return swh(auto_envvar_prefix="SWH")
  File "/home/dev/.local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/dev/.local/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/dev/.local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/dev/.local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/dev/.local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/dev/.local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/dev/.local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/dev/swh-environment/swh-lister/swh/lister/cli.py", line 68, in run
    get_lister(lister, **config).run()
  File "/home/dev/swh-environment/swh-lister/swh/lister/__init__.py", line 75, in get_lister
    raise ValueError(
ValueError: Invalid lister None: only supported listers are ['arch', 'aur', 'bitbucket', 'bower', 'cgit', 'conda', 'cpan', 'cran', 'crates', 'debian', 'fedora', 'gitea', 'github', 'gitlab', 'gnu', 'gogs', 'golang', 'hackage', 'hex', 'launchpad', 'maven', 'nixguix', 'npm', 'nuget', 'opam', 'packagist', 'phabricator', 'pubdev', 'puppet', 'pypi', 'rubygems', 'sourceforge', 'tuleap']
```

Starting with the first url. As soon as one detection succeeds, this stops and yields
the result. Otherwise, continue with the detection on the next mirror url.

This should fix the current misbehavior [1] when multiple mirror urls are not ok but the
first one is.

[1] swh/infra/sysadm-environment#4868 (comment 137483)

Refs. swh/infra/sysadm-environment#4868

Refs. swh/meta#4979

The http_retry decorator has been moved to swh-core package in order
to ease its reuse across swh packages.

Instead of fully consuming the get_origins_from_page generator into
a list and truncate it, prefer to consume the generator origin per
origin and abort the process when the max number of origin per page
is reached.

Indeed some non trivial listers like the cgit one can perform costly
processing, HTTP request for instance, for each origin in a page.
So better not consuming the full generator in a row to avoid such
side effects.

This unifies with other lister tasks modules. And this allow the cgit task to
be scheduled by the add-forge-now scheduler cli.

Refs. swh/infra/sysadm-environment#4813

Some URLs of the repositories endpoint from BitBucket REST API 2.0
can return an error 500. In that case, skip the buggy repositories
page and get next one to continue listing and avoid to end it
prematurely.

Related to #4239

Related to swh/meta#4960

Related to swh/meta#4959

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

requests_ratelimited fixture from swh-core was renamed to
github_requests_ratelimited.

remaining_requests parameter was added to the github_response_callback
function from swh-core, making it no longer compatible with requests_mock
callback for json responses.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496

Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4684

Hopefully one day we'll be able to replace all of this mess with PEP692
TypedDict kwargs, but that's only on track for Python 3.12.

Some GitLab instances use specific namespaces for transient repositories
that it doesn't make sense to archive (for example, gitlab.org has a set
of QA namespaces used for integration testing of their production
deployments; drupal has an `issues/` namespace with forks of repos that
are only used for collaboration on merge requests, and aren't that
useful to be archived).

This cuts down one more manual step in the add forge now validation
process: we can add the relevant origins to the staging scheduler
without enabling them at all.

This will allow more automation of the staging add forge now process:
for known-good listers, we can limit the number of origins being
processed and reduce the amount of manual steps taken for each instance.

The SQL dump contains ownership instructions that can't be run if you
don't have the right users in your database clusters. When someone has a
psqlrc with ON_ERROR_STOP, this fails the load of the dump.

Use the opportunity to trigger an exception when psql returns a non-zero
exit code, rather than continue with an empty/inconsistent database.

In a similar way to the debian lister, use the following versions in the
packages dictionary provided to the generic rpm loader:

- dict keys are package versions prefixed by the fedora release and
  edition they have been found (fedora{release}/{edition}/{version}),
  they will be used as branch names targeting releases in the snapshot
  created by the rpm loader

- version fields in dict values are the package intrinsic versions parsed
  from package repository metadata excluding any ".fcXY" suffixes to avoid
  the loader to create multiple releases targeting the same directory,
  they will be used as release names in the snapshot created by the rpm
  loader

Related to T4448

Use http api lastUpload argument in search query to retrieve new or
updated origins since last run

Related to T4597

Summary: Lister to ingest fedora mirrors (.rpm)

Reviewers: #reviewers, vlorentz

Subscribers: vlorentz, olasd

Maniphest Tasks: T4448

Differential Revision: https://forge.softwareheritage.org/D8386

The lister is incremental and based on the value of ``commitTimeStamp`` retrieved on index http api endpoint.

Related T1718

Use with_release_since api argument to retrieve modules that have been
updated since the last date the lister has been executed.

Related T4519

For consistency with Maven and Packagist listers

Deploying the nixguix lister, I realized that even though the credentials configuration
is properly set for all listers, the listers actually requiring github origin
canonicalization do not have access to the github credentials. It's lost during the
constructor to only focus on the lister's credentials. Which currently translates to
listers being rate-limited.

This commit fixes it by pushing the self.github_session instantiation in the constructor
when the lister explicitely requires the github session. Hence lifting the rate limit
for maven, packagist, nixguix, and github listers.

Related to infra/sysadm-environment#4655

As a last fallback after the content-type check, instead of raising immediately.

Related to T3781

Prior to this, some urls were detected as file because their version name were wrongly
detected as extension, hence not matching tarball extensions.

Related to T3781