Refs. swh/devel/swh-lister#1800

Refs. swh/devel/swh-lister#1800

This fixes tests hang when building package for debian buster.

When relisting an opam instance and the opam root directory is already
populated, the '--set-default' parameter must be provided otherwise the
following error is reported:

No switch is currently set. Please use 'opam switch' to set or install a switch

Related to swh/infra/sysadm-environment#4971.

Ensure opam errors are displayed when attempting to list all packages
in order to ease debugging.

Related to swh/infra/sysadm-environment#4971.

Use subprocess.run instead of subprocess.call and subprocess.Popen to
call opam commands and set check parameter to True in order to raise
CalledProcessError exception when an opam command failed.

This should help spotting issues with the opam lister.

Related to swh/infra/sysadm-environment#4971.

In contrary of gitea listing which does not require to provide the q query
parameter, it is required for the gogs case.

After reading the gogs source code, the /repos/search endpoint generates
a sql request of the form: "SELECT * FROM repos WHERE name LIKE '%{q}%'".
By setting the q parameter value to "_", the LIKE clause acts as a
wildcard and all repositories are ensured to be returned.

Fixes #4698.

Missing docstring prevents the task type to be registered in scheduler
database.

Pagure is a git-centered forge, python based using pygit2.

Its REST API enables to easily list all projects hosted in an
instance so the lister implementation is quite simple.

Related to swh/meta#5043.

The default behavior of subprocess is to pull executables from a
hardcoded list, which doesn't work when opam is installed manually in
the user's home directory.

mypy doesn't catch that multiple uses of
`self.listed_origins[origin_url]` in the same statement should be identical.
Using a temporary local variable for it seems to help.

The files we use weigh 440MB, and there are ~600MB of files we don't use

For the ones coming from a tarball. This matches the change happened in the associated
directory loader.

Refs. swh/infra/sysadm-environment#4906

Without this, the loader will fail.

Refs. swh/meta#4979

Prior to this, it was sending only 'directory' types for all vcs trees. Multiple
directory loaders now exist whose visit type are currently diverging, so the scheduling
would not happen correctly without it. This commit is the required adaptation for the
scheduling to work appropriately.

Refs. swh/meta#4979

Those will be ingested by the loader as "directory" with "nar" checksum layouts.

Refs. swh/infra/sysadm-environment#4868

Refs. swh/meta#4979

Some cgit instances are at a domain's root path so we can build their url directly from
their 'instance' parameter.

This unifies further the cli to register a lister and the cli to schedule the listed
origins from a forge.

[1]
```
https://git.kernel.org
https://source.codeaurora.org
https://git.trueelena.org
https://dev.sanctum.geek.nz
https://git.trueelena.org
https://git.dpkg.org
https://anongit.mindrot.org
https://git.aurel32.net
https://gitweb.gentoo.org
https://git.joeyh.name
https://git.adrian.geek.nz
```

Refs. #4693

This pushes the rather elementary logic within the lister's scope. This will simplify
and unify cli call between lister and scheduler clis. This will also allow to reduce
erroneous operations which can happen for example in the add-forge-now.

With the following, we will only have to provide the type and the instance, then
everything will be scheduled properly.

Refs. swh/devel/swh-lister#4693

```
$ swh lister run
Traceback (most recent call last):
  File "/home/dev/.local/bin/swh", line 33, in <module>
    sys.exit(load_entry_point('swh.core', 'console_scripts', 'swh')())
  File "/home/dev/swh-environment/swh-core/swh/core/cli/__init__.py", line 144, in main
    return swh(auto_envvar_prefix="SWH")
  File "/home/dev/.local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/dev/.local/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/dev/.local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/dev/.local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/dev/.local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/dev/.local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/dev/.local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/dev/swh-environment/swh-lister/swh/lister/cli.py", line 68, in run
    get_lister(lister, **config).run()
  File "/home/dev/swh-environment/swh-lister/swh/lister/__init__.py", line 75, in get_lister
    raise ValueError(
ValueError: Invalid lister None: only supported listers are ['arch', 'aur', 'bitbucket', 'bower', 'cgit', 'conda', 'cpan', 'cran', 'crates', 'debian', 'fedora', 'gitea', 'github', 'gitlab', 'gnu', 'gogs', 'golang', 'hackage', 'hex', 'launchpad', 'maven', 'nixguix', 'npm', 'nuget', 'opam', 'packagist', 'phabricator', 'pubdev', 'puppet', 'pypi', 'rubygems', 'sourceforge', 'tuleap']
```

Starting with the first url. As soon as one detection succeeds, this stops and yields
the result. Otherwise, continue with the detection on the next mirror url.

This should fix the current misbehavior [1] when multiple mirror urls are not ok but the
first one is.

[1] swh/infra/sysadm-environment#4868 (comment 137483)

Refs. swh/infra/sysadm-environment#4868

Refs. swh/meta#4979

The http_retry decorator has been moved to swh-core package in order
to ease its reuse across swh packages.

Instead of fully consuming the get_origins_from_page generator into
a list and truncate it, prefer to consume the generator origin per
origin and abort the process when the max number of origin per page
is reached.

Indeed some non trivial listers like the cgit one can perform costly
processing, HTTP request for instance, for each origin in a page.
So better not consuming the full generator in a row to avoid such
side effects.

This unifies with other lister tasks modules. And this allow the cgit task to
be scheduled by the add-forge-now scheduler cli.

Refs. swh/infra/sysadm-environment#4813

Some URLs of the repositories endpoint from BitBucket REST API 2.0
can return an error 500. In that case, skip the buggy repositories
page and get next one to continue listing and avoid to end it
prematurely.

Related to #4239

Related to swh/meta#4960

Related to swh/meta#4959

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

requests_ratelimited fixture from swh-core was renamed to
github_requests_ratelimited.

remaining_requests parameter was added to the github_response_callback
function from swh-core, making it no longer compatible with requests_mock
callback for json responses.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496