Remove use of --import-mode=importlib pytest option and use
new option consider_namespace_packages to fix tests execution
with latest pytest release.

Related to swh/meta#5075.

The db pytest plugin was deprecated in June 2022 (in commit
4fc5f601) and has no users left.

This is the last blocker to supporting pytest-postgresql v5.

To make it easier to debug code in the Docker stack, allow to set the
general log level through the `SWH_LOG_LEVEL` environment variable.
This can then easily be set in `docker.override.yml` like so:

    swh-lister:
      environment:
        SWH_LOG_LEVEL: "debug"

Click provides the plumbing to do this easily, see:
https://click.palletsprojects.com/en/8.1.x/options/#values-from-environment-variables

Typing stubs for click and flask are now maintained by upstreams
so remove their outdated typing packages.

When building package documentation outside tox by calling make in the
docs directory, the include of Makefile.sphinx inside the docs Makefile
was failing as its relative path was invalid.

So adapt this relative path according if the SWH_PACKAGE_DOC_TOX_BUILD
environment variable is set or not.

Also align black version in tox.ini file.

- remove sphinx-dev tox env,
- use non-editable install of swh-docs to build the sphinx doc in tox,
- remove (now useless) MANIFEST.in.

It's not needed any more.

This replaces the setup.py package file with pyproject.toml (keeping
setuptools and setuptool_scm) and use an implicit namespace declaration
for the swh namespace.

Also note that due to the poor support of PEP 420 in pytest, it needs a
pair of command line arguments to pass all tests (see tox.ini).

Instead of guessing archive format by first checking its extension then
its mime type, prefer to do the opposite as it is more reliable, especially
if an archive file has an incorrect extension.

It's currently happening for some lister, preventing them from finishing their listing.

This adds the missing tests on that part.
This also gives up after 5 attempts.

It was missing a commit to actually be granted.
Add a test for this specific behavior, making sure the guest user has
proper permissions.

Note that this will work if and only is the guest user does not inherit
from a bunch of default permissions when created (needs a NOINHERIT).

This allows to run the command `swh db init --dname DBURI` without
giving a configuration file while still populating the dbversion table
in the database.

Note that this may not work for all swh packages, when the postgresql
backend for the given swh package needs more configuration entries than
just the psql connection string.

This more or less drops support for "old style" db sql init scripts:

- the `swh db init` will still work for an 'old style' package (in which
  the sql init scripts do create dbversion table and initialize it) but
  will display an error message,
- the --initial-version option has been dropped, since this made sense
  only for old style packages,
- the test data 'cli_new' has been renamed as 'cli' (in place of the
  previous "old style" one)

Latest sentry-sdk release now raises a BadDsn exception when
DSN format does not match the expected one.

This allows using the "system" tox, if it's recent enough, instead of
always provisioning an internal .tox venv with tox 4.

Instead of going back to py3, pass through the environment name, so that
it can be called with an arbitrary interpreter version.

Historic files are present in our sql modules which cannot be executed by the migration
cli (without complicating too much the code). So simply ignore those when listing the
files to execute. That will stop making the cli refuse to work on some modules (e.g.
scheduler and its 30-bis.sql script).

Refs. swh/infra/sysadm-environment#4933

Before:

```
$ swh db init -h
Usage: swh db init [OPTIONS] MODULE

  Initialize a database for the Software Heritage <module>.

  The database connection string comes from the configuration file (see option
  ``--config-file`` in ``swh db --help``) in the section named after the
  MODULE argument.

  Example::

      $ cat conf.yml     storage:       cls: postgresql       db:
      postgresql://user:passwd@pghost:5433/swh-storage       objstorage:
      cls: memory

      $ swh db -C conf.yml init storage  # or     $
      SWH_CONFIG_FILENAME=conf.yml swh db init storage

  Note that the connection string can also be passed directly using the '--db-
  name' option, but this usage is about to be deprecated.
```

After:

```
$ swh db init -h
Usage: swh db init [OPTIONS] MODULE

  Initialize a database for the Software Heritage <module>.

  The database connection string comes from the configuration file (see option
  ``--config-file`` in ``swh db --help``) in the section named after the
  MODULE argument.

  Example::

          $ cat conf.yml
          storage:
            cls: postgresql
            db: postgresql://user:passwd@pghost:5433/swh-storage
            objstorage:
              cls: memory

          $ swh db -C conf.yml init storage  # or
          $ SWH_CONFIG_FILENAME=conf.yml swh db init storage

  Note that the connection string can also be passed directly using the '--db-
  name' option, but this usage is about to be deprecated.
```

1. When passed a .yaml path (or any path with a non-whitelisted extension),
   don't read .yml instead when the .yaml exists.
   It's an extremely surprising behavior.

2. If the .yaml file does not exist, it will still try alternative extensions
   in order not to break existing deployments which may rely on it, but it
   raises a warning now.

3. When given a non-existing path, show an error log, but keep parsing
   it as an empty config, in order not to break existing deployments.

eg. '413 Request Entity Too Large' raised by nginx

It returns the full JSON metadata of a GitHub repository by querying the
GitHub REST API.

Also refactor get_canonical_url method with it.

Related to swh-loader-git#3652.

GitHub API seems to have harmonized its responses when API rate limit is
exceeded regardless an user is authenticated or not: a 403 status code is
returned with an error message starting with "API rate limit exceeded"
(https://docs.github.com/en/rest/overview/resources-in-the-rest-api
?apiVersion=2022-11-28#exceeding-the-rate-limit).

Previously, a 429 status code was returned for authenticated users.

So update code detecting exceeded rate limit accordingly and keep the
429 status check for backward compatibility just in case.

Fix #4429

It enables to deduplicate HTTP retry code used in swh-lister and
swh-loader-core packages.

It is useful to compute the release name for sentry when the SWH_MAIN_PACKAGE
environment variable is not defined.

Related to swh/meta#4949.

We don't use positional arguments anywhere, except in the one
place where we accidentally passed namespace as host:
swh-journal!260

Jérémy Bobbio (Lunar) authored 2 years ago and Jérémy Bobbio (Lunar) committed 2 years ago

Related to swh/meta#4959