This allows to run the command `swh db init --dname DBURI` without
giving a configuration file while still populating the dbversion table
in the database.

Note that this may not work for all swh packages, when the postgresql
backend for the given swh package needs more configuration entries than
just the psql connection string.

This more or less drops support for "old style" db sql init scripts:

- the `swh db init` will still work for an 'old style' package (in which
  the sql init scripts do create dbversion table and initialize it) but
  will display an error message,
- the --initial-version option has been dropped, since this made sense
  only for old style packages,
- the test data 'cli_new' has been renamed as 'cli' (in place of the
  previous "old style" one)

Latest sentry-sdk release now raises a BadDsn exception when
DSN format does not match the expected one.

This allows using the "system" tox, if it's recent enough, instead of
always provisioning an internal .tox venv with tox 4.

Instead of going back to py3, pass through the environment name, so that
it can be called with an arbitrary interpreter version.

Historic files are present in our sql modules which cannot be executed by the migration
cli (without complicating too much the code). So simply ignore those when listing the
files to execute. That will stop making the cli refuse to work on some modules (e.g.
scheduler and its 30-bis.sql script).

Refs. swh/infra/sysadm-environment#4933

Before:

```
$ swh db init -h
Usage: swh db init [OPTIONS] MODULE

  Initialize a database for the Software Heritage <module>.

  The database connection string comes from the configuration file (see option
  ``--config-file`` in ``swh db --help``) in the section named after the
  MODULE argument.

  Example::

      $ cat conf.yml     storage:       cls: postgresql       db:
      postgresql://user:passwd@pghost:5433/swh-storage       objstorage:
      cls: memory

      $ swh db -C conf.yml init storage  # or     $
      SWH_CONFIG_FILENAME=conf.yml swh db init storage

  Note that the connection string can also be passed directly using the '--db-
  name' option, but this usage is about to be deprecated.
```

After:

```
$ swh db init -h
Usage: swh db init [OPTIONS] MODULE

  Initialize a database for the Software Heritage <module>.

  The database connection string comes from the configuration file (see option
  ``--config-file`` in ``swh db --help``) in the section named after the
  MODULE argument.

  Example::

          $ cat conf.yml
          storage:
            cls: postgresql
            db: postgresql://user:passwd@pghost:5433/swh-storage
            objstorage:
              cls: memory

          $ swh db -C conf.yml init storage  # or
          $ SWH_CONFIG_FILENAME=conf.yml swh db init storage

  Note that the connection string can also be passed directly using the '--db-
  name' option, but this usage is about to be deprecated.
```

1. When passed a .yaml path (or any path with a non-whitelisted extension),
   don't read .yml instead when the .yaml exists.
   It's an extremely surprising behavior.

2. If the .yaml file does not exist, it will still try alternative extensions
   in order not to break existing deployments which may rely on it, but it
   raises a warning now.

3. When given a non-existing path, show an error log, but keep parsing
   it as an empty config, in order not to break existing deployments.

eg. '413 Request Entity Too Large' raised by nginx

It returns the full JSON metadata of a GitHub repository by querying the
GitHub REST API.

Also refactor get_canonical_url method with it.

Related to swh-loader-git#3652.

GitHub API seems to have harmonized its responses when API rate limit is
exceeded regardless an user is authenticated or not: a 403 status code is
returned with an error message starting with "API rate limit exceeded"
(https://docs.github.com/en/rest/overview/resources-in-the-rest-api
?apiVersion=2022-11-28#exceeding-the-rate-limit).

Previously, a 429 status code was returned for authenticated users.

So update code detecting exceeded rate limit accordingly and keep the
429 status check for backward compatibility just in case.

Fix #4429

It enables to deduplicate HTTP retry code used in swh-lister and
swh-loader-core packages.

It is useful to compute the release name for sentry when the SWH_MAIN_PACKAGE
environment variable is not defined.

Related to swh/meta#4949.

We don't use positional arguments anywhere, except in the one
place where we accidentally passed namespace as host:
swh-journal!260

Jérémy Bobbio (Lunar) authored 2 years ago and Jérémy Bobbio (Lunar) committed 2 years ago

Related to swh/meta#4959

Related to swh/meta#4960

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

This imports code from swh.storage.api.server, so it can be reused
by other RPC servers for psycopg2 backends (swh-indexer-storage, swh-scheduler)

This module only depends on core python library.

That will allow to simplify deployment logging configuration per infrastructure.

Refs. swh/infra/sysadm-environment#4524

We will need to monitored token usage when scheduling recurring fetches
of extrinsic metadata.

Metrics on API requests and sleep time themselves are just an extra
that may be handy in the future.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496

It's already supported [1] but not wired in this module.

Related to T3781#97784

- pre-commit from 4.1.0 to 4.3.0,
- codespell from 2.2.1 to 2.2.2,
- black from 22.3.0 to 22.10.0 and
- flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).

.rstrip() is not .removesuffix()

When shutil.unpack_archive failed to unpack a tarball, fallback using
the tar command to perform that task.

Such issues were encountered when trying to unpack old tarballs coming
from CPAN.

Related to T2833

It exist cases when an uncompressed tarball has directories without
any traversal or read permissions so ensure to also fix directory
permissions in normalize_permissions function.

Related to T3781

This will allow to reuse it when we try to detect tarball nature in lister.

Related to T3781

This mirrors the behavior of the synchronous server.

In practice, this means the objstorage won't log ObjNotFoundError or
report them to Sentry.

In order for that useful fixture to be shared across multiple swh
packages, move it from swh-indexer to main swh-core pytest plugin.

Also add a couple of tests for it.

Without the return, 'psycopg2.connect(db_or_conninfo)' is actually reachable
when calling next() twice on the generator

This will be used to make the storage retry proxy retry harder
when the storage server raises errors like psycopg2.OperationalError;
which last longer than we want the proxy to retry for typical
exceptions.