1. When passed a .yaml path (or any path with a non-whitelisted extension),
   don't read .yml instead when the .yaml exists.
   It's an extremely surprising behavior.

2. If the .yaml file does not exist, it will still try alternative extensions
   in order not to break existing deployments which may rely on it, but it
   raises a warning now.

3. When given a non-existing path, show an error log, but keep parsing
   it as an empty config, in order not to break existing deployments.

eg. '413 Request Entity Too Large' raised by nginx

It returns the full JSON metadata of a GitHub repository by querying the
GitHub REST API.

Also refactor get_canonical_url method with it.

Related to swh-loader-git#3652.

GitHub API seems to have harmonized its responses when API rate limit is
exceeded regardless an user is authenticated or not: a 403 status code is
returned with an error message starting with "API rate limit exceeded"
(https://docs.github.com/en/rest/overview/resources-in-the-rest-api
?apiVersion=2022-11-28#exceeding-the-rate-limit).

Previously, a 429 status code was returned for authenticated users.

So update code detecting exceeded rate limit accordingly and keep the
429 status check for backward compatibility just in case.

Fix #4429

It enables to deduplicate HTTP retry code used in swh-lister and
swh-loader-core packages.

It is useful to compute the release name for sentry when the SWH_MAIN_PACKAGE
environment variable is not defined.

Related to swh/meta#4949.

We don't use positional arguments anywhere, except in the one
place where we accidentally passed namespace as host:
swh-journal!260

Jérémy Bobbio (Lunar) authored 2 years ago and Jérémy Bobbio (Lunar) committed 2 years ago

Related to swh/meta#4959

Related to swh/meta#4960

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

This imports code from swh.storage.api.server, so it can be reused
by other RPC servers for psycopg2 backends (swh-indexer-storage, swh-scheduler)

This module only depends on core python library.

That will allow to simplify deployment logging configuration per infrastructure.

Refs. swh/infra/sysadm-environment#4524

We will need to monitored token usage when scheduling recurring fetches
of extrinsic metadata.

Metrics on API requests and sleep time themselves are just an extra
that may be handy in the future.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496

It's already supported [1] but not wired in this module.

Related to T3781#97784

- pre-commit from 4.1.0 to 4.3.0,
- codespell from 2.2.1 to 2.2.2,
- black from 22.3.0 to 22.10.0 and
- flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).

.rstrip() is not .removesuffix()

When shutil.unpack_archive failed to unpack a tarball, fallback using
the tar command to perform that task.

Such issues were encountered when trying to unpack old tarballs coming
from CPAN.

Related to T2833

It exist cases when an uncompressed tarball has directories without
any traversal or read permissions so ensure to also fix directory
permissions in normalize_permissions function.

Related to T3781

This will allow to reuse it when we try to detect tarball nature in lister.

Related to T3781

This mirrors the behavior of the synchronous server.

In practice, this means the objstorage won't log ObjNotFoundError or
report them to Sentry.

In order for that useful fixture to be shared across multiple swh
packages, move it from swh-indexer to main swh-core pytest plugin.

Also add a couple of tests for it.

Without the return, 'psycopg2.connect(db_or_conninfo)' is actually reachable
when calling next() twice on the generator

This will be used to make the storage retry proxy retry harder
when the storage server raises errors like psycopg2.OperationalError;
which last longer than we want the proxy to retry for typical
exceptions.

It was removed from all RPC servers years ago.

They are client errors, therefore it does not make sense to log them
as exceptions (which is only useful for debugging the server).

According to
<https://github.com/pallets/flask/issues/4734#issuecomment-1204124984>,
app.test_client() should not be used as a long-lived context manager.

This causes swh-indexer's tests to break with Flask >= 2.2.0 because
some of them start threads within the lifespan of the context manager,
which causes a crash.

As far as I can tell, simply assigning app.test_client() instead of
entering does not break anything.

so one can create database (e.g. for unit tests) of given flavor.

We already have 'msgpack >= 1.0.0' in requirements.txt

_zst_unpack method added to support '.tar.zst' extension and 'application/zstd', 'application/x-zstd' mime type.
Add archive example file and a test for unpacking zst file.