Jérémy Bobbio (Lunar) authored 2 years ago and Jérémy Bobbio (Lunar) committed 2 years ago

Related to swh/meta#4959

Related to swh/meta#4960

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

This imports code from swh.storage.api.server, so it can be reused
by other RPC servers for psycopg2 backends (swh-indexer-storage, swh-scheduler)

This module only depends on core python library.

That will allow to simplify deployment logging configuration per infrastructure.

Refs. swh/infra/sysadm-environment#4524

We will need to monitored token usage when scheduling recurring fetches
of extrinsic metadata.

Metrics on API requests and sleep time themselves are just an extra
that may be handy in the future.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496

It's already supported [1] but not wired in this module.

Related to T3781#97784

- pre-commit from 4.1.0 to 4.3.0,
- codespell from 2.2.1 to 2.2.2,
- black from 22.3.0 to 22.10.0 and
- flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).

.rstrip() is not .removesuffix()

When shutil.unpack_archive failed to unpack a tarball, fallback using
the tar command to perform that task.

Such issues were encountered when trying to unpack old tarballs coming
from CPAN.

Related to T2833

It exist cases when an uncompressed tarball has directories without
any traversal or read permissions so ensure to also fix directory
permissions in normalize_permissions function.

Related to T3781

This will allow to reuse it when we try to detect tarball nature in lister.

Related to T3781

This mirrors the behavior of the synchronous server.

In practice, this means the objstorage won't log ObjNotFoundError or
report them to Sentry.

In order for that useful fixture to be shared across multiple swh
packages, move it from swh-indexer to main swh-core pytest plugin.

Also add a couple of tests for it.

Without the return, 'psycopg2.connect(db_or_conninfo)' is actually reachable
when calling next() twice on the generator

This will be used to make the storage retry proxy retry harder
when the storage server raises errors like psycopg2.OperationalError;
which last longer than we want the proxy to retry for typical
exceptions.

It was removed from all RPC servers years ago.

They are client errors, therefore it does not make sense to log them
as exceptions (which is only useful for debugging the server).

According to
<https://github.com/pallets/flask/issues/4734#issuecomment-1204124984>,
app.test_client() should not be used as a long-lived context manager.

This causes swh-indexer's tests to break with Flask >= 2.2.0 because
some of them start threads within the lifespan of the context manager,
which causes a crash.

As far as I can tell, simply assigning app.test_client() instead of
entering does not break anything.

so one can create database (e.g. for unit tests) of given flavor.

We already have 'msgpack >= 1.0.0' in requirements.txt

_zst_unpack method added to support '.tar.zst' extension and 'application/zstd', 'application/x-zstd' mime type.
Add archive example file and a test for unpacking zst file.

It exists many cases where using unzip to extract a jar archive
fails while using the zipfile module succeeds.

So prefer to use the zipfile module to uncompress jar archives.

Related to T4318

Discovered when working on [1].

Related to T4228

[1] D7949

This now automatically grant read-only access to the guest user at db initialization or
upgrade time. This way, sysadm no longer have to manually alter the schema after
initialization or upgrade.

Related to T4228

The sample of those exotic urls got extracted out the staging scheduler [1].

And the actual run of that code makes the github api (without mocks) returns the correct
canonical urls [2]

Related to T3874

[1] P1371

[2]
```
$ ipython
...
In [1]: from swh.core.github.utils import get_canonical_github_origin_url

In [2]: get_canonical_github_origin_url('git@github.com/huaweicloud/huaweicloud-sdk-java-v3.git')
No tokens set in configuration, using anonymous mode
Out[2]: 'https://github.com/huaweicloud/huaweicloud-sdk-java-v3'

In [3]: get_canonical_github_origin_url('git//github.com/powertac/powertac-server.git')
No tokens set in configuration, using anonymous mode
Out[3]: 'https://github.com/powertac/powertac-server'

In [4]: get_canonical_github_origin_url('https://${env.GITHUB_USER}:${env.GITHUB_TOKEN}@github.com/molgenis/vibe.git')
No tokens set in configuration, using anonymous mode
Out[4]: 'https://github.com/molgenis/vibe'

In [5]: get_canonical_github_origin_url('ssh://git@github.com/softwaremagico/ThinkMachine.git')
No tokens set in configuration, using anonymous mode
Out[5]: 'https://github.com/softwaremagico/ThinkMachine'

In [6]: get_canonical_github_origin_url('ssh://github.com:alibaba/SmartEngine.git')
No tokens set in configuration, using anonymous mode
Out[6]: 'https://github.com/alibaba/SmartEngine'

In [7]: get_canonical_github_origin_url('//github.com:networknt/light-tram-kafka.git')
No tokens set in configuration, using anonymous mode
Out[7]: 'https://github.com/networknt/light-tram-kafka'

In [8]: get_canonical_github_origin_url('[fetch=]git@github.com:turnonline/ecosystem-admin-widgets.git')
No tokens set in configuration, using anonymous mode
Out[8]: 'https://github.com/turnonline/ecosystem-admin-widgets'

In [9]: get_canonical_github_origin_url('git@github.com:ttulka/spring-boot-configuration-properties-store.git')
No tokens set in configuration, using anonymous mode
Out[9]: 'https://github.com/ttulka/spring-boot-configuration-properties-store'

```

to prepare the deprecation of db/pytest_plugin.py; it's the only
function from db/pytest_plugin.py still in use.

instead of the custom factory_fact.

This is the second part of 240ae09d.

Related to T4228

Some module cannot just define their module db configuration key entry as their module
name because they define more services than just their db (e.g. indexer, scrubber, ...)

Related to T4284