It now needs types-click which is indeed a dependency of
swh.core[testing].

- add config entry in pyproject.toml
- add a section in tox.ini (not executed by default for now)
- fix test_cli.py to please ruff (C401 rule)

and classify as Production/Stable.

this allows to make it actually benefit from multithreading, by not
sharing objstorage instances between threads.

Note: the doctest formely in process_replay_objects_content's docstring
has been moved to a proper test in test_replay.py; the required mocking
to make it work is not suitable for docstring anymore.

Related to swh/meta#4960

Related to swh/meta#4959

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496

This option allows to set a maximum size for content objects to be
replicated in the destination objstorage.

Add a few debug logging statements and improve the info-level summary
statement at the end of `process_replay_objects_content()`.

Fix tests accordingly.

to make it a bit more robust and give more details on what's going on in
case of errors.

Add tests for those error situations.

Side effect of rDJNL1d879f1dd6247b74d7e58dd0b4bd7b498ab506f0.

- pre-commit from 4.1.0 to 4.3.0,
- codespell from 2.2.1 to 2.2.2,
- black from 22.3.0 to 22.10.0 and
- flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).

This argument is required by swh-objstorage >= 2.0.0

The obj_id-less form is being deprecated.

That hook can be frustrating as it can discard a long commit message
if it finds a typo in it so better removing it.

Related to T3922

black is considered stable since release 22.1.0 and the version
we are currently using is quite outdated and not compatible with
click 8.1.0, so it is time to bump it to its latest stable release.

Please note that E501 pycodestyle warning related to line length
is replaced by B950 one from flake8-bugbear as recommended by black.
https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length

Related to T3922

pytest-postgresql 3.1.3 and pytest-redis 2.4.0 added support for
pytest >= 7 so we can now drop the pytest pinning.

Due to test modules being copied in subdirectories of the
build directory by setuptools, it makes pytest fail by raising
ImportPathMismatchError exceptions when invoked from root
directory of the module.

So ignore the build folder to discover tests.

To install the new hook:

  $ pre-commit install -t commit-msg

Related to T3916

for each batch of messages, dispatch the copy of individual objects in a
ThreadPoolExecutor.
The idea is to allow concurrency to ge beyong process parallelism
provided by kafka consumer groups. Since the copy a one object is mainly
IO bound (check existence in destination objstare, retrieve from source
objstorage, put in destination objstorage) with possibly large delays
(eg. retrieveing a blob from s3 imply a minimum 150/200ms delay before
the HTTP request is replied, whatever the size of the object);
this tries to parallelize those delays.

and improve the documentation of the replayer command.

This also drops spurious copyright headers to those files if present.

Related to T3812

which is the one available for buster on swh.

this is not an ideal solution since it's using a global variable
(swh.objstorage.replayer.replay.REPORTER) to activate the replaying
error reporting feature, but according the nature of tenacity (used to
implement retry capacity of interactions with the source and destination
objstorages), there is no easy solution (that I can see).

split the 'copy_object()' in 2 parts, 'get_object()' and 'put_object()'
and make each of these decorated by '@content_replay_retry' (instead of
the whole 'copy_object()'), so that a failing 'put_object' does not
trigger getting the object from the src objstorage again.

Also, only log in statsd the end result (error or success) instead of
logging each attempt. We don't need these intermediate results in
statsd, and it makes them much harder to use in a dashboard.

Last, for the sake of implementation ease, use function name as
"operation" tag in stastds reports (so "get_object", "put_object" and
"obj_in_storage") instead of "copy" and "in_dst".

Doing so, the 'operation' argument of the ReplayError exception has been
dropped.

make '_fill_objstorage_and_kafka()' take the actual objstorage as
argument, instead of the dict of objstorages ("src", "dst"). The "dst"
objstorage is not used in the function, and it make the intent of this
later clearer.