recent change broke the statsd reporting at swh.journal.client level by
missing the namespace argname while instanciating the Statsd class, thus
making "swh_journal_client" the host os the Statsd instance.

Add a test for the usage of statsd in the journal client.

Related to swh/meta#4960

Related to swh/meta#4960

Related to swh/meta#4959

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496

`make install` is not working since commit ed9d6827 (Feb 2021)
as the confluent-kafka-python mock broker removed the need for it
and has been in use since April 2020.

using a constructor 'auto_flush' bookean argument.
The idea is that in a test session, each call to 'flush()' takes ~1s to
run, so having the test handling a single call to flush() when needed
(instead of n calls) make test execution significantly faster.

For example, swh-storage's test_replay.py execution went from ~140s to
~40s, and test_backfill.py from ~40s to ~15s.

this actually speeds up tests quite a bit (preventing a 30s timeout when
the fixture is actually reused several times in a test session).

- pre-commit from 4.1.0 to 4.3.0,
- codespell from 2.2.1 to 2.2.2,
- black from 22.3.0 to 22.10.0 and
- flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).

So it can be cross-linked from another part of the documentation.

908f0154 introduced an unexpected change of behavior of the
InMemoryJournalWriter, making it always anonymize objects. This broke
tests for some dependencies (esp. swh-storage).

This fix adds an 'anonymize' (bool) argument to the class constructor
(similar to the KafkaJournalWriter).

and support the "-" value (for stdout).

similar to kafka_to_value but takes a stream as input and return a
msgpack.Unpacker object on which the user can iterate.

- add a JournalWriterInterface in a new interface.py module,
- add a flush() method to this interface,
- remove (now unused) write_update methods,
- move all type-related code in this module,
- fix InMemoryJournalWriter type annotations,
- remove usage of the TValue TypeVar in favor of using ValueProtocol
  directly.

for StreamJournalWriter and InMemoryJournalWriter, and make them
consistent with KafkaJournalWriter.

That hook can be frustrating as it can discard a long commit message
if it finds a typo in it so better removing it.

it may happen that an EOF condition is not properly detected during
execution of the handle_messages() method, thus making the waiting step
in process() hang forever.

This adds a check for the EOF condition once every 10 timeouts while
consuming messages.

Related to T3922

black is considered stable since release 22.1.0 and the version
we are currently using is quite outdated and not compatible with
click 8.1.0, so it is time to bump it to its latest stable release.

Please note that E501 pycodestyle warning related to line length
is replaced by B950 one from flake8-bugbear as recommended by black.
https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length

Related to T3922

swh.model.tests.swh_model_data.TEST_OBJECTS["revision"] now has
three sample revisions resulting in two revisions being processed
by kafka consumer in test_client_with_deserializer.

As there is no guarantee on the order messages are processed by
kafka consumer, we need to update test implementation to avoid
failures.

Due to test modules being copied in subdirectories of the
build directory by setuptools, it makes pytest fail by raising
ImportPathMismatchError exceptions when invoked from root
directory of the module.

So ignore the build folder to discover tests.

To install the new hook:

  $ pre-commit install -t commit-msg

namely add 2 metrics:
- swh_journal_client_handle_message_total: (counter) the total number  of kafka messages that have been handled by the client,
- swh_journal_client_status: (gauge) report the current status of the
  kafka consumer (waiting or processing).

Since these metrics will be created by any kafka consumer (any swh.journal.client), tags should be added to the statsd metrics (eg. via the STATS_TAGS env var) to distinguish between the different types of consumers (e.g. hostname, consumer type/role, etc.)

Especially for the swh_journal_client_status to work properly
in an elastic execution environment (docker swarm, k8s, ...) it is
mandatory to add a consumer-specific tag (e.g. the hostname) and make
sure the TTL for the metrics in prometherus-statsd-exporter is pretty
short (e.g. 1h), so that restarted consumers are not kept for too long/forever by statsd-exporter.

this options allows to define a callback which will be called once every
'statistics.interval.ms' ms by rdkafka with a bunch of statistics (as a
json string).

See https://github.com/edenhill/librdkafka/blob/master/STATISTICS.md

This allows to define this callback as a string of the form:

  "path.to.module:function"

it was not used anywhere, so simplify the code a bit.

This prevents flaky test_storage_replayer in swh-storage tests.

and make it the default configuration in the pytest fixture plugin.

since there is no guarantee the "expected" (considered invalid) revision will
be consumed first, the outcome of this test was flaky. So use the client
with 'stop_on_eof' instead of 'stop_after_objects'.

hopefully

and make this function an (optional) constructor argument.
If not given, stick to `kafka_to_value`.

If the returned value is None, it is ignored (not passed to the
`worker_fn` function).

This is needed in order to make it possible for the JournalClient to use
a special value_deserializer implementation that needs the object_type,
for example to make the value_deserializer directly instanciate
BaseModel object.

This will be used by an upcoming refactoring of the storage replayer
that will make sure any BaseModel object coming from the journal is valid,
and log invalid kafka objects in case it's not.

Related to T3693.

may be used to generate a on-disk representation of a Storage, for
example to produce test datasets, etc.

make it consistent with the KafkaJournalWriter.