- add a JournalWriterInterface in a new interface.py module,
- add a flush() method to this interface,
- remove (now unused) write_update methods,
- move all type-related code in this module,
- fix InMemoryJournalWriter type annotations,
- remove usage of the TValue TypeVar in favor of using ValueProtocol
  directly.

for StreamJournalWriter and InMemoryJournalWriter, and make them
consistent with KafkaJournalWriter.

That hook can be frustrating as it can discard a long commit message
if it finds a typo in it so better removing it.

it may happen that an EOF condition is not properly detected during
execution of the handle_messages() method, thus making the waiting step
in process() hang forever.

This adds a check for the EOF condition once every 10 timeouts while
consuming messages.

Related to T3922

black is considered stable since release 22.1.0 and the version
we are currently using is quite outdated and not compatible with
click 8.1.0, so it is time to bump it to its latest stable release.

Please note that E501 pycodestyle warning related to line length
is replaced by B950 one from flake8-bugbear as recommended by black.
https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length

Related to T3922

swh.model.tests.swh_model_data.TEST_OBJECTS["revision"] now has
three sample revisions resulting in two revisions being processed
by kafka consumer in test_client_with_deserializer.

As there is no guarantee on the order messages are processed by
kafka consumer, we need to update test implementation to avoid
failures.

Due to test modules being copied in subdirectories of the
build directory by setuptools, it makes pytest fail by raising
ImportPathMismatchError exceptions when invoked from root
directory of the module.

So ignore the build folder to discover tests.

To install the new hook:

  $ pre-commit install -t commit-msg

namely add 2 metrics:
- swh_journal_client_handle_message_total: (counter) the total number  of kafka messages that have been handled by the client,
- swh_journal_client_status: (gauge) report the current status of the
  kafka consumer (waiting or processing).

Since these metrics will be created by any kafka consumer (any swh.journal.client), tags should be added to the statsd metrics (eg. via the STATS_TAGS env var) to distinguish between the different types of consumers (e.g. hostname, consumer type/role, etc.)

Especially for the swh_journal_client_status to work properly
in an elastic execution environment (docker swarm, k8s, ...) it is
mandatory to add a consumer-specific tag (e.g. the hostname) and make
sure the TTL for the metrics in prometherus-statsd-exporter is pretty
short (e.g. 1h), so that restarted consumers are not kept for too long/forever by statsd-exporter.

this options allows to define a callback which will be called once every
'statistics.interval.ms' ms by rdkafka with a bunch of statistics (as a
json string).

See https://github.com/edenhill/librdkafka/blob/master/STATISTICS.md

This allows to define this callback as a string of the form:

  "path.to.module:function"

it was not used anywhere, so simplify the code a bit.

This prevents flaky test_storage_replayer in swh-storage tests.

and make it the default configuration in the pytest fixture plugin.

since there is no guarantee the "expected" (considered invalid) revision will
be consumed first, the outcome of this test was flaky. So use the client
with 'stop_on_eof' instead of 'stop_after_objects'.

hopefully

and make this function an (optional) constructor argument.
If not given, stick to `kafka_to_value`.

If the returned value is None, it is ignored (not passed to the
`worker_fn` function).

This is needed in order to make it possible for the JournalClient to use
a special value_deserializer implementation that needs the object_type,
for example to make the value_deserializer directly instanciate
BaseModel object.

This will be used by an upcoming refactoring of the storage replayer
that will make sure any BaseModel object coming from the journal is valid,
and log invalid kafka objects in case it's not.

Related to T3693.

may be used to generate a on-disk representation of a Storage, for
example to produce test datasets, etc.

make it consistent with the KafkaJournalWriter.

Enable to check package documentation can be built without producing
sphinx warnings.

The sphinx environment is designed to be used in continuous integration
in order to prevent breaking documentation build when committing changes.

The sphinx-dev environment is designed to be used inside a full swh
development environment.

Related to T3258

Adding a test that ensures that an exception is raised if the
object passed to write_addition function doesn't have a unique_key.

The author field can be optional in the Release model so handle
that edge case in kafka writer tests.

object_types is missing the newly added extid type.

When the local rdkafka producer queue is full, the producer would crash
with a BufferError. While this condition is rare, it can happen when
running a backfill operation.

When such a condition arises, retry sending the message a few times with
an increasing backoff. When giving up, use the common KafkaDeliveryError
mechanism rather than the plain BufferError.

Its content is outdated and it is no longer needed as tests can
now simply be executed using default test target running pytest.

Since the release of confluent-kafka v1.6.0, some swh-storage tests became
flaky and can hang in consumer fixture destruction when closing the wrapped
kafka consumer.

Explicitly performing the commit operation on the consumer before closing
it removes the hang issue.

prevent unnecessary coupling with swh-model.

Related to T2970.

Related to T2964

It has a major regression for timestamps:
https://github.com/msgpack/msgpack-python/issues/451

1. for consistency with the serializers in swh.core
2. it's not clear to me that we ever needed that (it was added in
   64cd01fc with no rational)
3. it messes with swh-search reading from the journal and expecting
   JSON-like data.

to have support for datetime/timestamp.

previous revision did in fact killed the dependencu on swh.core but did
not update the rquirements file accordingly.