This prevents flaky test_storage_replayer in swh-storage tests.

and make it the default configuration in the pytest fixture plugin.

since there is no guarantee the "expected" (considered invalid) revision will
be consumed first, the outcome of this test was flaky. So use the client
with 'stop_on_eof' instead of 'stop_after_objects'.

hopefully

and make this function an (optional) constructor argument.
If not given, stick to `kafka_to_value`.

If the returned value is None, it is ignored (not passed to the
`worker_fn` function).

This is needed in order to make it possible for the JournalClient to use
a special value_deserializer implementation that needs the object_type,
for example to make the value_deserializer directly instanciate
BaseModel object.

This will be used by an upcoming refactoring of the storage replayer
that will make sure any BaseModel object coming from the journal is valid,
and log invalid kafka objects in case it's not.

Related to T3693.

may be used to generate a on-disk representation of a Storage, for
example to produce test datasets, etc.

make it consistent with the KafkaJournalWriter.

Enable to check package documentation can be built without producing
sphinx warnings.

The sphinx environment is designed to be used in continuous integration
in order to prevent breaking documentation build when committing changes.

The sphinx-dev environment is designed to be used inside a full swh
development environment.

Related to T3258

Adding a test that ensures that an exception is raised if the
object passed to write_addition function doesn't have a unique_key.

The author field can be optional in the Release model so handle
that edge case in kafka writer tests.

object_types is missing the newly added extid type.

When the local rdkafka producer queue is full, the producer would crash
with a BufferError. While this condition is rare, it can happen when
running a backfill operation.

When such a condition arises, retry sending the message a few times with
an increasing backoff. When giving up, use the common KafkaDeliveryError
mechanism rather than the plain BufferError.

Its content is outdated and it is no longer needed as tests can
now simply be executed using default test target running pytest.

Since the release of confluent-kafka v1.6.0, some swh-storage tests became
flaky and can hang in consumer fixture destruction when closing the wrapped
kafka consumer.

Explicitly performing the commit operation on the consumer before closing
it removes the hang issue.

prevent unnecessary coupling with swh-model.

Related to T2970.

Related to T2964

It has a major regression for timestamps:
https://github.com/msgpack/msgpack-python/issues/451

1. for consistency with the serializers in swh.core
2. it's not clear to me that we ever needed that (it was added in
   64cd01fc with no rational)
3. it messes with swh-search reading from the journal and expecting
   JSON-like data.

to have support for datetime/timestamp.

previous revision did in fact killed the dependencu on swh.core but did
not update the rquirements file accordingly.

The encoding of datetime objects is done using msgpack extension types
(instead of former b'swhtype" custom types).

Note that this imply that the timezone of the encoded datime is "lost in
translation": the resulting datetime will be a (tz-aware) UTC datetime
object.

Support for decoding these later is however provided for backward
compatbility.

Also make the serialization code of the journal independant from
swh.core.api.serializer.

This later move aims at making the msgpack serialization process of the
journal to be specified (and predictable). The code from swh.core is
dedicated at RPC and can thus do much more "custom types" than what is
needed for the journal. So by not using the serialization code from the
swh.core pakage, we make sure we do not inadvertently encode unspecified
objects in the journal.

Related to T2834

So that errors in unique_key() are caught by tests using the in-mem writer.

The indexer-storage will use this.

Instead of being a method of KafkaJournalWriter.

This will allow using JournalWriters for values that aren't model objects.

The old attribute was deprecated in swh.model 0.7.2

We don't use @overload in this module anymore.

swh.journal.writer.kafka imports it from serializers.

Related to T2610

Related to T2610

kafka_to_value didn't recursively convert to tuples, so they were
of type Tuple[List[bytes], ...] instead of Tuple[Tuple[bytes, bytes], ...]

Following swh.model 0.6.6 bump