may be used to generate a on-disk representation of a Storage, for
example to produce test datasets, etc.

make it consistent with the KafkaJournalWriter.

Enable to check package documentation can be built without producing
sphinx warnings.

The sphinx environment is designed to be used in continuous integration
in order to prevent breaking documentation build when committing changes.

The sphinx-dev environment is designed to be used inside a full swh
development environment.

Related to T3258

Adding a test that ensures that an exception is raised if the
object passed to write_addition function doesn't have a unique_key.

The author field can be optional in the Release model so handle
that edge case in kafka writer tests.

object_types is missing the newly added extid type.

When the local rdkafka producer queue is full, the producer would crash
with a BufferError. While this condition is rare, it can happen when
running a backfill operation.

When such a condition arises, retry sending the message a few times with
an increasing backoff. When giving up, use the common KafkaDeliveryError
mechanism rather than the plain BufferError.

Its content is outdated and it is no longer needed as tests can
now simply be executed using default test target running pytest.

Since the release of confluent-kafka v1.6.0, some swh-storage tests became
flaky and can hang in consumer fixture destruction when closing the wrapped
kafka consumer.

Explicitly performing the commit operation on the consumer before closing
it removes the hang issue.

prevent unnecessary coupling with swh-model.

Related to T2970.

Related to T2964

It has a major regression for timestamps:
https://github.com/msgpack/msgpack-python/issues/451

1. for consistency with the serializers in swh.core
2. it's not clear to me that we ever needed that (it was added in
   64cd01fc with no rational)
3. it messes with swh-search reading from the journal and expecting
   JSON-like data.

to have support for datetime/timestamp.

previous revision did in fact killed the dependencu on swh.core but did
not update the rquirements file accordingly.

The encoding of datetime objects is done using msgpack extension types
(instead of former b'swhtype" custom types).

Note that this imply that the timezone of the encoded datime is "lost in
translation": the resulting datetime will be a (tz-aware) UTC datetime
object.

Support for decoding these later is however provided for backward
compatbility.

Also make the serialization code of the journal independant from
swh.core.api.serializer.

This later move aims at making the msgpack serialization process of the
journal to be specified (and predictable). The code from swh.core is
dedicated at RPC and can thus do much more "custom types" than what is
needed for the journal. So by not using the serialization code from the
swh.core pakage, we make sure we do not inadvertently encode unspecified
objects in the journal.

Related to T2834

So that errors in unique_key() are caught by tests using the in-mem writer.

The indexer-storage will use this.

Instead of being a method of KafkaJournalWriter.

This will allow using JournalWriters for values that aren't model objects.

The old attribute was deprecated in swh.model 0.7.2

We don't use @overload in this module anymore.

swh.journal.writer.kafka imports it from serializers.

Related to T2610

Related to T2610

kafka_to_value didn't recursively convert to tuples, so they were
of type Tuple[List[bytes], ...] instead of Tuple[Tuple[bytes, bytes], ...]

Following swh.model 0.6.6 bump

Related to T2501

Required, to have the 'object_type' attribute on metadata object classes.

TEST_OBJECT_DICTS isn't used in any other package.

instead of dicts.

This is needed because we are beginning to use converters in swh.model,
so the dict used as input is not guaranteed to be the exact same one
as the one retrieved from the journal.

For this, we need to include the data attribute in journal_data's
Content objects so we have "true" Content objects in tests.
Which then requires a few adjustments in journal tests because the
Content's data is never sent to the journal.