This allows us to only read the kafka topics once instead of twice in
the same tests, which is apparently a hard thing to do in a way
compatible with both confluent-kafka 1.5 and 1.6.

This stops using the origin_visit.type as fallback values as now, the database has been
migrated.

So this makes the origin_visit_status.type a not nullable column.

This also drops now redundant join instructions on origin_visit table when reading.

Related to T2968

Side effect of the following commit in librdkafka 1.6:
https://github.com/edenhill/librdkafka/commit/f418e0f721518d71ff533759698b647cb2e89b80

Tests was relying on a buggy behavior of the mocked kafka cluster: two
subsequent consumers setup with the same group id should receive a
different set of messages, rather than the same set of messages.

Also explicitly commit messages once consumed.

This new integration test checks that, when flushing the buffer storage,
the addition functions of the underlying storage backend are called in
topological order (content, directory, revision, release then snapshot).

This reduces the probability of "data consistency" regressions caused by
the use of the buffering storage proxy alone.

The earlier implementation would only return summary data from keys that
existed in the last `_add` backend method run, rather than collating all
the results.

This is mostly a consistency addition, considering that most (if not
all) loaders will only add a single snapshot.

The common pattern of loading objects in topological order (content >
directory > revision > release > snapshot), then flushing the storage,
is now fully consistent; Without this addition, the snapshot addition
would reach the backend storage before all other objects are added,
leading to potential inconsistencies if the flush of other object types
fails.

This returned a Tuple[OriginVisit, OriginVisitStatus].

This was required to have the missing information "type" for visit-status. This is no
longer needed as now OriginVisitStatus holds the type information.

This returned a Tuple[OriginVisit, OriginVisitStatus] which is no longer needed as now
OriginVisitStatus held the type information now.

Explains:

* when to use swh-web instead
* that `get_storage` should always be used to instantiate the storage
* `StorageInterface`
* model objects
* pagination
* backends

If the type is not present on an origin_visit_status, it
should be computed from the origin_visit.
There were some methods which only return the origin_visit_status
value. It breaks the webapp mangling the type to empty value on
the search result page.

Related to T3001

Related to T2961

useful when the type is not yet populated in the database

Related to T2966

(!) A new database upgrade is needed (165.sql) for postgresql backend

Related to T2964

Since 209de5db, it was failing because of:
TypeError("content_add() got an unexpected keyword argument 'db'")

Depends on D4848
Related to T2443

This allows to enforce a specific version of java to be used. For
example, since cassandra seems not to support java 14 yet, this allows
to run tests on bullseye:

  JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64/ pytest swh

hypothesis 6 upgraded a warning into an error: now raises a FailedHealthCheck
when using a pytest fixture with a @given generative test set.

See https://hypothesis.readthedocs.io/en/latest/healthchecks.html

prevent being fooled by a missing '\n'.

As it's exposed through the swh.storage.pytest_plugin itself used by other swh
modules, this needs to be declared to avoid other swh module build failures.

Related to T2746

requirements-swh.txt already declares the swh.core[db] dependency which
transitively pulls it.

Related to T2746

The previous code would nullify all non-null branches, and try to create a
SnapshotBranch out of null branches.

As this happens for about 50 revisions in the archive.

This module's use of write_addition predated the introduction of reliable
writing in swh.journal; Since this introduction, the backfiller has been
flushing the kafka writer after writing each single object, leading to a 3x
measured slowdown on backfilling contents.

as this happens quite often and isn't an error.

migrate_extrinsic_metadata: add support for guessing the origin of more PyPI packages from filenames.

Because it makes a lot of get requests and doesn't handle failures,
it crashed often.

The metadata migration script kept crashing otherwise.

The next version of swh-journal will remove the default value.

It complicates the signature and the code, and we don't have any use
for it currently.

1. rename branch_info to last_branch
2. exclude the last_branch from 'branches', so that:
   a) it never needs to contain a None value, so we don't need a cast
   b) no need for slicing