This reverts commit f4805a12.

swh-datasets does not pass CI yet, which prevents swh-docs publication

and add -j4 to the 'mr run pre-commit install' comand.

Related to swh/meta#5075.

Align with production settings and enable vault workers to fetch
contents bytes directly from the objstorage to speedup cooking.

Even when the docker_compose fixture itself fails.

Use a file of installed packages generated at image build time when no
python package is installed from source. This call ti 'pip install' for
all startup of the services can slow the startup time a bit.

Having the swh-web-cron service exit early make the test fail because
docker_compose fixture now waits for the services to be started.

Make it tell which origins are loaded on the fly, with a duration.

Replace the retry proxy by a tenacious one, also add a buffer proxy in
the pipeline (should help, maybe).

Instead of scaling up the loaders if needed in the origins fixture; the
issue with this approach was the time needed by the newly spawn loaders
to be ready to accept tasks from Celery, there is a good chance the
first loader takes (prefetch) them all, thus falling back in a
one-loader-loads-all-origins situation.

Replace the retry proxy by a tenacious one and move the filter one at
the entry of the pipeline.

It should mitigate spurious hash collision errors when loading several
origins in parallel (which happen when there are more than one origin
to load, now that the origins() fixture scales the swh-loader service
up if needed).

This allows to easily query elasticserch, which can be useful for tests.

Removing an object from Kafka requires writing a new message with
the same key as previously used and an empty value. These tombstones
then get later “compacted” depending on the topic settings `cleanup.policy`,
`max.compaction.lag.ms` and `delete.retention.ms`.

To test the presence or absence of objects in Kafka, we thus need to
find which is the most recent: a tombstone or a value. In order to do
so, we parse all messages into a single dict, associating SWHIDs with
the latest message timestamp and if it should be considered present or
absent. This is sadly a bit time and memory consuming but at least we
get accurate results.

While not strictly necessary, we now use a topic configuration in
Kafka that will aggressively try to remove “dead” messages. It
should improve slightly the time needed to inventory objects as
previously described.

We use the match syntax introduced in Python 3.10 in `handle_message()`,
so we bump black compatibility settings to Python 3.11.

Depends on swh-alter!7 (and a new release thereafter)

I prevents from a 301 back and forth on each http query.

It seems still broken albeit replying to HEAD queries...

Starting a service can be pretty slow, waiting for dependencies etc.
Double these to 6 (default being 3) in all service with a healthcheck
defined with default retries.

Otherwise the mimetype and fossology indexers are executed, making
tests very long to execute.

It's not used any more and is about to be removed from swh-indexer.

This adds a new test scenario for the cassandra backend. It includes:
- add a journal_writer in storage_cassandra.yml config file,
- add a simple test in which the cassandra backend is used in a
  simple git loading scenario.

- use a dedicated group_id for the replayer (to prevent confusion),
- use a dedicated cassandra storage configuration to prevent alter tests from
  being subject to side effects of modifications in the main cassandra storage
  configuration,
- simplify a bit the alter_host fixture,
- improve the way verified_origins fixture check for the replayer to be
  done (as kafka about consumer lag).

Make it possible to declare one origin to be loaded with multiple urls
and choose the first one to be accessible as url for the test. This can
prevent the tests from failing due to a transient downtime of an origin.

Use this to make the mercurial origin used in test_mirror iand test_graphql
more robust (sr.ht being currently down).