Adapt it to recent changes in the docker environment and remove a few
deprecated sections.

It seems a deposit can be verified by the time the deposit upload/update
query is done from the CLI.

This allows the service to not exit before the `compose up --wait`
command is completed in the test's `docker_compose` fixture, making this
later fail (it seems there is no way of telling `compose up --wait` not
to consider a completed service as an error).

Instead of registering them in each worker, so it's done only once.

Use the already runnin ``swh-deposit`` service to provide the
``deposit_host`` fixture instead of spawning a short-live service (aka
use ``docker compoer exec`` instead of ``docker compose run``).

Also ensure deposit task types are registered by the scheduler in this
fixture.

Add a simple bash function to wait for an HTTP url to be up and running
(returning an actual 200 status code) rather than just waiting for the
tcp socket to be OK.

This should prevent some resource usage and improve the startup
time and processing time, especially during tests.

This allows to have resolvable vault download URLs when using azurite as
cache backend.

This service allows to retrieve some context information from the
running compose environment.
For now, only served public IP ports for a running service is available.
This allows to adapt the configuration of services that need to
generate public facing URLs (like azurite with vault or keycloak).

Use this service to configure the frontend URL at startup in keycloak.

Put the config for all swh services in /srv/softwareheritage/config.yml;
put the entrypoint.sh script in this directory, and make this later the
default working directory.

This allows to remove many useless config files from the compose files.

Can help introspecting the status of kafka. This seems easier to use
than cmak.

Make it available via the nginx edge router.

This is useful to get the public port used for some services. Use this
to properly configure the keycloak frontend_api on startup.

Ideally, we should also listen to docker events to detect updates in the
public ports used by services.

The 'direct' is not supported any more by grafana, and it plays poorly
in this compose setup.

So it can be kept between restarts of the kafka service.

i.e. print 'COMPOSE_PROJECT_NAME=xxx' and 'COMPOSE_FILE=xxx' so it is easier to
copy/paste in a terminal to play with the compose command in the running test
env (e.g. show the compose logs).

Only expose ports for nginx services using compose's automatic host port
selection. As a consequence, there is no need for patching compose files
any more to run tests.

Also make nginx services to use 80 as internal port (there is no need
for using 508x anywhere).

This requires a small hack in test_vault when using azure as backend,
because the azure service is expected to be accessible from both the
host (where tests are running) and the compose services with the same
url, which is not easily feasible when running azurite in the compose
environment. The issue is, when using azure as backend for vault, it
generates a 301 response with the url of the asset stored in azure as
Location header, but in this context, the URL cannot be resoved from
the host.

It used to be handled by patching the configuration and compose files on
the fly when running tests. The proposed approach is a bit less invasive
(but still not very pretty).

It is now required otherwise swh-web will refuse to start with the
following:

  The SECRET_KEY value matches the development default,
  check the contents of SWH_CONFIG_FILENAME

The wait-for-it tool expect both the host and the port. This later could
be missing when extracted from a configuration with the URL (from which
they are extracted) does not explicitely specify the port.

Introduce a (naive) host-port-from-url shell command in swhutils.sh to
help this.

This fixture is actually only state validation logic making sure the
mirror has been properly replicated. Overloading the 'origins' fixture
with it ensures it's used for every test in the test_mirror.py test set,
especially the 2 tests imported from test_vault.

Also simplify it a bit using the 'docker_compose' fixture in place of the
couple 'docker_host' + 'compose_cmd'.

They are getting removed from the loader itself.

Refs. swh/meta#3781

We now test that `swh alter remove` actually removes content from the
primary objstorage and from extra objstorages. We also test that `swh
alter recovery-bundle restore` actually restores content to the
objstorage.

To do this, we need to compute hashes (only SHA1 for now) of the content
bytes. We extract these bytes from the recovery bundle for
convenience, but this is a fairly slow operation. Therefore we only take
a random subset (currently 5) of the removed Content objects in order
to reduce the tests runtime.

In order to test extra objstorages, we need to also spin a content
replayer. We adapt what was made for the mirror to do so.

Depends swh-alter!15

This was introduced in swh-alter v0.0.6 (see swh/devel/swh-alter!17)

We waited for the replayer after the initial loading of the origins. We
now also wait for the replayer after restoring the recovery bundle.

The required function has been factored out.

originally from Franck Bret <franck.bret@octobus.net> merge request on
swh-environment.

Align with production settings and enable vault workers to fetch
contents bytes directly from the objstorage to speedup cooking.

Even when the docker_compose fixture itself fails.