Since the migration to GitLab, merge request branches are now stored in
the git repository of swh-core.

As the total number of branches now exceeds 1000, multiple Web API calls
are now required to get all branches from the snapshot of that repository
loaded into the test archive.

The hierarchy changed recently.

This is using the new swh container gitlab registry and install such binary.
This demonstrates how to use that new container while allowing dev on the svn
loader (using that new binary) to happen.

Install rpm2cpio and cpio packages in the docker image.

Declare task queue in swh-loader service configuration.

flower service has been removed so we can also remove its reverse proxy
configuration.

As in production settings for the objstorage reverse proxy, set HTTP read
timeout to 1h in order to avoid 504 error (Gateway Time-out) when posting
a large amount of contents to the objects storage.

swh-graphql is now using a UvicornWorker. Change entrypoint to
reflect this.
Add the missing search configuration in the graphql config.

The new swh directory loader will check recursive nar checksum
for downloaded tarball content using the nix-store command if
it is available in execution environment.

Add task queues declaration for following listers: conda, cpan, fedora,
hackage, puppet, rubygems.

Add task queue declaration for following loaders: cpan, hackage, opam,
puppet, rubygems.

Ensure lister task queues are sorted in lexicograhical order.

In order to have a search behavior close to the production one, add services
related to archive search in the default docker-compose.yml file:

  - swh-search: remote search service using a memory backend

  - swh-search-journal-client-objects: journal client feeding swh-search with
    objects loaded into the archive (typically origins)

  - swh-search-journal-client-indexed: journal client feeding swh-search with
    results of swh-indexer processing on archived objects

As a consequence, it simplifies the docker-compose.search.yml file enabling
to use a swh-search service with a more costly elasticsearch backend.

It enables to quickly schedule the loading of listed origins in the
docker environment without having to restart the service dedicated
to this task (swh-scheduler-schedule-recurrent).

Docker now officially supports Docker Compose in the form of a plugin
for the `docker` command.
See: https://www.docker.com/blog/announcing-compose-v2-general-availability/

Following the official Docker documentation for Linux will install the
Docker Compose plugin. We can remove the mention of the `docker-compose`
Debian package.

Command lines and pointers to the documentation are now updated
accordingly.

It enables to update the metrics regarding the number of loaded
origins into the archive.

Those are consumed by the archive coverage widget in the webapp
homepage.

It enables to test those in the docker environment.

Recent versions of pip can now properly install all swh packages
in editable mode plus their dependencies in a single command.

Previously a pip install command was issued for each swh package.

pip 21.3 added support to editable installs for projects that have a
pyproject.toml and use a build backend that supports PEP 660, but mypy
does not detect yet such editable installs and following errors are
reported when type checking swh modules:

  Cannot find implementation or library stub for module named swh...

So temporarily pin pip until mypy properly supports PEP 660 editable
installs.

seems a better idea than sticking to the default behavior (using
the directory name, hence 'docker'), and may even fix the
"network has active endpoints" error on 'compose down'.

make it yield the docker_host on which a check_compose_output lambda has
been added; this allows users of the docker_compose fixture to use it
directly to run `compose` commands in addition to usual commands without
having to explicitly use the compose_cmd fixture.

instead of the old 'docker-compose' tool.

See https://docs.docker.com/compose/#compose-v2-and-the-new-docker-compose-command

mostly for the sake of consistency.

They are now redundant with the pytest-testinfra based ones.

Towards removing the shell based tests for the docker environment,
synchronize the features they cover by checking in Python based test
that all git objects have been correctly loaded into the archive.

Also prefer to use dulwich to manipulate a git repository instead
of git command.

There is no swh directory in swh-environment repository so
the mypy check was failing when committing changes to a
Python test file.

to try to make it a bit less confusing.

it's small and pretty useful

and make them all properly configured with a compact retention policy.

- use 'postgresql' instead of deprecated 'local' cls
- remove deprecated 'args' in the journal_writer section
- activate journal anonymized topics

so that these 2 files are as similar as possible.

Related to T4228

It will be needed to run 'swh db upgrade'

and configure swh-web to use them, making them available from the webapp main page.