The new swh directory loader will check recursive nar checksum
for downloaded tarball content using the nix-store command if
it is available in execution environment.

Add task queues declaration for following listers: conda, cpan, fedora,
hackage, puppet, rubygems.

Add task queue declaration for following loaders: cpan, hackage, opam,
puppet, rubygems.

Ensure lister task queues are sorted in lexicograhical order.

In order to have a search behavior close to the production one, add services
related to archive search in the default docker-compose.yml file:

  - swh-search: remote search service using a memory backend

  - swh-search-journal-client-objects: journal client feeding swh-search with
    objects loaded into the archive (typically origins)

  - swh-search-journal-client-indexed: journal client feeding swh-search with
    results of swh-indexer processing on archived objects

As a consequence, it simplifies the docker-compose.search.yml file enabling
to use a swh-search service with a more costly elasticsearch backend.

It enables to quickly schedule the loading of listed origins in the
docker environment without having to restart the service dedicated
to this task (swh-scheduler-schedule-recurrent).

Docker now officially supports Docker Compose in the form of a plugin
for the `docker` command.
See: https://www.docker.com/blog/announcing-compose-v2-general-availability/

Following the official Docker documentation for Linux will install the
Docker Compose plugin. We can remove the mention of the `docker-compose`
Debian package.

Command lines and pointers to the documentation are now updated
accordingly.

It enables to update the metrics regarding the number of loaded
origins into the archive.

Those are consumed by the archive coverage widget in the webapp
homepage.

It enables to test those in the docker environment.

Recent versions of pip can now properly install all swh packages
in editable mode plus their dependencies in a single command.

Previously a pip install command was issued for each swh package.

pip 21.3 added support to editable installs for projects that have a
pyproject.toml and use a build backend that supports PEP 660, but mypy
does not detect yet such editable installs and following errors are
reported when type checking swh modules:

  Cannot find implementation or library stub for module named swh...

So temporarily pin pip until mypy properly supports PEP 660 editable
installs.

seems a better idea than sticking to the default behavior (using
the directory name, hence 'docker'), and may even fix the
"network has active endpoints" error on 'compose down'.

make it yield the docker_host on which a check_compose_output lambda has
been added; this allows users of the docker_compose fixture to use it
directly to run `compose` commands in addition to usual commands without
having to explicitly use the compose_cmd fixture.

instead of the old 'docker-compose' tool.

See https://docs.docker.com/compose/#compose-v2-and-the-new-docker-compose-command

mostly for the sake of consistency.

They are now redundant with the pytest-testinfra based ones.

Towards removing the shell based tests for the docker environment,
synchronize the features they cover by checking in Python based test
that all git objects have been correctly loaded into the archive.

Also prefer to use dulwich to manipulate a git repository instead
of git command.

There is no swh directory in swh-environment repository so
the mypy check was failing when committing changes to a
Python test file.

to try to make it a bit less confusing.

it's small and pretty useful

and make them all properly configured with a compact retention policy.

- use 'postgresql' instead of deprecated 'local' cls
- remove deprecated 'args' in the journal_writer section
- activate journal anonymized topics

so that these 2 files are as similar as possible.

Related to T4228

It will be needed to run 'swh db upgrade'

and configure swh-web to use them, making them available from the webapp main page.

nodejs 12.x reached EOL so it is time to upgrade it.

Related to T4259

the value '1.' is not a valid float representation in yaml, making the
prometheur-statsd-exporter service fail.

It will be used by the git loader for metrics representing the ratio
of ignored refs.

Add swh-graphql to the Dockerfile with default configuration

When installed swh-loader-metadata hooks itself in the loader process,
and fetches metadata for origins created from a known lister (currently
only github).

Note that by default, requests are unauthenticated, so they will be
throttled after 60 requests per hour, causing the git loader to be
stuck while waiting for the rate-limit to expire.