Refs. swh/meta#5048

Remove overridden scheduling policies for opam origins as default ones
have been updated in swh-scheduler source code.

Override scheduling policies for git origins by using the default ones
without table sampling as listed_origins table in scheduler database
has usually a small size in docker environment and thus a low sampling
value does not return any origins when querying the table.

Refs. swh/infra/sysadm-environment#4984

In order for the origin loading tasks resulting from a listing to be
successfully scheduled by that service, the loading task types must
have been registered in the scheduler database prior the execution of
the main command of that service.

So make the service depends on the swh-loader one and use a Python script
to check the celery worker used to execute loading tasks have been started.

Each worker has its own search instance otherwise which can lead to
surprising results when searching from the webapp (different results
between two successive searches).

Remove the swh-loader-opam service as it is no longer needed.

Ensure listed opam origins get scheduled by updating scheduling
policies configuration as those origins do not have a last_update
date available.

As mypy does not support editable installs for projects that have a
pyproject.toml and use a build backend that supports PEP 660, force
the legacy behavior of editable installs by passing the configuration
settings "editable_mode=compat" to the build backend.

It enables to remove the pip version restriction in the swh venv and
avoid bad surprises when the mypy pre-commit hook is called when
committing in a swh repository.

See https://github.com/python/mypy/issues/13392 for more details.

Related to swh/meta#5043.

python:3.7 upgraded itself to debian bookworm, so openjdk 11 isn't
available anymore. Pull it from eclipse-temurin instead.

We don't need the nix binaries anymore, so drop that.

Related to swh-web#4746

It prevents connection errors when cooking a large bundle.

With these changes, GraphQL web client can use Keycloak to login a user
in order to perform authenticated calls to the GraphQL API.

Related to swh-graphql#4652

It enables to test graphql authentication feature in the docker environment.

It is a lightweight service and is getting ready for production so
better moving it in the main docker-compose file.

Also update configuration as there is no keycloak service available
by default in the docker environment and bump max query cost as
it is a testing environment.

Related to swh/infra/sysadm-environment#4803

Use the same mypy version as in tox files for swh packages.

Related to swh/meta#4960

This script does the following:
 - check if the repo is hosted in the SWH GitLab
 - check if the repo is already forked, then forks it
 - if requested, adds Jenkins as Developer
 - if requested, sets up a `forked` remote to the forked repository

Add two new config entries
1: introspection: to enable/disable schema introspection
2: max_query_cost: to reject too big queries

Since the migration to GitLab, merge request branches are now stored in
the git repository of swh-core.

As the total number of branches now exceeds 1000, multiple Web API calls
are now required to get all branches from the snapshot of that repository
loaded into the test archive.

The hierarchy changed recently.

This is using the new swh container gitlab registry and install such binary.
This demonstrates how to use that new container while allowing dev on the svn
loader (using that new binary) to happen.

Install rpm2cpio and cpio packages in the docker image.

Declare task queue in swh-loader service configuration.

flower service has been removed so we can also remove its reverse proxy
configuration.

As in production settings for the objstorage reverse proxy, set HTTP read
timeout to 1h in order to avoid 504 error (Gateway Time-out) when posting
a large amount of contents to the objects storage.

swh-graphql is now using a UvicornWorker. Change entrypoint to
reflect this.
Add the missing search configuration in the graphql config.

The new swh directory loader will check recursive nar checksum
for downloaded tarball content using the nix-store command if
it is available in execution environment.

Add task queues declaration for following listers: conda, cpan, fedora,
hackage, puppet, rubygems.

Add task queue declaration for following loaders: cpan, hackage, opam,
puppet, rubygems.

Ensure lister task queues are sorted in lexicograhical order.

In order to have a search behavior close to the production one, add services
related to archive search in the default docker-compose.yml file:

  - swh-search: remote search service using a memory backend

  - swh-search-journal-client-objects: journal client feeding swh-search with
    objects loaded into the archive (typically origins)

  - swh-search-journal-client-indexed: journal client feeding swh-search with
    results of swh-indexer processing on archived objects

As a consequence, it simplifies the docker-compose.search.yml file enabling
to use a swh-search service with a more costly elasticsearch backend.

It enables to quickly schedule the loading of listed origins in the
docker environment without having to restart the service dedicated
to this task (swh-scheduler-schedule-recurrent).

Docker now officially supports Docker Compose in the form of a plugin
for the `docker` command.
See: https://www.docker.com/blog/announcing-compose-v2-general-availability/

Following the official Docker documentation for Linux will install the
Docker Compose plugin. We can remove the mention of the `docker-compose`
Debian package.

Command lines and pointers to the documentation are now updated
accordingly.