Related to T3079

This is trying to make bin/install (again) a click-button developer setup
installation tool, simplifying the process for newcomers and making it faster
for everyone.

To that end:

- we ensure that pip is up to date before using it for anything else

- we install the prerequisites (flake8, mypy, etc.) to avoid having to name
them one by one in user documentation

- we install packages one by one, rather them all together, to avoid that pip
failures force users to restart the whole process from scratch. In passing,
this makes also things much easier for the new pip resolver. To avoid
dependency inconsistencies we install packages in bin/pip-swh-packages, which
is already supposed to be topologically sorted

- we also install packages only *once*, rather than first without -with-testing
and later with it (as currently documented in the developer setup), because the
"[testing]" markers are extra requires, which always depend on the base
package (hopefully)

When this gets landed, the developer setup can be shortened to just say "run
bin/install".

So as to be able to play around with keycloak and deposit together.

Related to T2858

Since Keycloak 12, it is no longer possible to use a custom theme from
a directory symbolic link, so make a raw copy of the swh theme in the
Keycloak theme folder instead.

Related Keycloak issue: https://issues.redhat.com/browse/KEYCLOAK-17048

Related to T3042

Fix the build [1]

Loaders have been reworked to only deal with configuration as constructor parameters. As
the current docker configuration is shared amongst all loaders, this can no longer work.

The current "next-gen" loaders share a subset of those configuration though so most can
run together. Except for the loader deposit which needs dedicated extra keys (deposit,
url).

Note that some configuration keys (scheduler for example) referenced in the current
configuration are not for loaders. This make instantiation fails. So they need to be
removed.

All in all, trying to separate the dedicated deposit loader with its configuration in
its own container and let the other loaders running as before fixes the build.

That and stop referecing the scheduler configuration in the loader configuration.
Instead use a dedicated environment variable to specify the scheduler url to use.

Related to T1410

[1] https://jenkins.softwareheritage.org/view/all/job/swh-docker-dev/730/console

rDENV5db372fcb5abf45f3a95545029e525a2a149a91c introduced the use of the
healthcheck feature from docker-compose.

The healtcheck feature was introduced with version 2.1 of the Compose
file format.

So bump Compose file format version to 2.1 in compose files in order
to avoid the following error when using docker-compose < 1.27:

"depends_on contains an invalid type, it should be an array"

Services acting as journal clients were always exiting prematurely due
to not created kafa topics they were trying to subscribe to.

So ensure needed topics will be created when starting the kafka service
and wait for their creation before starting the journal client services.

As a consequence, this makes a call to "docker-compose up -d" command a
little bit longer before it returns.

flower has always been deactivated and kafka-manager is not mandatory to run
the swh stack.

Since swh-lister v0.8.0, lister implementations no longer require a
dedicated PostgreSQL database so remove related configuration and
fix swh-lister entrypoint.

Related to T2978

This will allow to run a search journal client subscribed to
`swh.journal.indexed.origin_intrinsic_metadata` topic

Related to D4701

Related to T2871

This adds the `object_types` entry which is now mandatory.

Related to D4701

This was initially changed because the assumption was the vault tests were
directly using the vault backend api. They are not, they are using the web api.

So there was actually nothing to change in the test_vault.py as the web vault
api got fixed and released.

This reverts commit 84e927d2.

Names got changed in the vault api backend. Unfortunately, this passed
undetected both by the vault and the webapp tests.

This commit fixes the situation by adapting to from the old names to the new
ones (this should no longer happen and new tests on the vault should make sure
this is caught early).

Use the new "init-admin" command to initialize the database and fallback to the standard postgresql initialization.
The database creation is delegated to the postgresql entrypoint, the extensions and the tables are configured respectively by the init-admin and init commands.

Related to T2736

This should avoid having deprecated warnings in the output logs.

Note that the configuration is unified for all services now except the vault.

The (django) migration steps are happening during the docker `up` phase.
Deposit wise, one migration script (`deposit.0018_migrate_swhids`) is requiring
a storage. As this key got removed, this makes that migration step fail.

This commit adds back that storage key.

If there were no spaces between the dependency name and its pinned version,
the latter was not removed by the getdeps function.

Thibault Allançon authored 4 years ago and Stefano Zacchiroli committed 4 years ago

Reviewers: #reviewers, zack

Reviewed By: #reviewers, zack

Differential Revision: https://forge.softwareheritage.org/D4199

- Rename storage to storage_metadata
- Drop no longer loader-version