Use the new "init-admin" command to initialize the database and fallback to the standard postgresql initialization.
The database creation is delegated to the postgresql entrypoint, the extensions and the tables are configured respectively by the init-admin and init commands.

Related to T2736

This should avoid having deprecated warnings in the output logs.

Note that the configuration is unified for all services now except the vault.

The (django) migration steps are happening during the docker `up` phase.
Deposit wise, one migration script (`deposit.0018_migrate_swhids`) is requiring
a storage. As this key got removed, this makes that migration step fail.

This commit adds back that storage key.

If there were no spaces between the dependency name and its pinned version,
the latter was not removed by the getdeps function.

Thibault Allançon authored 4 years ago and Stefano Zacchiroli committed 4 years ago

Reviewers: #reviewers, zack

Reviewed By: #reviewers, zack

Differential Revision: https://forge.softwareheritage.org/D4199

- Rename storage to storage_metadata
- Drop no longer loader-version

Add roles defining permissions to the swh-web Keycloak client.

 - reorder --app to come after the `celery` command
 - drop --without-* which are buggy currently (and not really useful for this
 environment). See https://github.com/celery/celery/pull/6365.

It's a breaking change and the local environements need to be reset

This configuration adds a new swh.storage database and backend using logical
postgresql replication. On startup of the read-replica storage backend, the
replication is configured between both database hosts.

The configuration also overrides the swh.web and swh.vault containers to use the
new read-only replica.

This allows us to wait for a different database, for instance the source of a
replication setup.

This allows reordering entries without getting caught by > vs. >>.

Since swh.core 0.3.0, the database creation (superuser) and database
initialization (user) operations have been split for databases managed using raw
SQL.

- Override the postgresql containers for these services to avoid creating an
empty database.
- Add an entry for the template1 database to .pgpass so other containers can
connect to it
- extend wait_pgsql to take a database name as first argument

replica has another precise meaning in the swh stack, so let's try not
to confuse people.

in their pipeline configuration

context: D3976 for some DVCS loaders now send one object at a time to the
storage.

So this will allow batching calls to the *_missing endpoints (for dvcs loaders
e.g. git loader).

This slighly impacts the package loaders but this should tend towards null.

Prior to this we filtered unknown objects and kept a buffer of those unknown
objects to flush to the storage given a threshold hit.

Now, we will buffer all objects and then filter on said buffer of objects. So
we may increase calls to the *_missing endpoints.

Related to D3976
Related to T2373

no need for a fstring if no placeholder is used.

The webapp does not talk to the objstorage directly.

Also remove the swh-web -> swh-objstorage dependency in the
docker-compose.yml file.

It kind of duplicates mr's features, but it adds some sanity checks
(clean checkout, master branch, showing the diff on each repo)

Recently, for the latest deposit migration, the script needed a storage entry.
If it's not present, the script should fail.

So that may be the reason of the current build failure.

Related to D3153

For some reason, pip used to work without write access to the source
directory, but doesn't anymore; so from now on we'll copy the source
somewhere writeable.

This solves the permission issues we had with private directories
not being available in docker; so swhclean is no longer needed.

it prints 'Metrics updated' every 15 seconds.

it prints 'k.m.a.KafkaManagerActor - Updating internal state...'
every 10 seconds.