This explicitely checks the different scenario possible with the current
subcommands (create, init-admin, init).

Using a subset of sql schema representation we have in our existing datasets.

Closes T2741

Both requests.Request and _pytest.FixtureRequest objects are passed to
test fixtures as argument 'request'. This hurts readability.
Adding typing annotations makes it clear which one is meant.

In staging, the db is already created by puppet, the current create database
stanza forced within the create-db cannot work.

Ths subsequent steps of postgresql extensions installation is required though.

This will allow it.

Related to T2736

Fixes build [1]

[1] https://jenkins.softwareheritage.org/job/DENV/job/tests/lastBuild/artifact/swh-core.log

This also adds more tests around the instantiation of such servers.

Defining 'pytest_plugins' in a non-top-level conftest is no longer supported
because it affects the entire directory tree in a non-explicit way.

This will allow to align the Vault's client/server code with the other rpc
servers. And start adding one typed interface with more consistent checks in
between implementations.

This moves the SWHDatabaseJanitor from swh.storage to swh.core. This also adds
tests to it.

This will allow to share the behavior on other swh services with postgres
backends (vault, scheduler, etc...).

All clients of this (loader, lister, indexer...) have migrated to use
`load_from_envvar` or some derivative (pre-load_from_envvar) of it (deposit)
instead.

Related to T1532

That new function declares exactly what happens today during our configuration
loading (for loader/lister/indexer/... and rpc services in general not for the
CLIs which will be dealt with later).

Loads and parses the yaml configuration file out of the SWH_CONFIG_FILENAME
environment variable. Allowing eventual dict enrichment with default
configuration if provided.

Related to T1532

A class should only inherit from UserList if the type of data it presents is
the same as the data in the 'data' attribute, which isn't true here.

This means, for example, that SortedList.__contains__ checked if the value
is in self.data, which always returns False (unless unlucky, but then it
returns True while it shouldn't).

By removing this inheritance, methods that are no longer implemented no longer
default to a buggy implementation.

We used to log all task arguments, which was a bit too much for the logging
pipeline to handle. Now that a lot of tasks arguments have been "canonicalized",
we can go back to logging some specific arguments directly.

Use the opportunity to drop an unused argument and improve the documentation of
these logging-related functions.

This allows us to create databases with names containing dashes, such as the
ones advertised in the documentation of the `swh db create` command.

This matches the functionality advertised as examples in the documentation for
the command.

This adds a "flavor initialization step" to the initialization of the database,
after the execution of the sql file named *-flavor.sql.

This adds a few sanity checks to warn if one of the underlying assumptions for
this feature breaks.

the idea is now to be a bit more explicit, and expect thoses commands
to have the main `swh` click group as parent.

Then the swh.cli.subcommands entrypoint hooks only have to load the
declared cli modules.

So we now expect to:

- use swh.core.cli.swh as parent group when declaring
  sub-groups/commands in a swh package.
- the swh.cli.subcommand entrypoint to only refer the module rather than
  the click group/command, ie. have:

  '''
    [swh.cli.subcommands]
    name = swh.path.to.cli_module
  '''

  instead of:

  '''
    [swh.cli.subcommands]
    name = swh.path.to.cli_module:click_command
  '''

A backward compatilibility mech is provided: if the loaded entrypoint
object is indeed a click command, auto add it to the main swh group.

The idea it to have the 'swh db create' command to create the database
and prepare it by executing SQL -superuser- files, i.e. SQL files which
execution requires superuser permissions -- typically install languages
or extensions.

Then the 'swh db init' execute the initialization SQL files that do not
require superuser permissions.

Also get rid of the previous (unused) 'swh db init' command.

This allows us to reuse an existing connection if needed, and to be more
specific which errors we ignore or not.

When the code was migrated from swh.storage, some other ancillary functions were
merged into this file. So bits of it are under the LGPLv3 (the code pulled from
psycopg2) and bits are under the GPLv3 (the code we wrote).

Related to T2610

Related to T2610

flake8 hook has been removed from https://github.com/pre-commit/pre-commit-hooks
so now use the one from https://gitlab.com/pycqa/flake8

While named similarly, these two CLI utilities used to take quite a different
approach to doing the same thing.

They now both call out the same initialization function for a given module. The
main difference is now the source of information for what databases to
initialize: `swh db init` uses the SWH_CONFIG_FILENAME configuration file, while
`swh db-init` only uses its own command line arguments.

This is no longer used.

This also adds some types on impacted changed functions.

Related to T1532

by moving import statements in functions.

Related to T2575.

pytest wastes a lot of time in .hypothesis and .git; this commit excludes them.

It's used in swh-indexer-storage, but swh-storage is about to stop using it.

This avoids duplication which is growing (right now storage and search proposes
the same pattern in different functions).

Related to T645

Before this commit, decorating with @remote_api_endpoint made
mypy completely unable to detect the type of the decorated function.

This will allow typing paginated endpoints across our modules (swh.storage,
swh.search, swh.indexer, swh.scheduler, ...).

Related to T645

Related to D3460