The previous implementation would always append extra_encoders to the main
ENCODERS list. Refactor encoder and decoder extensions both with a common getter
which doesn't mutate the global variable.

This is to support some of the metadata imported from npm package.jsons in
revisions.

Those db utilities were only used within swh.core. By dropping those, we can
then stop installing pifpaf within tox for the db tests.

The tests are now probably faster than before as well.

This renames the test file to test_pytest_plugin which is clearer since it
relates to the actual module being tested.

This also moves the tests into its own dedicated arborescence because that's a
different module now.

This drops the pytest-postgresql from requirements-db.txt. This is not a
required dependency for the swh.core.db module itself.

It's a requirement for the swh.core.db.pytest_plugin. This module should become
its own module as a test requirements for other swh modules (including
swh.core.db but not limited to it).

So this commit adds a dedicated requirements-db-pytestplugin.txt file with such
dependency. This opens the db-pytestplugin key within the setup.py and then
references that test dependency for the swh.core.db modules (as it's needed for
tests of the module).

Related to T2746

Reraising exceptions in RPCClient was not honored anymore due to a mistake
introduced in last commit.

An exception can be constructed with another exception argument so handle
that special case by adding a dedicated encoder and decoder to avoid remote
exception serialization error.

This explicitely checks the different scenario possible with the current
subcommands (create, init-admin, init).

Using a subset of sql schema representation we have in our existing datasets.

Closes T2741

Both requests.Request and _pytest.FixtureRequest objects are passed to
test fixtures as argument 'request'. This hurts readability.
Adding typing annotations makes it clear which one is meant.

In staging, the db is already created by puppet, the current create database
stanza forced within the create-db cannot work.

Ths subsequent steps of postgresql extensions installation is required though.

This will allow it.

Related to T2736

Fixes build [1]

[1] https://jenkins.softwareheritage.org/job/DENV/job/tests/lastBuild/artifact/swh-core.log

This also adds more tests around the instantiation of such servers.

Defining 'pytest_plugins' in a non-top-level conftest is no longer supported
because it affects the entire directory tree in a non-explicit way.

This will allow to align the Vault's client/server code with the other rpc
servers. And start adding one typed interface with more consistent checks in
between implementations.

This moves the SWHDatabaseJanitor from swh.storage to swh.core. This also adds
tests to it.

This will allow to share the behavior on other swh services with postgres
backends (vault, scheduler, etc...).

All clients of this (loader, lister, indexer...) have migrated to use
`load_from_envvar` or some derivative (pre-load_from_envvar) of it (deposit)
instead.

Related to T1532

That new function declares exactly what happens today during our configuration
loading (for loader/lister/indexer/... and rpc services in general not for the
CLIs which will be dealt with later).

Loads and parses the yaml configuration file out of the SWH_CONFIG_FILENAME
environment variable. Allowing eventual dict enrichment with default
configuration if provided.

Related to T1532

A class should only inherit from UserList if the type of data it presents is
the same as the data in the 'data' attribute, which isn't true here.

This means, for example, that SortedList.__contains__ checked if the value
is in self.data, which always returns False (unless unlucky, but then it
returns True while it shouldn't).

By removing this inheritance, methods that are no longer implemented no longer
default to a buggy implementation.

We used to log all task arguments, which was a bit too much for the logging
pipeline to handle. Now that a lot of tasks arguments have been "canonicalized",
we can go back to logging some specific arguments directly.

Use the opportunity to drop an unused argument and improve the documentation of
these logging-related functions.

This allows us to create databases with names containing dashes, such as the
ones advertised in the documentation of the `swh db create` command.

This matches the functionality advertised as examples in the documentation for
the command.

This adds a "flavor initialization step" to the initialization of the database,
after the execution of the sql file named *-flavor.sql.

This adds a few sanity checks to warn if one of the underlying assumptions for
this feature breaks.

the idea is now to be a bit more explicit, and expect thoses commands
to have the main `swh` click group as parent.

Then the swh.cli.subcommands entrypoint hooks only have to load the
declared cli modules.

So we now expect to:

- use swh.core.cli.swh as parent group when declaring
  sub-groups/commands in a swh package.
- the swh.cli.subcommand entrypoint to only refer the module rather than
  the click group/command, ie. have:

  '''
    [swh.cli.subcommands]
    name = swh.path.to.cli_module
  '''

  instead of:

  '''
    [swh.cli.subcommands]
    name = swh.path.to.cli_module:click_command
  '''

A backward compatilibility mech is provided: if the loaded entrypoint
object is indeed a click command, auto add it to the main swh group.

The idea it to have the 'swh db create' command to create the database
and prepare it by executing SQL -superuser- files, i.e. SQL files which
execution requires superuser permissions -- typically install languages
or extensions.

Then the 'swh db init' execute the initialization SQL files that do not
require superuser permissions.

Also get rid of the previous (unused) 'swh db init' command.

This allows us to reuse an existing connection if needed, and to be more
specific which errors we ignore or not.

When the code was migrated from swh.storage, some other ancillary functions were
merged into this file. So bits of it are under the LGPLv3 (the code pulled from
psycopg2) and bits are under the GPLv3 (the code we wrote).

Related to T2610

Related to T2610

flake8 hook has been removed from https://github.com/pre-commit/pre-commit-hooks
so now use the one from https://gitlab.com/pycqa/flake8

While named similarly, these two CLI utilities used to take quite a different
approach to doing the same thing.

They now both call out the same initialization function for a given module. The
main difference is now the source of information for what databases to
initialize: `swh db init` uses the SWH_CONFIG_FILENAME configuration file, while
`swh db-init` only uses its own command line arguments.