- compress the syntactic overview, notably removing mostly redundant
grammar elements wrt YAML
- remove or rephrase confusing elements about types, instances and singletons

- do not overly abbreviate some terms and be more consistent
- update remark about existing config handling
- be more explicit on the relation to YAML, notably regarding references
- rephrase presentation of singletons
- be more explicit on what the new system provides (updates the
rationale which aims to present that)

This fixes build [1]

[1] https://jenkins.softwareheritage.org/view/swh%20master/job/DCORE/job/tests/1183/console

It exists cases where a custom exception type cannot be rebuilt from its
serialized form.

As a fallback, build a base Exception with the message of the serialized
exception.

decoding custom format is still supported for backward compat.

Note that this revision modifies the exception type raised when attempting to
encode a naive datetime object for both formats (json amd msgpack): it now
raises a TypeError instead of a ValueError.

to make it clear which codecs are specific to the json format.

instead of json.dumps(..., cls=SWHJSONEncoder) and so.

This simplify a bit the code and make sure the json_dumps/loads
functions are actually tested.

it should not be used anymore.

we should be free of them by now. The journal has been recreated from
scratch since then.

This allows users to pass multiple --log-level values to the swh cli, to allow
finer grained configuration of the logging of some modules, without having to
resort to a full-fledged, very verbose log config file.

Close T2819.

We now match the command line output per section, in a fuzzier way than before.

The previous implementation would always append extra_encoders to the main
ENCODERS list. Refactor encoder and decoder extensions both with a common getter
which doesn't mutate the global variable.

This is to support some of the metadata imported from npm package.jsons in
revisions.

Those db utilities were only used within swh.core. By dropping those, we can
then stop installing pifpaf within tox for the db tests.

The tests are now probably faster than before as well.

This renames the test file to test_pytest_plugin which is clearer since it
relates to the actual module being tested.

This also moves the tests into its own dedicated arborescence because that's a
different module now.

This drops the pytest-postgresql from requirements-db.txt. This is not a
required dependency for the swh.core.db module itself.

It's a requirement for the swh.core.db.pytest_plugin. This module should become
its own module as a test requirements for other swh modules (including
swh.core.db but not limited to it).

So this commit adds a dedicated requirements-db-pytestplugin.txt file with such
dependency. This opens the db-pytestplugin key within the setup.py and then
references that test dependency for the swh.core.db modules (as it's needed for
tests of the module).

Related to T2746

Reraising exceptions in RPCClient was not honored anymore due to a mistake
introduced in last commit.

An exception can be constructed with another exception argument so handle
that special case by adding a dedicated encoder and decoder to avoid remote
exception serialization error.

This explicitely checks the different scenario possible with the current
subcommands (create, init-admin, init).

Using a subset of sql schema representation we have in our existing datasets.

Closes T2741

Both requests.Request and _pytest.FixtureRequest objects are passed to
test fixtures as argument 'request'. This hurts readability.
Adding typing annotations makes it clear which one is meant.

In staging, the db is already created by puppet, the current create database
stanza forced within the create-db cannot work.

Ths subsequent steps of postgresql extensions installation is required though.

This will allow it.

Related to T2736

Fixes build [1]

[1] https://jenkins.softwareheritage.org/job/DENV/job/tests/lastBuild/artifact/swh-core.log

This also adds more tests around the instantiation of such servers.

Defining 'pytest_plugins' in a non-top-level conftest is no longer supported
because it affects the entire directory tree in a non-explicit way.

This will allow to align the Vault's client/server code with the other rpc
servers. And start adding one typed interface with more consistent checks in
between implementations.

This moves the SWHDatabaseJanitor from swh.storage to swh.core. This also adds
tests to it.

This will allow to share the behavior on other swh services with postgres
backends (vault, scheduler, etc...).

All clients of this (loader, lister, indexer...) have migrated to use
`load_from_envvar` or some derivative (pre-load_from_envvar) of it (deposit)
instead.

Related to T1532

That new function declares exactly what happens today during our configuration
loading (for loader/lister/indexer/... and rpc services in general not for the
CLIs which will be dealt with later).

Loads and parses the yaml configuration file out of the SWH_CONFIG_FILENAME
environment variable. Allowing eventual dict enrichment with default
configuration if provided.

Related to T1532

A class should only inherit from UserList if the type of data it presents is
the same as the data in the 'data' attribute, which isn't true here.

This means, for example, that SortedList.__contains__ checked if the value
is in self.data, which always returns False (unless unlucky, but then it
returns True while it shouldn't).

By removing this inheritance, methods that are no longer implemented no longer
default to a buggy implementation.

We used to log all task arguments, which was a bit too much for the logging
pipeline to handle. Now that a lot of tasks arguments have been "canonicalized",
we can go back to logging some specific arguments directly.

Use the opportunity to drop an unused argument and improve the documentation of
these logging-related functions.