When the code was migrated from swh.storage, some other ancillary functions were
merged into this file. So bits of it are under the LGPLv3 (the code pulled from
psycopg2) and bits are under the GPLv3 (the code we wrote).

Related to T2610

Related to T2610

flake8 hook has been removed from https://github.com/pre-commit/pre-commit-hooks
so now use the one from https://gitlab.com/pycqa/flake8

While named similarly, these two CLI utilities used to take quite a different
approach to doing the same thing.

They now both call out the same initialization function for a given module. The
main difference is now the source of information for what databases to
initialize: `swh db init` uses the SWH_CONFIG_FILENAME configuration file, while
`swh db-init` only uses its own command line arguments.

This is no longer used.

This also adds some types on impacted changed functions.

Related to T1532

by moving import statements in functions.

Related to T2575.

pytest wastes a lot of time in .hypothesis and .git; this commit excludes them.

It's used in swh-indexer-storage, but swh-storage is about to stop using it.

This avoids duplication which is growing (right now storage and search proposes
the same pattern in different functions).

Related to T645

Before this commit, decorating with @remote_api_endpoint made
mypy completely unable to detect the type of the decorated function.

This will allow typing paginated endpoints across our modules (swh.storage,
swh.search, swh.indexer, swh.scheduler, ...).

Related to T645

Related to D3460

Related to T2105

Ideally we should refactor those entirely.

Should definitely fix the debian build [1]

[1] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DCORE/job/gbp-buildpackage/118/console

This fixes the debian build yet to come

Instead of the brittle csv we used to use, use postgresql text mode to transfer
data in the `copy` operation. Reference all relevant bits of the PostgreSQL
documentation inline with the code.

This also adds test coverage for most supported types, as well as making sure
all escaping corner cases are covered.

This fixes the ci build [1]

[1] https://jenkins.softwareheritage.org/job/DCORE/job/tests/844/console

There's no reason to allow this, and it's a potential source of
invalid data.

Related to T2367

msgpack only has built-in support for ints that fit in 64 bits. However, we
happen to be storing arbitrary json in the archive, which itself has support for
integers of arbitrary length, which themselves are mapped to "long" integers in
Python, which make the msgpack encoder blow up.

Fortunately, overflowing integers are passed to the default object hook. We
generate a msgpack "extended type" with code 1 for arbitrary integers.

I forgot to run it on files at the root in the previous commit.

The next commit will enable Black; which would break this test,
as lineno() no longer matches the right line.

The contents of the journal up to February 2020 are encoded with this schema, so
we should keep supporting it, until all these objects end up rewritten

msgpack 1.0 disables support for non-str/bytes keys by default, but we use
None as key in the output of swh-storage's snapshot_count_branches;
and we use ints in swh-scheduler.

msgpack now defaults to raw=False.

This commit preserves compatibility with older versions of msgpack.

Some remote api (for instance the scheduler one) does not declare any
extra_encoders in the negotiate decorator. This was raising an error
when querying any declared endpoint.

I forgot to change the async server in
08289ce5.

iso8601 is more strict, and less inefficient.