Related to T1410

We stopped using it months ago.

Looks like kombu finally axed python2 support.

Related to T2610

Related to T2610

flake8 hook has been removed from https://github.com/pre-commit/pre-commit-hooks
so now use the one from https://gitlab.com/pycqa/flake8

by moving import statements in functions and using conditional
import of typechecking modules (especially StorageInterface which
triggers the loading of 300+ modules).

Related to T2575.

pytest wastes a lot of time in .hypothesis and .git; this commit excludes them.

Related to T645

Related to T2494

Defining the pytest-plugin though the pytest-plugin [1] makes it loaded by default.
This creates loading issues on modules depending on scheduler but not on
the pytest plugin scheduler exposes as explained in the doc [2]

Instead we'll explicitely define to modules depending on the pytest plugins in
their root conftest [3]:

pytest_plugins = [ "swh.scheduler.pytest_plugin" ]

[1] https://docs.pytest.org/en/stable/writing_plugins.html#setuptools-entry-points

[2] https://docs.pytest.org/en/stable/writing_plugins.html#plugin-discovery-order-at-tool-startup

[3] https://docs.pytest.org/en/stable/writing_plugins.html#requiring-loading-plugins-in-a-test-module-or-conftest-file

Related to D3475
Related to T2484

This was needed for celery 4.4.4 but that version is not used anymore.

This allows us to reuse these fixtures in other modules without brittle
swh.scheduler.tests.conftest star imports.

Unfortunately, we can't really override pytest fixtures from one plugin to
another. We therefore reimplement the fixtures provided by celery, inlining the
static configuration and renaming them to our names in the process.

This also adds a backwards-compatibility import from pytest_plugin to conftest,
to allow old users of the conftest fixtures to keep working.

This avoids having to run `from swh.scheduler.tests.conftest import *` in other
modules, e.g. swh.lister, to import and use the swh_scheduler pytest fixture.

This paginated endpoint allows retrieving information about the origins recorded
by listers.

We need to use our serialization hook recursively to make sure that we can
deserialize nested data structures.

This new API endpoint allows listers to record the origins they have seen during
their current run.

Origins are identified by the lister instance, the url of the origin, and the
type of loader that should be used to load this origin.

The implementation allows listers just send the list of origins they've
seen (with some lightweight extra information), leaving the backend to handle
whether to do an insertion or an update to an existing origin.

The current implementation doesn't disable origins that have disappeared when
doing a full listing run. This step will be done by a separate "origin garbage
collection" endpoint, which will peruse the `last_seen` field.

This lets us keep the tests a little DRYer.

This splits primary keys across "automatic" primary keys (handled by the
database) and manual primary keys (managed by the user). Use the opportunity to
improve/clarify the documentation of field metadata attributes.

This adds a pair a functions to the backend:
 - `get_or_create_lister` pulls the record for a given lister from the database
 - `update_lister` updates the record for a given lister in the database

This is one of the basic building blocks for the integration of lister
information directly in the scheduler database.

Related to T2442.

This allows us to automatically serialize/deserialize exceptions under this base
class within our RPC framework.

Alongside swh.model.model, this allows us to define data models for the objects
the scheduler is working with, and to serialize/deserialize these objects
transparently at the RPC layer.

This also introduces some mild ORM-like logic so we can keep the actual SQL a
little DRYer.

This new subcommand has two commands:

 - ping: checks whether the given worker instance answers within a given timeout
 - list-running: lists running tasks on the given worker instance

The logging module is already initialized by the main swh.core cli; This only
creates double logging with no advantages whatsoever.

When running the cli with default settings (i.e. pointing to a
softwareheritage-scheduler-dev database), and the database doesn't exist, an
OperationalError is raised.

This shouldn't prevent (some of the) cli subcommands from working, so catch this
error and ignore it as one of the scheduler backend setup failure modes.

We've been using pytest-postgresql for... a year (4117d5ab).