Scheduler temporary backend has been removed in favor of a
more efficient memory backend.

It enables to track last lister execution date and will be used to schedule
first visits with high priority for listed origins.

Related to swh-scheduler#4687.

For now this information is not used downstream, but it can be useful
for specific analysis or one-shot scheduling.

The oldest part of the scheduler API was updated to use model classes
(based on attr package) instead of dictionaries in order to improve
typing.

In order to simplify the testing of listers, allow to call the run command
of swh-lister CLI without scheduler configuration. In that case a temporary
scheduler instance with a postgresql backend is created and used.

It enables to easily test a lister with the following command:

$ swh -l DEBUG lister run <lister_name> url=<forge_url>

The http_retry decorator has been moved to swh-core package in order
to ease its reuse across swh packages.

In particular, there seems to be a negligeable number of origins
using SSH instead of HTTPS, which the git loader cannot deal with.

Related to T3781

That means detected github urls {https,git,http}://github.com/${user_repo}(.git) are
canonicalized to https://github.com/${user_repo} format.

This avoids duplication of origins.

Related to T4232

Related to T4232

Without the following, and the new swh.scheduler bump, some dependencies were
no longer resolved properly.

Related to T2746

This also modifies the test dataset to filename with url decoded filename.
As this is what the latest pytest plugin requires.

This is no longer shared between the new debian loader and the lister.

The swh.storage.schemata module is still part of the swh.storage module though.
As this is still a dependency for the current swh.loader.debian production
loader. This will be cleaned up later.

Related D2135

Listers are declared as plugins via the `swh.workers` entry_point.

As such, the registry function is expected to return a dict with the
`task_modules` field (as for generic worker plugins), plus:

- `lister`: the lister class,
- `models`: list of SQLAlchemy models used by this lister,
- `init` (optionnal): hook (callable) used to initialize the lister's state
  (typically, create/initialize the database for this lister).
  If not set, the default implementation creates database tables (after
  optionally having deleted exisintg ones) according to models declared in
  the `models` register field.

There is no need for explicitely add lister task modules in the main
`conftest` module, but any new/extra lister to be tested must be registered
(the tested lister module must be properly installed in the test environment).

Also refactor a bit the cli tools:
- add support for the standard --config-file option at the 'lister' group
  level,
- move the --db-url to the 'lister' group,
- drop the --lister option for the `swh lister db-init` cli tool:
  initializing (especially with --drop-tables) the database for a single
  lister is unreliable, since all tables are created using a sibgle MetaData
  (in the same namespace).

pytest<4 because of https://github.com/pytest-dev/pytest/issues/4641

Related T1138

Sushant Sushant authored 7 years ago and Nicolas Dandrimont committed 7 years ago

This work is based on Sushant's internship and D229.

Related T613