The http_retry decorator has been moved to swh-core package in order
to ease its reuse across swh packages.

In particular, there seems to be a negligeable number of origins
using SSH instead of HTTPS, which the git loader cannot deal with.

Related to T3781

That means detected github urls {https,git,http}://github.com/${user_repo}(.git) are
canonicalized to https://github.com/${user_repo} format.

This avoids duplication of origins.

Related to T4232

Related to T4232

Without the following, and the new swh.scheduler bump, some dependencies were
no longer resolved properly.

Related to T2746

This also modifies the test dataset to filename with url decoded filename.
As this is what the latest pytest plugin requires.

This is no longer shared between the new debian loader and the lister.

The swh.storage.schemata module is still part of the swh.storage module though.
As this is still a dependency for the current swh.loader.debian production
loader. This will be cleaned up later.

Related D2135

Listers are declared as plugins via the `swh.workers` entry_point.

As such, the registry function is expected to return a dict with the
`task_modules` field (as for generic worker plugins), plus:

- `lister`: the lister class,
- `models`: list of SQLAlchemy models used by this lister,
- `init` (optionnal): hook (callable) used to initialize the lister's state
  (typically, create/initialize the database for this lister).
  If not set, the default implementation creates database tables (after
  optionally having deleted exisintg ones) according to models declared in
  the `models` register field.

There is no need for explicitely add lister task modules in the main
`conftest` module, but any new/extra lister to be tested must be registered
(the tested lister module must be properly installed in the test environment).

Also refactor a bit the cli tools:
- add support for the standard --config-file option at the 'lister' group
  level,
- move the --db-url to the 'lister' group,
- drop the --lister option for the `swh lister db-init` cli tool:
  initializing (especially with --drop-tables) the database for a single
  lister is unreliable, since all tables are created using a sibgle MetaData
  (in the same namespace).

pytest<4 because of https://github.com/pytest-dev/pytest/issues/4641

Related T1138

Sushant Sushant authored 7 years ago and Nicolas Dandrimont committed 7 years ago

This work is based on Sushant's internship and D229.

Related T613