for runner and listener commands.

related to T1410

ie. being able to give the celery config dict as parameter.

related to T1410

so that one can type (typically in a venv, with services running in dockers):

  (venv) swh-environment$ export SWH_SCHEDULER_URL=http://127.0.0.1:5008
  (venv) swh-environment$ swh-scheduler task-type list

and is not needed any more there.

also ensure args dict does not have default db settings (unsupported by
the RemoteScheduler class).

when deleting a click.option, one would better delete the function argument
as well...

eg. ImportErrors when negotiate is not installed.

and strip these urls as well.

It makes no sense to register 2 different URLs when they are equal to a
trailing ws detail.
As a result, we must preaggregate them because the swh_cache_put() sql
function won't allow several 'on conflict' for the same id.

also implement the test with more precise expected behavior, especially the
content pre-aggregation of events and the url stripping.

As is, this test will fail, the pre-aggregation and url stripping being
implemented in the following revision.

we can now use directly the implementation of the copy_to() method from
swh.core's BaseDb, so we just have to extract the format_query as a simple
function (which it should have been since the beginning).

Adapt updater/backend.py acordingly.

- use the same config system as the main backend, with the same conventions,
- do not make SchedulerUpdaterBackend inherit from DbBackend, use a simple
  association, similar to the SchedulerBackend class,
- refactor the ghtorrent the same (explicit config),
- same for the updater writer
- move their main functions in cli.py
- adapt tests accordingly

adapt the configuration to latest refactorings (especially the 'scheduling_db'
config entry which is now just 'db').

Also refactor the scheduler tests to simplify the test class a bit: constants
used in the test do not need to be in the setUp of the test class, making this
setUp method useless.

Doing so, ensure to use simple dicts as test constants instead of
copy-the-modify them. This is (unit)test file, the dumbest, the better.

Move the config file loading in the main cli group so that every
command in this group have a consistent config loading behavior.

This means that some cli commands "signatures" have changed:

- every command now accepts a -C/--config-file option
- the --cls has been dropped: either you give a config file, or passing
  a --database or --url option determine the 'class' of backend to use,
- the api-server command 'config-path' argument has been dropped (use
  --config-file instead),

instead of looking the config file for this entity in a hard to find dedicated
file, expect a config object (dict) as constructor kwargs and look for its
config under the 'elasticsearch' section.

it's now in cli.py

so it uses a proper connection pool to access the database.

so it uses a proper connection pool to access the database.

do only report the number of scheduled tasks at info level, if any, and
lower the 'Run ready tasks' log message at debug level.

when respawning several tasks at once.

Note sure whether we should be concerned by the fact this sometimes occurs
(under heavy load in the docker env).

especially the infamous psycopg2 one.

It's going very fast now, enough to empty its queue between
two scheduler-runner runs.

This will work around the current debian package build failure

Related T1498

Works around https://github.com/ClearcodeHQ/pytest-postgresql/issues/16

instead of replicating the same logging code everywhere.

This reverts commit f267f454.

Pushed by mistake.

see https://github.com/pytest-dev/pytest/issues/4641
It should be possible to let this constraint go when celery 4.3 is out

also remove now useless celery_testing.py and scheduler_testing.py files.

this uses the pytest-postgresql package to manage the database life cycle.

the result backend must be configured to be able to save (celery) group
results (as it is now the case for lister tasks for example).

that allows to respawn any task immediately (or any later date).

so this method can be used to respawn a task immediately.

and not only pending tasks.

to display also the status and priority fields.