Prior to this, we considered not to do it.  However, we do have some listers
which are not able to list origins with a last update. And we still need to be
able to schedule those origins nonetheless hence this change.

Refs. swh/infra/sysadm-environment#4971

Previously, it was not possible to create sequentially two oneshot tasks
whose only differ by their next_run value.

Related to swh/devel/swh-web#4548

This will allow to schedule add-forge-now requests to different queues. For example [1]
will send git tasks to the add_forge_now_slow:swh.loader.git... queue.

[1]
```
swh ... add-forge-now ... \
  --preset $ENVIRONMENT \
  schedule-first-visits \
    --type-name git \
    --prefix-queue add_forge_now_slow
```

Otherwise, listing types without a 'list-%-full', 'list-%-incremental' pattern (e.g
list-cgit, ...) are systematically delayed 1 day the first time the add-forge-now
schedules them.

Refs. swh/infra/sysadm-environment#4845

Add a task_prerun celery signal handler in order to set Sentry DSN based
on task name or package name.

The mapping between a task/package name and its DSN must be stored in
configuration under a "sentry_settings_for_celery_tasks" key.

For this feature to work, no SWH_SENTRY_DSN and SWH_MAIN_PACKAGE environment
variables should be defined as they override the sentry_dsn and main_package
values passed to init_sentry function.

Related to swh/meta#4949.

Refs. swh/infra/sysadm-environment#4813

Related to swh/meta#4960

Related to swh/meta#4959

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

When not provided, this uses the logging configuration coded in the scheduler (as
before).

Refs. swh/infra/sysadm-environment#4524

Because of that missing comma, an exception was raised (tuple object is
not callable) but it was caught and displayed by the _print_errors decorator
so tests could not detect it.

As a consequence, the logging configuration of celery workers was broken.

Add a test to check if an exception was raised by the setup_log_handler
function to avoid bad surprises when deploying to production or in
docker.

vlorentz authored 2 years ago and vlorentz committed 2 years ago

It is used by the pytest plugin

This will allow proper logging configuration for the services which are currently
running in the dynamic infrastructure. Their logs are current written in the wrong
elasticsearch indices.

Ref. swh/infra/sysadm-environment#4524

Otherwise, at some point, this will get discarded as per the debian build
warning [1]

[1] https://jenkins.softwareheritage.org/view/swh-debian%20(draft)/job/debian/job/packages/job/DSCH/job/gbp-buildpackage/182/console

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496

This will ease scheduling of new add-forge-now requests, on:
- staging: this will list a subset of disabled origins once
- production: this will register recurring tasks (full, incremental if any) to list that
  new forge

This also unifies the previous subcommand schedule-first-visits with the --preset flag.

So, the following would be enough to list appropriately in staging/production:

```
swh scheduler add-forge-now \
( --preset [production|staging] \ # to enable a pre-defined set of rules )
  register-lister \
    gitea \
    url=https://git.afpy.org/api/v1/
```

Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674

This should ease scheduling the first visits for add-forge-now request.

The following would be enough to fetch and schedule the forge just listed (be it in
production or staging):

```
swh scheduler add-forge-now \
  schedule-first-visits \
    --visit-type git \
    (--visit-type svn \ # if a lister lists multiple kinds of visit, we can mention it )
    --lister-name gitea \
    --lister-instance-name git.afpy.org \
    ( --production | --staging )  # to list only enabled | disabled origins
```

Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674

Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674

Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674

This allows to bypass the lister id retrieval step using directly the name and instance
name of the lister to discover the uuid.

This also drops the --lister-uuid flag which is somewhat difficult to use.

Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674

The scheduled_cooldown only applies to tasks that have not been executed
yet. absolute_cooldown avoids archiving objects faster than that.

Avoids inconsistencies between the database state and an ongoing
recurrent task scheduler.

Recent celery versions generate serialized messages with mime types incompatible
with older versions when using msgpack

- pre-commit from 4.1.0 to 4.3.0,
- codespell from 2.2.1 to 2.2.2,
- black from 22.3.0 to 22.10.0 and
- flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).

This currently fails all swh related builds which depend on the celery/kombu stack due
to that dependency's latest version release.

Related to T4545

The send_visits_for_visit_type function uses a default schedule backoff
of 20 minutes where there is no origins to schedule for a given visit
type.

It exists use cases when we would like that schedule backoff to be
shorter in order to schedule listed origins for loading into the
archive more rapidly, typically in the docker environment.

So allow to set that backoff value through configuration.

Those files have been renamed so the database could not be filled.

This adds type to the function, update its docstring and clarify its associated tests as
well.

and replace usage of the "local" scheduler cls with "postgresql"
everywhere.

Attribute current_version is already set and directly used by swh db
[version|init|upgrade] clis.

Related to T4305

instead of (soon-to-be-deprecated) swh-core's postgresql_fact one.

Add a new enabled_only parameter set to True by default in
get_listed_origins scheduler method.

It enables to filter out by default disabled listed origins
when requesting the result of a listing and avoid possible
errors in listers implementation.

Fix "more than one target found for cross-reference 'Origin'"
sphinx warning.