Wrap calls to attr.as_dict and attr.as_tuple in methods to_dict and
to_tuple to avoid explicit import of the attr package in client code.

Add a new TaskRun class based on attr package in order to improve
scheduler interface typing and code readability.

Add a new Task class based on attr package in order to improve
scheduler interface typing and code readability.

Add a new TaskType class based on attr package in order to improve
scheduler interface typing and code readability.

As we include `README.rst`, having the same introductory paragraph
in `docs/index.rst` resulted in duplication in the documentation
once built.

Since the release of sentry-python 2.0, it is now possible to use the
sentry_events fixture in tests for celery tasks.

swh-core v3.0.1 now allows to initialize sentry with a None dsn so
using a fake DSN for first sentry initialization is no longer needed.

The get_available_slots method already returns the exact available slots
in a queue so there is no need to recompute it with the current
threshold. As a side effect, the queue_size was always 0 when there was no
messages in the queue.

The behavior was also fixed when the queue does not exist, the max_lengh
is returned if provided instead of the default MAX_NUM_TASKS

About a year ago, we updated the way sentry is configured for celery
tasks by not relying anymore on the worker_init signal and the
SWH_SENTRY_DSN environment variable but rather by using a task_prerun
signal callback that sets sentry DSN through a dict mapping a task name
to its sentry DSN.

But as a result it broke our celery integration for sentry (that notably
adds task name and parameters in sentry events) as integrations must be
initialized in the worker_init signal callback to work properly. So ensure
they are by first setting a fake DSN in the worker_init signal callback and
update tests to check celery integration is enabled.

Related to swh/meta#4949.

Remove use of --import-mode=importlib pytest option and move conftest.py
file from swh/scheduler/tests to swh/scheduler to fix tests execution
with latest pytest release.

That somehow breaks the build.

This allows to load the configuration from a file if provided (flag -C/--config-file),
or an environment variable SWH_CONFIG_FILENAME if provided or as a final fallback from
the default configuration already defined in the module.

It's expected as some other swh clis are already loading their configuration.

As we are expecting it to work like other parts of our swh modules.

This cli is used by operator/sysadm. It's renamed to avoid confusion with the new
subcommand send-origins-from-file-to-celery.

Refs. swh/infra/sysadm-environment#4872

It lived in the snippet repository for years (2017 and prior) and used regularly since
then. It's been recently reused for the osdn origins scheduling. And as usual,
improvments on it ensued. It's time to make it an official cli.

It's now migrated under the the 'swh scheduler origin' subcommand. It's name is
'send-origins-from-file-to-celery'. The file to read is either a local file or the
standard input. This expects a list of origins (urls) to be pushed directly in the
proper queue according to the <task-type> argument.

For example:
```
export SWH_CONFIG_FILENAME=~/.config/swh/scheduler.yml; \
  head -20 /tmp/20230509-1539-priority.list.github | \
  shuf | \
  swh scheduler -C $SWH_CONFIG_FILENAME origin send-origins-from-file-to-celery \
    --queue-name-prefix large-repository \  # optional
    --debug \
    --dry-run \
    load-git
```

or directly:
```
swh scheduler -C $SWH_CONFIG_FILENAME origin send-origins-from-file-to-celery \
    --queue-name-prefix large-repository \  # optional
    --debug \
    --dry-run \
    --limit 10 \
    load-git
    /tmp/20230509-1539-priority.list.github  # note that here, the full file is sent,
                                             # without shuffling it
```

Origins can be routed to extra queue with the help of the <--queue-name-prefix> flag.
This will use standard (configured in the scheduler) queue name with the dedicated
prefix (<queue-name-prefix>:<standard-queue-name>). This also expects that the
destination queue is being consumed on the infra side (ping sysadm for it if not).

The cli can be parametric to limit the number of messages with the <--limit> flag. It
can also just be tested with <--dry-run> to do nothing but print actions. Some extra
logging can be triggered with the <--debug> flag.

Refs. swh/infra/sysadm-environment#4872

And let the caller function do the actual display. This also make the utility function
compute only the required data to be displayed (the listing data is optionally
outputted).

This also renames the 'list' bool to 'with_listing'. 'list' is a python keyword (it's ok
to reuse but editors usually color it differently than plain function name and that can
be confusing).

This adapts the print output to expose the success rate even with --watch is activated.
This also change the formatting of the 2-digits float percentage.

Refs. swh/infra/sysadm-environment#5230

The previous default of 10 minutes is too high a period.

Refs. swh/infra/sysadm-environment#5230

Refs. swh/infra/sysadm-environment#5230

Related to swh/meta#5075.

This didn't actually fail in CI because the dash at the start of the
line, when either slow or !slow were enabled, ignores failures.

Typing stubs for click and flask are now maintained by upstreams
so remove their outdated typing packages.

And replace comment type annotations by explicit ones.

Convert README from markdown to ReST to make it embeddable in
docs/index.rst

David Douard authored 1 year ago and vlorentz committed 1 year ago

This is now handled by the worker plugin system.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

When using that backend, a temporary scheduler database is spawned
then removed when the backend gets destroyed.

It can be used for testing SWH components that require a scheduler
instance (listers for instance).

The issue has been solved for some time now.

Nicolas Dandrimont authored 1 year ago and Antoine R. Dumont committed 1 year ago

This also improves:
- the spacing between time period, making it optional.
- the units can be simple up to the full units (e.g h, hr, hrs, hour, hours...)

This should ease rescheduling origins manually in staging for testing purposes.

Because the register task type routine does not provide the value, it's left unchecked.
Once a new lister starts listing origins, the scheduler keeps on scheduling new origins
in the queue without limits. As the consumption may be slower than the production, that
tends towards too much resources usage in rabbitmq.

This should limit the issue for new deployments.

The swh.lister package was required as testing dependency to check
registration of celery tasks for listers through plugins declared
in swh.workers entrypoints.

However, it is easy to create a fake lister in tests data to check
its correct registration through scheduler CLI so that swh.lister
dependency is not really needed.

As a consequence, worker plugins are now discovered in the function
register_task_types from the swh.scheduler.cli.task_type module,
previoulsy it was done when importing the module.