Messages are now first sent to rabbitmq then postgresql.

In the nominal case where all writes are ok, that changes nothing vs the
previous implementation (postgresql first then rabbitmq).

In degraded performance though, that's supposedly better.

1. If we cannot write to rabbitmq, then we won't write to postgresql either,
that function will raise and stop.

2. If we can write to rabbitmq first, then the messages will be consumed
independently from this. And then, if we cannot write to postgresql (for some
reason), then we just lose the information we sent the task already. This
means the same task will be rescheduled and we'll have a go at it again. As
those kind of tasks are supposed to be idempotent, that should not a major
issue for their upstream.

Also, those tasks are mostly listers now and they have a state management of
their own, so that should definitely mostly noops (if the ingestion from the
previous run went fine). Edge cases scenario like down site will behave as
before.

Refs. swh/infra/sysadm-environment#5512

To explicit its current behavior.

Refs. swh/infra/sysadm-environment#5512

Bump development tools: mypy, codespell, isort, ...

Move all tools configuration in pyproject.toml.

Remove no longer needed mypy overrides.

Antoine Lambert authored 2 months ago and Antoine Lambert committed 1 month ago

In the CSV file consumed by the schedule command, allow to use the celery
backend name as task type name because the mapping between a backend name
and its task type name can be easily retrieved from the scheduler API and
only the celry backend name is available in sentry events data.

An origin can have be listed by the bulk-save lister but never
scheduled so we need to handle that case to avoid errors when
attempting to schedule priority first visits.

A memory backend was recently introduced so that temporary backend
relying on a postgresql server is no longer needed.

From now on requests to the scheduler remote API will be retried when
encountering connection errors and transient remote exceptions.

These have been deprecated for ages now.

The former has been deprecated for ages now.

Normalize the scheduler db for swh.core 3.6 with improved `swh db`
handling capabilities.

Remove test_init.py, it's now outdated.

This log message serves as a crude healt check so we keep it but we make it a bit more
interesting.

Refs. swh/devel/swh-scheduler#4687

Refs. swh/devel/swh-scheduler#4687

Refs. swh/devel/swh-scheduler#4687

Refs. swh/devel/swh-scheduler#4687

This also makes the function return the number of scheduled origins.

Refs. swh/devel/swh-scheduler#4687

The current opened cli was not looping. In effect, doing one round, schedule origins and
then crash in production-like environment.

There is no issue in the docker environment as the loop is implemented outside the
pre-existing cli. This kept said cli to avoid breaking the docker environment.

Refs. swh/devel/swh-scheduler#4687

This new command in the origin group enables to schedule first
visits with high priority for origins registered by listers having
the first_visits_priority_queue attribute set.

The command ensures the visits of all origins registered by such
listers will be scheduled with high priority after the first listing
regardless if some have already been scheduled prior it.

Subsequent executions of such listers will no longer trigger visits
with high priority though, those will be scheduled by the recurrent
visits runner.

Related to #4687.

It allows to return the set of visit types from the origins listed
by a specific lister.

Related to #4687.

This new optional parameter enables to only return listers whose first
visits of listed origins must be scheduled with high priority after a
first listing but were not scheduled yet.

Those types of listers have the first_visits_queue_prefix attribute set.

Related to #4687.

In order to implement a new scheduler runner that will schedule first
visits of listed origins with high priority, add the following new
columns to the Lister model:

- last_listing_finished_at: Timestamp at which the last execution of
  the lister finished

- first_visits_queue_prefix: Optional prefix of message queue names
  to schedule first visits with high priority

- first_visits_scheduled_at: Timestamp at which all the first visits
  of listed origins with high priority were scheduled

Related to #4687.

It exist cases (for instance when running tests on Jenkins) where
more than one log record is captured during that test, making it
flaky.

This should allow a failed task to return the reason for the failure and
make it possible to be displayed to the end user, especially for
Save-Code-Now.

Related to swh/devel/swh-web#4805

This expands the schema to add fork (boolean yes/no/unknown) and fork
source (URL field) if available from the listers, which will be usable for
scheduling heuristics (e.g. for GitHub).

Antoine Lambert authored 8 months ago and Antoine Lambert committed 8 months ago

It enables to efficiently filter by URLs the origins recorded by a lister.

Related to swh/devel/swh-web#4802.

Antoine Lambert authored 10 months ago and Antoine Lambert committed 8 months ago

Database table priority_ratio and function swh_scheduler_nb_priority_tasks
are no longer used by swh-scheduler interface so better removing them.

Add missing description of ids argument notably.

Related to swh/devel/swh-web#4802.

Not all celery tasks are recorded in the task_run table, but the celery
listener needs to be able to process all events, even for tasks it
doesn't know. The stricter type checking has broken this, coincidental,
ignoring.

Closes: #4688

Wrap calls to attr.as_dict and attr.as_tuple in methods to_dict and
to_tuple to avoid explicit import of the attr package in client code.