Antoine Lambert authored 2 months ago and Antoine Lambert committed 2 months ago

In the CSV file consumed by the schedule command, allow to use the celery
backend name as task type name because the mapping between a backend name
and its task type name can be easily retrieved from the scheduler API and
only the celry backend name is available in sentry events data.

An origin can have be listed by the bulk-save lister but never
scheduled so we need to handle that case to avoid errors when
attempting to schedule priority first visits.

A memory backend was recently introduced so that temporary backend
relying on a postgresql server is no longer needed.

From now on requests to the scheduler remote API will be retried when
encountering connection errors and transient remote exceptions.

These have been deprecated for ages now.

The former has been deprecated for ages now.

Normalize the scheduler db for swh.core 3.6 with improved `swh db`
handling capabilities.

Remove test_init.py, it's now outdated.

This log message serves as a crude healt check so we keep it but we make it a bit more
interesting.

Refs. #4687

Refs. #4687

Refs. #4687

Refs. #4687

This also makes the function return the number of scheduled origins.

Refs. #4687

The current opened cli was not looping. In effect, doing one round, schedule origins and
then crash in production-like environment.

There is no issue in the docker environment as the loop is implemented outside the
pre-existing cli. This kept said cli to avoid breaking the docker environment.

Refs. #4687

This new command in the origin group enables to schedule first
visits with high priority for origins registered by listers having
the first_visits_priority_queue attribute set.

The command ensures the visits of all origins registered by such
listers will be scheduled with high priority after the first listing
regardless if some have already been scheduled prior it.

Subsequent executions of such listers will no longer trigger visits
with high priority though, those will be scheduled by the recurrent
visits runner.

Related to #4687.

It allows to return the set of visit types from the origins listed
by a specific lister.

Related to #4687.

This new optional parameter enables to only return listers whose first
visits of listed origins must be scheduled with high priority after a
first listing but were not scheduled yet.

Those types of listers have the first_visits_queue_prefix attribute set.

Related to #4687.

In order to implement a new scheduler runner that will schedule first
visits of listed origins with high priority, add the following new
columns to the Lister model:

- last_listing_finished_at: Timestamp at which the last execution of
  the lister finished

- first_visits_queue_prefix: Optional prefix of message queue names
  to schedule first visits with high priority

- first_visits_scheduled_at: Timestamp at which all the first visits
  of listed origins with high priority were scheduled

Related to #4687.

It exist cases (for instance when running tests on Jenkins) where
more than one log record is captured during that test, making it
flaky.

This should allow a failed task to return the reason for the failure and
make it possible to be displayed to the end user, especially for
Save-Code-Now.

Related to swh-web#4805

This expands the schema to add fork (boolean yes/no/unknown) and fork
source (URL field) if available from the listers, which will be usable for
scheduling heuristics (e.g. for GitHub).

Antoine Lambert authored 9 months ago and Antoine Lambert committed 9 months ago

It enables to efficiently filter by URLs the origins recorded by a lister.

Related to swh-web#4802.

Antoine Lambert authored 11 months ago and Antoine Lambert committed 9 months ago

Database table priority_ratio and function swh_scheduler_nb_priority_tasks
are no longer used by swh-scheduler interface so better removing them.

Add missing description of ids argument notably.

Related to swh-web#4802.

Not all celery tasks are recorded in the task_run table, but the celery
listener needs to be able to process all events, even for tasks it
doesn't know. The stricter type checking has broken this, coincidental,
ignoring.

Closes: #4688

Wrap calls to attr.as_dict and attr.as_tuple in methods to_dict and
to_tuple to avoid explicit import of the attr package in client code.

Add a new TaskRun class based on attr package in order to improve
scheduler interface typing and code readability.

Add a new Task class based on attr package in order to improve
scheduler interface typing and code readability.

Add a new TaskType class based on attr package in order to improve
scheduler interface typing and code readability.

As we include `README.rst`, having the same introductory paragraph
in `docs/index.rst` resulted in duplication in the documentation
once built.

Since the release of sentry-python 2.0, it is now possible to use the
sentry_events fixture in tests for celery tasks.

swh-core v3.0.1 now allows to initialize sentry with a None dsn so
using a fake DSN for first sentry initialization is no longer needed.

The get_available_slots method already returns the exact available slots
in a queue so there is no need to recompute it with the current
threshold. As a side effect, the queue_size was always 0 when there was no
messages in the queue.

The behavior was also fixed when the queue does not exist, the max_lengh
is returned if provided instead of the default MAX_NUM_TASKS