- Jan 31, 2023
-
-
Antoine R. Dumont authored
When not provided, this uses the logging configuration coded in the scheduler (as before). Refs. swh/infra/sysadm-environment#4524
-
- Jan 30, 2023
-
-
Antoine R. Dumont authored
-
- Jan 26, 2023
-
-
Antoine Lambert authored
Because of that missing comma, an exception was raised (tuple object is not callable) but it was caught and displayed by the _print_errors decorator so tests could not detect it. As a consequence, the logging configuration of celery workers was broken. Add a test to check if an exception was raised by the setup_log_handler function to avoid bad surprises when deploying to production or in docker.
-
- Jan 25, 2023
-
-
vlorentz authored
-
- Jan 23, 2023
-
-
Antoine R. Dumont authored
This will allow proper logging configuration for the services which are currently running in the dynamic infrastructure. Their logs are current written in the wrong elasticsearch indices. Ref. swh/infra/sysadm-environment#4524
-
- Jan 02, 2023
-
-
Antoine R. Dumont authored
Otherwise, at some point, this will get discarded as per the debian build warning [1] [1] https://jenkins.softwareheritage.org/view/swh-debian%20(draft)/job/debian/job/packages/job/DSCH/job/gbp-buildpackage/182/console
-
- Dec 19, 2022
-
-
Antoine Lambert authored
In order to remove warnings about /apidoc/*.rst files being included multiple times in toc when building full swh documentation, prefer to include module indices only when building standalone package documentation. Also include them the proper sphinx way. Related to T4496
-
- Dec 08, 2022
-
-
Antoine R. Dumont authored
This will ease scheduling of new add-forge-now requests, on: - staging: this will list a subset of disabled origins once - production: this will register recurring tasks (full, incremental if any) to list that new forge This also unifies the previous subcommand schedule-first-visits with the --preset flag. So, the following would be enough to list appropriately in staging/production: ``` swh scheduler add-forge-now \ ( --preset [production|staging] \ # to enable a pre-defined set of rules ) register-lister \ gitea \ url=https://git.afpy.org/api/v1/ ``` Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674
-
- Dec 07, 2022
-
-
Antoine R. Dumont authored
This should ease scheduling the first visits for add-forge-now request. The following would be enough to fetch and schedule the forge just listed (be it in production or staging): ``` swh scheduler add-forge-now \ schedule-first-visits \ --visit-type git \ (--visit-type svn \ # if a lister lists multiple kinds of visit, we can mention it ) --lister-name gitea \ --lister-instance-name git.afpy.org \ ( --production | --staging ) # to list only enabled | disabled origins ``` Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674
-
- Dec 06, 2022
-
-
Antoine R. Dumont authored
This allows to bypass the lister id retrieval step using directly the name and instance name of the lister to discover the uuid. This also drops the --lister-uuid flag which is somewhat difficult to use. Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674
- Oct 25, 2022
-
-
Nicolas Dandrimont authored
The scheduled_cooldown only applies to tasks that have not been executed yet. absolute_cooldown avoids archiving objects faster than that.
-
Nicolas Dandrimont authored
Avoids inconsistencies between the database state and an ongoing recurrent task scheduler.
-
Nicolas Dandrimont authored
Recent celery versions generate serialized messages with mime types incompatible with older versions when using msgpack
-
- Oct 18, 2022
-
-
David Douard authored
- pre-commit from 4.1.0 to 4.3.0, - codespell from 2.2.1 to 2.2.2, - black from 22.3.0 to 22.10.0 and - flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies. Also change flake8's repo config to github (the gitlab mirror being outdated).
-
- Oct 03, 2022
-
-
Antoine R. Dumont authored
This currently fails all swh related builds which depend on the celery/kombu stack due to that dependency's latest version release.
-
- Sep 23, 2022
-
-
Antoine R. Dumont authored
Related to T4545
-
- Sep 15, 2022
-
-
Antoine Lambert authored
The send_visits_for_visit_type function uses a default schedule backoff of 20 minutes where there is no origins to schedule for a given visit type. It exists use cases when we would like that schedule backoff to be shorter in order to schedule listed origins for loading into the archive more rapidly, typically in the docker environment. So allow to set that backoff value through configuration.
-
- Aug 22, 2022
-
-
Antoine Lambert authored
Those files have been renamed so the database could not be filled.
-
- Jul 29, 2022
-
-
Antoine R. Dumont authored
This adds type to the function, update its docstring and clarify its associated tests as well.
-
- Jul 08, 2022
-
-
David Douard authored
and replace usage of the "local" scheduler cls with "postgresql" everywhere.
-
- Jun 03, 2022
-
-
Antoine R. Dumont authored
Attribute current_version is already set and directly used by swh db [version|init|upgrade] clis. Related to T4305
-
- May 31, 2022
-
-
David Douard authored
instead of (soon-to-be-deprecated) swh-core's postgresql_fact one.
-
- May 12, 2022
-
-
Antoine Lambert authored
Add a new enabled_only parameter set to True by default in get_listed_origins scheduler method. It enables to filter out by default disabled listed origins when requesting the result of a listing and avoid possible errors in listers implementation.
-
- May 09, 2022
-
-
Pratyush authored
-
- May 06, 2022
-
-
Antoine Lambert authored
Fix "more than one target found for cross-reference 'Origin'" sphinx warning.
-
- Apr 28, 2022
-
-
Benoit Chauvet authored
-
- Apr 27, 2022
-
- Apr 26, 2022
-
-
vlorentz authored
-
vlorentz authored
This will allow loaders to guess the forge type, and use the right API to fetch extrinsic metadata for the origin from the forge.
-
David Douard authored
-
- Apr 21, 2022
-
-
vlorentz authored
It feels off as an object method; and I am going to make it use joins in a future commit, so it makes more sense this way.
-
vlorentz authored
-
Antoine Lambert authored
That hook can be frustrating as it can discard a long commit message if it finds a typo in it so better removing it.
-
- Apr 20, 2022
-
-
David Douard authored
Add support for a configuration option "scheduling_policy" in the config file loaded by the 'swh scheduler schedule-recurrent' command. This config entry allows to specify the scheduling policies used by the schedule-recurrent tool, instead of having them hardcoded in the source code. A visit type policy config entry should have at least a 'weight' value for each policy. Default values are unchanged. Eg.: scheduling_policy: git: - policy: already_visited_order_by_lag weight: 55 tablesample: 0.5 - policy: never_visited_oldest_update_first weight: 45 tablesample: 0.5 Note: there may not be configuration entries for all visit types, but if a visit type policy is configured, the config entry should be complete (in other words, the merging of the configuration with the default values is only done at first config level).
-
- Apr 08, 2022
-
-
Antoine Lambert authored
-
Antoine Lambert authored
Related to T3922
-