Add a new cli endpoint to schedule recurrent visits in Celery
For each known visit type, we run a loop which:
- monitors the size of the relevant celery queue
- schedules more visits of the relevant type once the number of available slots goes over a given threshold (currently set to 5% of the max queue size).
The scheduling of visits combines multiple scheduling policies, for now
using static ratios set in the POLICY_RATIOS
dict. We emit a warning
if the ratio of origins fetched for each policy is skewed with respect
to the original request (allowing, for now, manual adjustement of the
ratios).
The CLI endpoint spawns one thread for each visit type, which all handle connections to RabbitMQ and the scheduler backend separately. For now, we handle exceptions in the visit scheduling threads by (stupidly) respawning the relevant thread directly. We should probably improve this to give up after a specific number of tries.
Co-authored-by: Antoine R. Dumont (@ardumont) ardumont@softwareheritage.org
Related to swh/infra/sysadm-environment#3667 (closed)
Test Plan
Migrated from D6520 (view on Phabricator)
Merge request reports
Activity
Build has FAILED
Patch application report for D6520 (id=23701)
Rebasing onto 243a69fc...
Current branch diff-target is up to date.
Changes applied before test
commit 36e313851fb1ba14d6b365232085a12fca551f8a Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Oct 20 16:47:05 2021 +0200 Bootstrap orchestrator in charge of scheduling recurring tasks Related to swh/infra/sysadm-environment#3667
Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/467/ See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/467/console
Build has FAILED
Patch application report for D6520 (id=23709)
Rebasing onto 243a69fc...
Current branch diff-target is up to date.
Changes applied before test
commit 94ea07639ed3c79c3968eac3e8ce4a05b3d53d92 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Oct 20 16:47:05 2021 +0200 Bootstrap orchestrator in charge of scheduling recurring tasks Related to swh/infra/sysadm-environment#3667
Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/468/ See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/468/console
Build has FAILED
Patch application report for D6520 (id=23710)
Rebasing onto 243a69fc...
Current branch diff-target is up to date.
Changes applied before test
commit 3ba4aec9afc7ca3ae4844d11daf1469f1ce32af4 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Oct 20 16:47:05 2021 +0200 Bootstrap orchestrator in charge of scheduling recurring tasks Related to swh/infra/sysadm-environment#3667
Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/469/ See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/469/console
Build is green
Patch application report for D6520 (id=23711)
Rebasing onto 243a69fc...
Current branch diff-target is up to date.
Changes applied before test
commit e66923ef0b1449f277f40466539192c340c3afaa Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Oct 20 16:47:05 2021 +0200 Bootstrap orchestrator in charge of scheduling recurring tasks Related to swh/infra/sysadm-environment#3667
See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/470/ for more details.
Build has FAILED
Patch application report for D6520 (id=23716)
Rebasing onto 243a69fc...
Current branch diff-target is up to date.
Changes applied before test
commit 0a5cc81a0ca28edc2fffaa9f9fd7ee060565020f Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Oct 20 16:47:05 2021 +0200 Bootstrap orchestrator in charge of scheduling recurring tasks Related to swh/infra/sysadm-environment#3667
Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/471/ See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/471/console
Build has FAILED
Patch application report for D6520 (id=23742)
Rebasing onto 243a69fc...
Current branch diff-target is up to date.
Changes applied before test
commit 505446ab47248a12779a36616694cf34f0704694 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Oct 20 16:47:05 2021 +0200 Bootstrap orchestrator in charge of scheduling recurring tasks This new module is specifically in charge of scheduling the recurring tasks (loader ones, either dvcs loader like: load-git, load-svn, or package loader: pypi, npm, ..., etc...). This new cli call [1] will replace the manual processes started on scheduler nodes (saatchi, scheduler0.staging). - [1] ``` swh scheduler start-orchestrator ``` Related to swh/infra/sysadm-environment#3667
Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/472/ See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/472/console
Build is green
Patch application report for D6520 (id=23743)
Rebasing onto 243a69fc...
Current branch diff-target is up to date.
Changes applied before test
commit b29ef4254ce31f32fe9a374d5cacf0284dd9cbdc Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Oct 20 16:47:05 2021 +0200 Bootstrap orchestrator in charge of scheduling recurring tasks This new module is specifically in charge of scheduling the recurring tasks (loader ones, either dvcs loader like: load-git, load-svn, or package loader: pypi, npm, ..., etc...). This new cli call [1] will replace the manual processes started on scheduler nodes (saatchi, scheduler0.staging). - [1] ``` swh scheduler start-orchestrator ``` Related to swh/infra/sysadm-environment#3667
See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/473/ for more details.
Can you describe the purpose, design, etc. in
docs/
?Good point, but i won't do that here. I'd like this to land (and be deployed) soonish so i can stop worrying about the non-puppetized stuff i'm actually maintaining on the side so the ingestion happens.
Let's create a task to actually document the scheduler architecture [1]. The readme is bare bone.
- [1] #3684
By the way, relatedly to doc, I've actually started documenting the modules in the source code:
- listener.py got dropped 'cause it was dead
- orchestrator.py here has a module docstring
- pika_listener.py
- runner.py
The last 2, i just updated yesterday with some module docstring
Build is green
Patch application report for D6520 (id=23782)
Rebasing onto ecc0e280...
Current branch diff-target is up to date.
Changes applied before test
commit 517f420c0a5406b0121bc77efdd682b9f4fd3133 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Oct 20 16:47:05 2021 +0200 Bootstrap orchestrator in charge of scheduling recurring tasks This new module is specifically in charge of scheduling the recurring tasks (loader ones, either dvcs loader like: load-git, load-svn, or package loader: pypi, npm, ..., etc...). This new cli call [1] will replace the manual processes started on scheduler nodes (saatchi, scheduler0.staging). - [1] ``` swh scheduler start-orchestrator ``` Related to swh/infra/sysadm-environment#3667
See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/477/ for more details.
Discussing with @olasd, possible improvments:
- add backoff in between task type iteration
- one thread per task type