Skip to content

scheduler: Add scheduler-schedule-recurrent service to schedule origins regularly

Antoine R. Dumont requested to merge staging-schedule-recurrent-in-elastic into staging

Contains chart services:

helm template and minikube are happy about it:

$ chart=swh; helm template $chart $chart/ --values values-swh-application-versions.yaml --values $chart/values.yaml --values $chart/values/default.yaml --values $chart/values/minikube.yaml --debug
...
# Source: swh/templates/scheduler/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: swh
  name: scheduler-configuration-template
data:
  config.yml.template: |
    scheduler:
      cls: remote
      url: http://fake-scheduler.i.s.s.n:5008

    celery:
      task_broker: amqp://guest:guest@fake-scheduler.i.s.s.n:5672/%2f

    scheduling_policy:
      default:
      - policy: never_visited_oldest_update_first
        weight: 100
      opam:
      - policy: origins_without_last_update
        weight: 100
...
...

[1] Starting (and crashing as expected as there ain't any scheduler db behind).

Running swh command scheduler schedule-recurrent
Traceback (most recent call last):
  File "/opt/swh/.local/bin/swh", line 8, in <module>
    sys.exit(main())
  File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/__init__.py", line 174, in main
    return swh(auto_envvar_prefix="SWH")
  File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/swh/.local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/swh/.local/lib/python3.10/site-packages/swh/scheduler/cli/admin.py", line 168, in schedule_recurrent
    all_task_types = scheduler.get_task_types()
AttributeError: 'NoneType' object has no attribute 'get_task_types'
Stream closed EOF for swh/scheduler-schedule-recurrent-b95d646cd-2cqj4 (scheduler-schedule-recurrent)

[2] rpc happy (not doing anything in minikube though as i don't have the rest configured):

│ Starting the swh-scheduler API server                                                                                                                                                                                                       │
│ [2023-09-06 13:46:41 +0000] [1] [INFO] Starting gunicorn 21.2.0                                                                                                                                                                             │
│ [2023-09-06 13:46:41 +0000] [1] [INFO] Listening at: http://0.0.0.0:5008 (1)                                                                                                                                                                │
│ [2023-09-06 13:46:41 +0000] [1] [INFO] Using worker: sync                                                                                                                                                                                   │
│ [2023-09-06 13:46:41 +0000] [7] [INFO] Booting worker with pid: 7

Refs. swh/infra/sysadm-environment#5035 (closed)

Edited by Antoine R. Dumont

Merge request reports