archive-staging/swh-cassandra-next-version: Deploy scheduler services independent from staging's
This activates:
- the rabbitmq operator in the archive-staging-rke2 cluster [1]
- adds a new rabbitmq cluster (1 instance) in the archive-staging-rke2 cluster [2]
- the scheduler services [3] which are configured to use the rabbit installed in the same namespace (i.e. scheduler-runner, scheduler-runner-priority, scheduler-listener, scheduler-schedule-recurrent)
[1] helm diff [cluster-configuration] Install rabbitmq operator
[cluster-configuration] Comparing changes between branches production and deploy-scheduler-service-in-next-version...
Your branch is up to date with 'origin/production'.
[cluster-configuration] Generate config in production branch for cluster-configuration/values/admin-rke2.yaml...
[cluster-configuration] Generate config in production branch for cluster-configuration/values/archive-production-rke2.yaml...
[cluster-configuration] Generate config in production branch for cluster-configuration/values/archive-staging-rke2.yaml...
[cluster-configuration] Generate config in production branch for cluster-configuration/values/gitlab-production.yaml...
[cluster-configuration] Generate config in production branch for cluster-configuration/values/gitlab-staging.yaml...
[cluster-configuration] Generate config in production branch for cluster-configuration/values/minikube.yaml...
[cluster-configuration] Generate config in production branch for cluster-configuration/values/rancher.yaml...
[cluster-configuration] Generate config in production branch for cluster-configuration/values/test-staging-rke2.yaml...
[cluster-configuration] Generate config in deploy-scheduler-service-in-next-version branch for cluster-configuration/values/admin-rke2.yaml...
[cluster-configuration] Generate config in deploy-scheduler-service-in-next-version branch for cluster-configuration/values/archive-production-rke2.yaml...
[cluster-configuration] Generate config in deploy-scheduler-service-in-next-version branch for cluster-configuration/values/archive-staging-rke2.yaml...
[cluster-configuration] Generate config in deploy-scheduler-service-in-next-version branch for cluster-configuration/values/gitlab-production.yaml...
[cluster-configuration] Generate config in deploy-scheduler-service-in-next-version branch for cluster-configuration/values/gitlab-staging.yaml...
[cluster-configuration] Generate config in deploy-scheduler-service-in-next-version branch for cluster-configuration/values/minikube.yaml...
[cluster-configuration] Generate config in deploy-scheduler-service-in-next-version branch for cluster-configuration/values/rancher.yaml...
[cluster-configuration] Generate config in deploy-scheduler-service-in-next-version branch for cluster-configuration/values/test-staging-rke2.yaml...
------------- diff for cluster-configuration/values/admin-rke2.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-configuration.SP4YNs81/admin-rke2.yaml.before, six documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-configuration.SP4YNs81/admin-rke2.yaml.after, six documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for cluster-configuration/values/archive-production-rke2.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-configuration.SP4YNs81/archive-production-rke2.yaml.before, seven documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-configuration.SP4YNs81/archive-production-rke2.yaml.after, seven documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for cluster-configuration/values/archive-staging-rke2.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-configuration.SP4YNs81/archive-staging-rke2.yaml.before, eight documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-configuration.SP4YNs81/archive-staging-rke2.yaml.after, nine documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned one difference
|___/
(file level)
---
# Source: Argocd applications commonly used in to configure a SWH cluster/templates/rabbitmq-operator/operator.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: archive-staging-rke2-rabbitmq-operator
namespace: argocd
spec:
source:
chart: rabbitmq-cluster-operator
repoURL: "https://charts.bitnami.com/bitnami"
targetRevision: 4.3.6
helm:
# must match the application name (https://github.com/argoproj/argo-cd/issues/2871)
releaseName: archive-staging-rke2-rabbitmq-operator
revisionHistoryLimit: 2
project: default
destination:
server: "https://rancher.euwest.azure.internal.softwareheritage.org/k8s/clusters/c-m-9n5h9nrf"
namespace: rabbitmq
syncPolicy:
automated:
prune: false
selfHeal: false
------------- diff for cluster-configuration/values/gitlab-production.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-configuration.SP4YNs81/gitlab-production.yaml.before, five documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-configuration.SP4YNs81/gitlab-production.yaml.after, five documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for cluster-configuration/values/gitlab-staging.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-configuration.SP4YNs81/gitlab-staging.yaml.before, five documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-configuration.SP4YNs81/gitlab-staging.yaml.after, five documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for cluster-configuration/values/minikube.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-configuration.SP4YNs81/minikube.yaml.before, two documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-configuration.SP4YNs81/minikube.yaml.after, two documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for cluster-configuration/values/rancher.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-configuration.SP4YNs81/rancher.yaml.before, five documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-configuration.SP4YNs81/rancher.yaml.after, five documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for cluster-configuration/values/test-staging-rke2.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-configuration.SP4YNs81/test-staging-rke2.yaml.before, eight documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-configuration.SP4YNs81/test-staging-rke2.yaml.after, eight documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
[2] helm diff [cluster-components] Install rabbitmq cluster
[cluster-components] Comparing changes between branches production and deploy-scheduler-service-in-next-version...
Your branch is up to date with 'origin/production'.
[cluster-components] Generate config in production branch for cluster-components/values/admin-rke2.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/archive-production-rke2.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/archive-staging-rke2.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/default.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/gitlab-production.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/gitlab-staging.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/minikube.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/rancher.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/test-staging-rke2.yaml...
[cluster-components] Generate config in deploy-scheduler-service-in-next-version branch for cluster-components/values/admin-rke2.yaml...
[cluster-components] Generate config in deploy-scheduler-service-in-next-version branch for cluster-components/values/archive-production-rke2.yaml...
[cluster-components] Generate config in deploy-scheduler-service-in-next-version branch for cluster-components/values/archive-staging-rke2.yaml...
[cluster-components] Generate config in deploy-scheduler-service-in-next-version branch for cluster-components/values/default.yaml...
[cluster-components] Generate config in deploy-scheduler-service-in-next-version branch for cluster-components/values/gitlab-production.yaml...
[cluster-components] Generate config in deploy-scheduler-service-in-next-version branch for cluster-components/values/gitlab-staging.yaml...
[cluster-components] Generate config in deploy-scheduler-service-in-next-version branch for cluster-components/values/minikube.yaml...
[cluster-components] Generate config in deploy-scheduler-service-in-next-version branch for cluster-components/values/rancher.yaml...
[cluster-components] Generate config in deploy-scheduler-service-in-next-version branch for cluster-components/values/test-staging-rke2.yaml...
------------- diff for cluster-components/values/admin-rke2.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-components.S68DlmaR/admin-rke2.yaml.before, 31 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-components.S68DlmaR/admin-rke2.yaml.after, 31 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for cluster-components/values/archive-production-rke2.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-components.S68DlmaR/archive-production-rke2.yaml.before, 17 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-components.S68DlmaR/archive-production-rke2.yaml.after, 17 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for cluster-components/values/archive-staging-rke2.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-components.S68DlmaR/archive-staging-rke2.yaml.before, 21 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-components.S68DlmaR/archive-staging-rke2.yaml.after, 22 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned one difference
|___/
(file level)
---
# Source: cluster-config/templates/rabbitmq/rabbitmq.yaml
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: rabbitmq-scheduler
namespace: swh-cassandra-next-version
spec:
resources:
requests:
replicas: 1
persistence:
storageClassName: local-persistent
storage: 1Gi
rabbitmq:
additionalConfig: |
log.console.level = info
additionalPlugins:
- rabbitmq_management
- rabbitmq_prometheus
secretBackend:
externalSecret:
name: rabbitmq-scheduler-secret
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/backend
operator: In
values:
- "true"
------------- diff for cluster-components/values/default.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-components.S68DlmaR/default.yaml.before
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-components.S68DlmaR/default.yaml.after
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for cluster-components/values/gitlab-production.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-components.S68DlmaR/gitlab-production.yaml.before
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-components.S68DlmaR/gitlab-production.yaml.after
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for cluster-components/values/gitlab-staging.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-components.S68DlmaR/gitlab-staging.yaml.before
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-components.S68DlmaR/gitlab-staging.yaml.after
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for cluster-components/values/minikube.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-components.S68DlmaR/minikube.yaml.before, two documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-components.S68DlmaR/minikube.yaml.after
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned one difference
|___/
(file level)
- one document removed:
---
# Source: cluster-config/templates/rabbitmq/rabbitmq.yaml
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: rabbitmq-scheduler
namespace: cluster-components
spec:
resources:
requests:
replicas: 3
persistence:
storageClassName: standard
storage: 1Gi
rabbitmq:
additionalConfig: |
log.console.level = info
additionalPlugins:
- rabbitmq_management
- rabbitmq_prometheus
secretBackend:
externalSecret:
name: rabbitmq-scheduler-secret
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/backend
operator: In
values:
- "true"
------------- diff for cluster-components/values/rancher.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-components.S68DlmaR/rancher.yaml.before
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-components.S68DlmaR/rancher.yaml.after
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for cluster-components/values/test-staging-rke2.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-components.S68DlmaR/test-staging-rke2.yaml.before, twelve documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-components.S68DlmaR/test-staging-rke2.yaml.after, twelve documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
[3] helm diff [swh-charts] Deploy scheduler services with newly internal backends
[swh] Comparing changes between branches production and deploy-scheduler-service-in-next-version (per environment)...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment staging, namespace swh...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra-next-version...
[swh] Generate config in deploy-scheduler-service-in-next-version branch for environment staging...
[swh] Generate config in deploy-scheduler-service-in-next-version branch for environment staging...
[swh] Generate config in deploy-scheduler-service-in-next-version branch for environment staging...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment production, namespace swh...
[swh] Generate config in production branch for environment production, namespace swh-cassandra...
[swh] Generate config in production branch for environment production, namespace swh-cassandra-next-version...
[swh] Generate config in deploy-scheduler-service-in-next-version branch for environment production...
[swh] Generate config in deploy-scheduler-service-in-next-version branch for environment production...
[swh] Generate config in deploy-scheduler-service-in-next-version branch for environment production...
------------- diff for environment staging namespace swh -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.zDzr5fQS/staging-swh.before, 134 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.zDzr5fQS/staging-swh.after, 134 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for environment staging namespace swh-cassandra -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.zDzr5fQS/staging-swh-cassandra.before, 417 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.zDzr5fQS/staging-swh-cassandra.after, 417 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for environment staging namespace swh-cassandra-next-version -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.zDzr5fQS/staging-swh-cassandra-next-version.before, 182 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.zDzr5fQS/staging-swh-cassandra-next-version.after, 191 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned twelve differences
|___/
(file level)
---
# Source: swh/templates/scheduler/extra-services-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: extra-services-configuration-template
namespace: swh-cassandra-next-version
data:
config.yml.template: |
scheduler:
cls: remote
url: http://scheduler-rpc
celery:
task_broker: amqp://${AMQP_USERNAME}:${AMQP_PASSWORD}@rabbitmq-scheduler:5672/%2f
# Source: swh/templates/scheduler/recurrent-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra-next-version
name: scheduler-schedule-recurrent-configuration-template
data:
config.yml.template: |
scheduler:
cls: remote
url: http://scheduler-rpc
celery:
task_broker: amqp://${AMQP_USERNAME}:${AMQP_PASSWORD}@rabbitmq-scheduler:5672/%2f
scheduling_policy:
content:
- policy: origins_without_last_update
weight: 100
default:
- policy: already_visited_order_by_lag
weight: 40
- policy: never_visited_oldest_update_first
weight: 40
- policy: origins_without_last_update
weight: 20
git:
- policy: origins_without_last_update
tablesample: 0.1
weight: 85
- policy: never_visited_oldest_update_first
tablesample: 0.1
weight: 10
- policy: already_visited_order_by_lag
tablesample: 0.1
weight: 5
git-checkout:
- policy: origins_without_last_update
weight: 100
hg-checkout:
- policy: origins_without_last_update
weight: 100
opam:
- policy: origins_without_last_update
weight: 100
svn-export:
- policy: origins_without_last_update
weight: 100
tarball-directory:
- policy: origins_without_last_update
weight: 100
# Source: swh/templates/scheduler/rpc-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra-next-version
name: scheduler-rpc-configuration-template
data:
config.yml.template: |
scheduler:
cls: postgresql
db: host=next-version-dbs-rw port=5432 user=swh-scheduler dbname=swh-scheduler password=${POSTGRESQL_PASSWORD}
# Source: swh/templates/scheduler/rpc-service.yaml
apiVersion: v1
kind: Service
metadata:
name: scheduler-rpc
namespace: swh-cassandra-next-version
labels:
app: scheduler-rpc
spec:
type: ClusterIP
selector:
app: scheduler-rpc
ports:
- port: 5008
targetPort: 5008
name: rpc
# Source: swh/templates/scheduler/extra-services-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: swh-cassandra-next-version
name: scheduler-listener
labels:
app: scheduler-listener
spec:
revisionHistoryLimit: 2
replicas: 1
selector:
matchLabels:
app: scheduler-listener
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scheduler-listener
annotations:
checksum/config: deb261d7fdb159aea6900b94a25dc22b19872d77d0129aa8c3e33be92974fe99
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scheduler
operator: In
values:
- "true"
priorityClassName: swh-cassandra-next-version-frontend-rpc-workload
initContainers:
- name: prepare-configuration
image: "debian:bullseye"
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- "-c"
- "eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config.yml"
env:
- name: AMQP_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: rabbitmq-scheduler-secret
optional: false
- name: AMQP_USERNAME
valueFrom:
secretKeyRef:
key: username
name: rabbitmq-scheduler-secret
optional: false
volumeMounts:
- name: configuration
mountPath: /etc/swh
- name: configuration-template
mountPath: /etc/swh/configuration-template
containers:
- name: scheduler-listener
resources:
requests:
memory: 100Mi
cpu: 10m
image: "container-registry.softwareheritage.org/swh/infra/swh-apps/scheduler:20240613.1"
command:
- /opt/swh/entrypoint.sh
args:
- swh
- "--log-level"
- INFO
- scheduler
- "--config-file"
- /etc/swh/config.yml
- start-listener
env:
- name: STATSD_HOST
value: prometheus-statsd-exporter
- name: STATSD_PORT
value: 9125
- name: STATSD_TAGS
value: "deployment:scheduler-listener"
- name: SWH_CONFIG_FILENAME
value: /etc/swh/config.yml
- name: SWH_LOG_LEVEL
value: INFO
imagePullPolicy: IfNotPresent
volumeMounts:
- name: configuration
mountPath: /etc/swh
volumes:
- name: configuration
emptyDir: {}
- name: configuration-template
configMap:
name: extra-services-configuration-template
items:
- key: config.yml.template
path: config.yml.template
# Source: swh/templates/scheduler/extra-services-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: swh-cassandra-next-version
name: scheduler-runner
labels:
app: scheduler-runner
spec:
revisionHistoryLimit: 2
selector:
matchLabels:
app: scheduler-runner
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scheduler-runner
annotations:
checksum/config: deb261d7fdb159aea6900b94a25dc22b19872d77d0129aa8c3e33be92974fe99
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scheduler
operator: In
values:
- "true"
priorityClassName: swh-cassandra-next-version-frontend-rpc-workload
initContainers:
- name: prepare-configuration
image: "debian:bullseye"
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- "-c"
- "eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config.yml"
env:
- name: AMQP_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: rabbitmq-scheduler-secret
optional: false
- name: AMQP_USERNAME
valueFrom:
secretKeyRef:
key: username
name: rabbitmq-scheduler-secret
optional: false
volumeMounts:
- name: configuration
mountPath: /etc/swh
- name: configuration-template
mountPath: /etc/swh/configuration-template
containers:
- name: scheduler-runner
resources:
requests:
memory: 100Mi
cpu: 10m
image: "container-registry.softwareheritage.org/swh/infra/swh-apps/scheduler:20240613.1"
command:
- /opt/swh/entrypoint.sh
args:
- swh
- "--log-level"
- INFO
- scheduler
- "--config-file"
- /etc/swh/config.yml
- start-runner
- "--period"
- 10
env:
- name: STATSD_HOST
value: prometheus-statsd-exporter
- name: STATSD_PORT
value: 9125
- name: STATSD_TAGS
value: "deployment:scheduler-runner"
- name: SWH_CONFIG_FILENAME
value: /etc/swh/config.yml
- name: SWH_LOG_LEVEL
value: INFO
imagePullPolicy: IfNotPresent
volumeMounts:
- name: configuration
mountPath: /etc/swh
volumes:
- name: configuration
emptyDir: {}
- name: configuration-template
configMap:
name: extra-services-configuration-template
items:
- key: config.yml.template
path: config.yml.template
# Source: swh/templates/scheduler/extra-services-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: swh-cassandra-next-version
name: scheduler-runner-priority
labels:
app: scheduler-runner-priority
spec:
revisionHistoryLimit: 2
selector:
matchLabels:
app: scheduler-runner-priority
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scheduler-runner-priority
annotations:
checksum/config: deb261d7fdb159aea6900b94a25dc22b19872d77d0129aa8c3e33be92974fe99
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scheduler
operator: In
values:
- "true"
priorityClassName: swh-cassandra-next-version-frontend-rpc-workload
initContainers:
- name: prepare-configuration
image: "debian:bullseye"
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- "-c"
- "eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config.yml"
env:
- name: AMQP_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: rabbitmq-scheduler-secret
optional: false
- name: AMQP_USERNAME
valueFrom:
secretKeyRef:
key: username
name: rabbitmq-scheduler-secret
optional: false
volumeMounts:
- name: configuration
mountPath: /etc/swh
- name: configuration-template
mountPath: /etc/swh/configuration-template
containers:
- name: scheduler-runner-priority
resources:
requests:
memory: 50Mi
cpu: 5m
image: "container-registry.softwareheritage.org/swh/infra/swh-apps/scheduler:20240613.1"
command:
- /opt/swh/entrypoint.sh
args:
- swh
- "--log-level"
- INFO
- scheduler
- "--config-file"
- /etc/swh/config.yml
- start-runner
- "--period"
- 10
- "--with-priority"
- "--task-type"
- load-bzr
- "--task-type"
- load-cvs
- "--task-type"
- load-git
- "--task-type"
- load-svn
- "--task-type"
- load-archive-files
- "--task-type"
- load-hg
env:
- name: STATSD_HOST
value: prometheus-statsd-exporter
- name: STATSD_PORT
value: 9125
- name: STATSD_TAGS
value: "deployment:scheduler-runner-priority"
- name: SWH_CONFIG_FILENAME
value: /etc/swh/config.yml
- name: SWH_LOG_LEVEL
value: INFO
imagePullPolicy: IfNotPresent
volumeMounts:
- name: configuration
mountPath: /etc/swh
volumes:
- name: configuration
emptyDir: {}
- name: configuration-template
configMap:
name: extra-services-configuration-template
items:
- key: config.yml.template
path: config.yml.template
# Source: swh/templates/scheduler/recurrent-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: swh-cassandra-next-version
name: scheduler-schedule-recurrent
labels:
app: scheduler-schedule-recurrent
spec:
revisionHistoryLimit: 2
selector:
matchLabels:
app: scheduler-schedule-recurrent
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scheduler-schedule-recurrent
annotations:
checksum/config: 44b2af07bf6140bf7d2b2b9a72b7850fbb72fc2806469ac8e47909a1f1d563e7
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scheduler
operator: In
values:
- "true"
priorityClassName: swh-cassandra-next-version-frontend-rpc-workload
initContainers:
- name: prepare-configuration
image: "debian:bullseye"
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- "-c"
- "eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config.yml"
env:
- name: AMQP_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: rabbitmq-scheduler-secret
optional: false
- name: AMQP_USERNAME
valueFrom:
secretKeyRef:
key: username
name: rabbitmq-scheduler-secret
optional: false
volumeMounts:
- name: configuration
mountPath: /etc/swh
- name: configuration-template
mountPath: /etc/swh/configuration-template
containers:
- name: scheduler-schedule-recurrent
resources:
requests:
memory: 400Mi
cpu: 10m
image: "container-registry.softwareheritage.org/swh/infra/swh-apps/scheduler:20240613.1"
command:
- /opt/swh/entrypoint.sh
args:
- swh
- scheduler
- "--config-file"
- /etc/swh/config.yml
- schedule-recurrent
env:
- name: STATSD_HOST
value: prometheus-statsd-exporter
- name: STATSD_PORT
value: 9125
- name: STATSD_TAGS
value: "deployment:scheduler-schedule-recurrent"
- name: SWH_CONFIG_FILENAME
value: /etc/swh/config.yml
- name: SWH_LOG_LEVEL
value: INFO
imagePullPolicy: IfNotPresent
volumeMounts:
- name: configuration
mountPath: /etc/swh
volumes:
- name: configuration
emptyDir: {}
- name: configuration-template
configMap:
name: scheduler-schedule-recurrent-configuration-template
items:
- key: config.yml.template
path: config.yml.template
# Source: swh/templates/scheduler/rpc-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: swh-cassandra-next-version
name: scheduler-rpc
labels:
app: scheduler-rpc
spec:
revisionHistoryLimit: 2
replicas: 2
selector:
matchLabels:
app: scheduler-rpc
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scheduler-rpc
annotations:
checksum/config: 2db8e1801391f20004d468848e6a4638656ea80d0dec6eead3a28eaed3c544dd
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scheduler
operator: In
values:
- "true"
priorityClassName: swh-cassandra-next-version-frontend-rpc
initContainers:
- name: prepare-configuration
image: "debian:bullseye"
imagePullPolicy: IfNotPresent
command:
- /bin/bash
args:
- "-c"
- "eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config.yml"
env:
- name: POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
key: postgres-swh-scheduler-password
name: swh-scheduler-postgresql-common-secret
optional: false
volumeMounts:
- name: configuration
mountPath: /etc/swh
- name: configuration-template
mountPath: /etc/swh/configuration-template
containers:
- name: scheduler-rpc
resources:
requests:
memory: 700Mi
cpu: 50m
image: "container-registry.softwareheritage.org/swh/infra/swh-apps/scheduler:20240613.1"
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5008
name: rpc
readinessProbe:
httpGet:
path: /
port: rpc
initialDelaySeconds: 15
failureThreshold: 30
periodSeconds: 5
livenessProbe:
tcpSocket:
port: rpc
initialDelaySeconds: 10
periodSeconds: 5
command:
- /bin/bash
args:
- "-c"
- /opt/swh/entrypoint.sh
env:
- name: THREADS
value: 5
- name: WORKERS
value: 4
- name: TIMEOUT
value: 60
- name: STATSD_HOST
value: prometheus-statsd-exporter
- name: STATSD_PORT
value: 9125
- name: STATSD_TAGS
value: "deployment:scheduler-rpc"
- name: SWH_LOG_LEVEL
value: INFO
volumeMounts:
- name: configuration
mountPath: /etc/swh
volumes:
- name: configuration
emptyDir: {}
- name: configuration-template
configMap:
name: scheduler-rpc-configuration-template
items:
- key: config.yml.template
path: config.yml.template
data.config.yml.template (v1/ConfigMap/swh-cassandra-next-version/loader-metadata-template)
± value change in multiline text (one insert, one deletion)
- url: http://scheduler.internal.staging.swh.network
+ url: http://scheduler-rpc
data.config.yml.template (v1/ConfigMap/swh-cassandra-next-version/toolbox-scheduler-template)
± value change in multiline text (two inserts, two deletions)
- db: host=db1.internal.staging.swh.network port=5432 user=swh-scheduler dbname=swh-scheduler
- password=${POSTGRESQL_PASSWORD}
+ db: host=next-version-dbs-rw port=5432 user=swh-scheduler dbname=swh-scheduler password=${POSTGRESQL_PASSWORD}
- task_broker: amqp://swhproducer:${AMQP_PASSWORD}@scheduler0.internal.staging.swh.network:5672/%2f
+ task_broker: amqp://${AMQP_USERNAME}:${AMQP_PASSWORD}@rabbitmq-scheduler:5672/%2f
data.config.yml.template (v1/ConfigMap/swh-cassandra-next-version/vault-rpc-configuration-template)
± value change in multiline text (one insert, one deletion)
- url: http://scheduler.internal.staging.swh.network
+ url: http://scheduler-rpc
data.config.yml.template (v1/ConfigMap/swh-cassandra-next-version/web-cassandra-configuration-template)
± value change in multiline text (one insert, one deletion)
- url: http://scheduler.internal.staging.swh.network
+ url: http://scheduler-rpc
spec.template.metadata.annotations.checksum/config (apps/v1/Deployment/swh-cassandra-next-version/loader-metadata)
± value change
- dd44c12116da9d32fb9b3896c5550afb4542f04176acb826173c1d61342163db
+ b16f9065d32c7b045797f6c93e81990aae790fbfee5f49e1ddbf86851bb585c4
spec.template.metadata.annotations.checksum/config (apps/v1/Deployment/swh-cassandra-next-version/swh-toolbox)
± value change
- bd3c06ad2a4906a589c2b46e0128b44dd7aa57eee9218b66e5552a31dfaa3f9d
+ 75635bd12068a9ed3031b51be365490814a184e4391ee4ac444e0d0f29c02abb
spec.template.spec.initContainers.prepare-configuration-scheduler.env (apps/v1/Deployment/swh-cassandra-next-version/swh-toolbox)
+ one list entry added:
- name: AMQP_USERNAME
valueFrom:
secretKeyRef:
name: rabbitmq-scheduler-secret
key: username
optional: false
spec.template.spec.initContainers.prepare-configuration-scheduler.env.AMQP_PASSWORD.valueFrom.secretKeyRef.key (apps/v1/Deployment/swh-cassandra-next-version/swh-toolbox)
± value change
- swhproducer-password
+ password
spec.template.spec.initContainers.prepare-configuration-scheduler.env.AMQP_PASSWORD.valueFrom.secretKeyRef.name (apps/v1/Deployment/swh-cassandra-next-version/swh-toolbox)
± value change
- amqp-secrets
+ rabbitmq-scheduler-secret
spec.template.metadata.annotations.checksum/config (apps/v1/Deployment/swh-cassandra-next-version/vault-rpc)
± value change
- 9ef8715deaaa7835d75f5dbb55300b55deb4db043101b22a2c90573ed39a67d5
+ cc0eb40218ffe2bb4cbed52bb42b92e6e30cad87e64297cabbdaa8e8b2be01b6
spec.template.metadata.annotations.checksum/config (apps/v1/Deployment/swh-cassandra-next-version/web-cassandra)
± value change
- 8d66a4de829397d086b9ae6e50ca5c1776ec2894b940554382e38d9436082557
+ 547fb610c38ca77c899a94f58740a4eeb0ae305f9260764fd7aa3ca63229e924
------------- diff for environment production namespace swh -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.zDzr5fQS/production-swh.before, 447 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.zDzr5fQS/production-swh.after, 447 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for environment production namespace swh-cassandra -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.zDzr5fQS/production-swh-cassandra.before, 117 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.zDzr5fQS/production-swh-cassandra.after, 117 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
Edited by Antoine R. Dumont