swh: Allow centralized pv/pvcs declaration
-
Review changes -
-
Download -
Patches
-
Plain diff
This allows to share the pv/pvcs declaration instead of duplicating it inside the various templates using a duplicated declaration (e.g. graph, alter, cassandra-checks, ...).
It's best to review the mr commit per commit:
- 1 commit allows pvs and pvcs declaration
- 1 commit reworks the graph deployment to use pvs/pvcs declaration [1]
- the alter commit aligns the alter deployment to pvs/pvcs declaration [2]
- 1 commit drops the unused cassandra pvc declaration check which was not used.
- 1 commit adapts the objstorage deployment to add a new objstorage-ro rpc instance which now shares the newly declared volume between the objstorage rpc rw and the rpc ro instances. [3]
- 1 commit does does the same for the local cluster. This got tested to work in the local-cluster with a similar setup (commit included) [4]
- At last, 1 commit in the cluster-components which allows to declare volumes as well. It's then moving the objstorage-pathslicing-pvc declared in the next-version namespace in the cluster-components. That way, the pvc won't get decommissionned when the argocd next-version app is closed (after the chart mr is merged). The pv associated is thus no longer resetted (which is the case currently) [5]
Note: The diff should be null for the pvs/pvcs changes but as the toYaml somehow generates different order the pv/pvcs, the diffs are minimalistics (some entries are shuffled around but the values remains the same).
[1] helm diff on graph deployment
[swh] Comparing changes between branches production and mr/centralize-pv-pvcs (per environment)...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment staging, namespace swh...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra...
[swh] Generate config in production branch for environment staging, namespace next-version...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment staging...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment staging...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment staging...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment production, namespace swh...
[swh] Generate config in production branch for environment production, namespace swh-cassandra...
[swh] Generate config in production branch for environment production, namespace next-version...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment production...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment production...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment production...
------------- diff for environment staging namespace swh -------------
No differences
------------- diff for environment staging namespace swh-cassandra -------------
--- /tmp/swh-chart.swh.urjwplFS/staging-swh-cassandra.before 2025-03-07 12:01:59.920857760 +0100
+++ /tmp/swh-chart.swh.urjwplFS/staging-swh-cassandra.after 2025-03-07 12:02:00.464840181 +0100
@@ -9859,38 +9859,38 @@
namespace: swh-cassandra
spec:
storageClassName: cephfs
accessModes:
- ReadWriteMany
volumeMode: Filesystem
resources:
requests:
storage: 30Gi
---
-# Source: swh/templates/graph/persistent-volume-claims.yaml
+# Source: swh/templates/volumes/persistent-volume-claims.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: graph-python3k-inmemory-pvc
namespace: swh-cassandra
labels:
app: graph-grpc-python3k
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: local-path
volumeMode: Filesystem
---
-# Source: swh/templates/graph/persistent-volume-claims.yaml
+# Source: swh/templates/volumes/persistent-volume-claims.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: graph-python3k-persistent-pvc
namespace: swh-cassandra
labels:
app: graph-grpc-python3k
spec:
accessModes:
- ReadWriteOnce
------------- diff for environment staging namespace next-version -------------
--- /tmp/swh-chart.swh.urjwplFS/staging-next-version.before 2025-03-07 12:02:00.048853625 +0100
+++ /tmp/swh-chart.swh.urjwplFS/staging-next-version.after 2025-03-07 12:02:00.592836045 +0100
@@ -4600,38 +4600,38 @@
auth_token: ${SVIX_AUTH_TOKEN}
server_url: https://svix-next-version.internal.staging.swh.network
journal:
brokers:
- kafka-cluster-kafka-brokers:9092
auto_offset_reset: latest
group_id: staging-next-version-archive-webhooks
object_types:
- origin_visit_status
---
-# Source: swh/templates/graph/persistent-volume-claims.yaml
+# Source: swh/templates/volumes/persistent-volume-claims.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: swh-graph-grpc-dataset-example-pvc
namespace: swh-cassandra-next-version
labels:
app: graph-grpc-example
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: local-persistent
volumeMode: Filesystem
---
-# Source: swh/templates/graph/persistent-volume-claims.yaml
+# Source: swh/templates/volumes/persistent-volume-claims.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: swh-graph-grpc-inmemory-pvc
namespace: swh-cassandra-next-version
labels:
app: graph-grpc-example
spec:
accessModes:
- ReadWriteOnce
------------- diff for environment production namespace swh -------------
No differences
------------- diff for environment production namespace swh-cassandra -------------
--- /tmp/swh-chart.swh.urjwplFS/production-swh-cassandra.before 2025-03-07 12:02:01.068820664 +0100
+++ /tmp/swh-chart.swh.urjwplFS/production-swh-cassandra.after 2025-03-07 12:02:01.540805413 +0100
@@ -11595,92 +11595,94 @@
brokers:
- kafka1.internal.softwareheritage.org:9092
- kafka2.internal.softwareheritage.org:9092
- kafka3.internal.softwareheritage.org:9092
- kafka4.internal.softwareheritage.org:9092
auto_offset_reset: latest
group_id: swh-archive-prod-webhooks
object_types:
- origin_visit_status
---
-# Source: swh/templates/graph/persistent-volume-claims.yaml
+# Source: swh/templates/volumes/persistent-volume.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: graph-20241206-persistent-local-pv
namespace: swh-cassandra
spec:
- capacity:
- storage: 1Gi
- volumeMode: Filesystem
accessModes:
- ReadWriteOnce
- persistentVolumeReclaimPolicy: Retain
- storageClassName: local-storage
+ capacity:
+ storage: 1Gi
local:
path: /srv/softwareheritage/ssd/graph/2024-12-06/compressed
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
- values:
- - rancher-node-highmem01
+ values:
+ - rancher-node-highmem01
+ persistentVolumeReclaimPolicy: Retain
+ storageClassName: local-storage
+ volumeMode: Filesystem
---
# Source: swh/templates/alter/deployment.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: alter-recovery-bundles-pvc
namespace: swh-cassandra
spec:
storageClassName: cephfs
accessModes:
- ReadWriteMany
volumeMode: Filesystem
resources:
requests:
storage: 30Gi
---
-# Source: swh/templates/graph/persistent-volume-claims.yaml
+# Source: swh/templates/volumes/persistent-volume-claims.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: graph-20241206-inmemory-pvc
namespace: swh-cassandra
labels:
app: graph-grpc-20241206
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: local-path
volumeMode: Filesystem
---
-# Source: swh/templates/graph/persistent-volume-claims.yaml
+# Source: swh/templates/volumes/persistent-volume-claims.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: graph-20241206-persistent-pvc
namespace: swh-cassandra
+ labels:
+ app: graph-grpc-20241206
spec:
+ accessModes:
+ - ReadWriteOnce
resources:
requests:
storage: 1Gi
+ storageClassName: local-storage
volumeMode: Filesystem
volumeName: graph-20241206-persistent-local-pv
- storageClassName: local-storage
- accessModes:
- - ReadWriteOnce
---
# Source: swh/templates/counters/rpc-service.yaml
apiVersion: v1
kind: Service
metadata:
name: counters-rpc
namespace: swh-cassandra
labels:
app: counters-rpc
spec:
[2] helm diff on alter deployment
[swh] Comparing changes between branches production and mr/centralize-pv-pvcs (per environment)...
Your branch is ahead of 'origin/production' by 5 commits.
(use "git push" to publish your local commits)
[swh] Generate config in production branch for environment staging, namespace swh...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra...
[swh] Generate config in production branch for environment staging, namespace next-version...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment staging...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment staging...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment staging...
Your branch is ahead of 'origin/production' by 5 commits.
(use "git push" to publish your local commits)
[swh] Generate config in production branch for environment production, namespace swh...
[swh] Generate config in production branch for environment production, namespace swh-cassandra...
[swh] Generate config in production branch for environment production, namespace next-version...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment production...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment production...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment production...
------------- diff for environment staging namespace swh -------------
No differences
------------- diff for environment staging namespace swh-cassandra -------------
--- /tmp/swh-chart.swh.WMKOZRCT/staging-swh-cassandra.before 2025-03-07 12:28:01.318374812 +0100
+++ /tmp/swh-chart.swh.WMKOZRCT/staging-swh-cassandra.after 2025-03-07 12:28:01.878356693 +0100
@@ -9844,34 +9844,36 @@
auth_token: ${SVIX_AUTH_TOKEN}
server_url: https://svix.internal.staging.swh.network
journal:
brokers:
- journal2.internal.staging.swh.network:9092
auto_offset_reset: latest
group_id: swh-archive-stg-webhooks
object_types:
- origin_visit_status
---
-# Source: swh/templates/alter/deployment.yaml
+# Source: swh/templates/volumes/persistent-volume-claims.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: alter-recovery-bundles-pvc
namespace: swh-cassandra
+ labels:
+ app: alter
spec:
- storageClassName: cephfs
accessModes:
- - ReadWriteMany
- volumeMode: Filesystem
+ - ReadWriteMany
resources:
requests:
storage: 30Gi
+ storageClassName: cephfs
+ volumeMode: Filesystem
---
# Source: swh/templates/volumes/persistent-volume-claims.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: graph-python3k-inmemory-pvc
namespace: swh-cassandra
labels:
app: graph-grpc-python3k
spec:
------------- diff for environment staging namespace next-version -------------
No differences
------------- diff for environment production namespace swh -------------
No differences
------------- diff for environment production namespace swh-cassandra -------------
--- /tmp/swh-chart.swh.WMKOZRCT/production-swh-cassandra.before 2025-03-07 12:28:02.482337151 +0100
+++ /tmp/swh-chart.swh.WMKOZRCT/production-swh-cassandra.after 2025-03-07 12:28:02.954321880 +0100
@@ -11622,34 +11622,36 @@
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- rancher-node-highmem01
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
volumeMode: Filesystem
---
-# Source: swh/templates/alter/deployment.yaml
+# Source: swh/templates/volumes/persistent-volume-claims.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: alter-recovery-bundles-pvc
namespace: swh-cassandra
+ labels:
+ app: alter
spec:
- storageClassName: cephfs
accessModes:
- - ReadWriteMany
- volumeMode: Filesystem
+ - ReadWriteMany
resources:
requests:
storage: 30Gi
+ storageClassName: cephfs
+ volumeMode: Filesystem
---
# Source: swh/templates/volumes/persistent-volume-claims.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: graph-20241206-inmemory-pvc
namespace: swh-cassandra
labels:
app: graph-grpc-20241206
spec:
[3] helm diff to evolve the next-version/local-cluster objstorage
[swh] Comparing changes between branches production and mr/centralize-pv-pvcs (per environment)...
Your branch is ahead of 'origin/production' by 6 commits.
(use "git push" to publish your local commits)
[swh] Generate config in production branch for environment staging, namespace swh...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra...
[swh] Generate config in production branch for environment staging, namespace next-version...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment staging...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment staging...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment staging...
Your branch is ahead of 'origin/production' by 6 commits.
(use "git push" to publish your local commits)
[swh] Generate config in production branch for environment production, namespace swh...
[swh] Generate config in production branch for environment production, namespace swh-cassandra...
[swh] Generate config in production branch for environment production, namespace next-version...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment production...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment production...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment production...
------------- diff for environment staging namespace swh -------------
No differences
------------- diff for environment staging namespace swh-cassandra -------------
No differences
------------- diff for environment staging namespace next-version -------------
--- /tmp/swh-chart.swh.Q0zchbWG/staging-next-version.before 2025-03-07 13:56:16.446480298 +0100
+++ /tmp/swh-chart.swh.Q0zchbWG/staging-next-version.after 2025-03-07 13:56:17.098459168 +0100
@@ -1877,24 +1877,107 @@
root:
level: "INFO"
handlers:
- console
---
# Source: swh/templates/objstorage/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra-next-version
+ name: objstorage-ro-pathslicing-configuration-template
+data:
+ config.yml.template: |
+ objstorage:
+ client_max_size: 1073741824
+ cls: pathslicing
+ root: /srv/swh/objects
+ slicing: 0:1/1:5
+---
+# Source: swh/templates/objstorage/configmap.yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ namespace: swh-cassandra-next-version
+ name: objstorage-ro-pathslicing-configuration-logging
+data:
+ logging-gunicorn.json: |
+ {
+ "disable_existing_loggers": true,
+ "filters": {
+ "throttle_accesslog": {
+ "interval": 60,
+ "status_codes": [
+ 200,
+ 400
+ ]
+ }
+ },
+ "formatters": {
+ "default": {
+ "class": "pythonjsonlogger.jsonlogger.JsonFormatter",
+ "format": "[%(asctime)s: %(levelname)s/%(processName)s-%(process)d %(threadName)s(%(pathname)s:%(lineno)s:%(funcName)s)]%(name)s %(message)s"
+ }
+ },
+ "handlers": {
+ "console": {
+ "class": "logging.StreamHandler",
+ "formatter": "default",
+ "level": "INFO",
+ "stream": "ext://sys.stdout"
+ }
+ },
+ "loggers": {
+ "azure": {
+ "level": "WARN"
+ },
+ "gunicorn": {
+ "level": "INFO"
+ },
+ "gunicorn.access": {
+ "filters": [
+ "throttle_accesslog"
+ ],
+ "handlers": [
+ "console"
+ ],
+ "level": "WARN"
+ },
+ "gunicorn.error": {
+ "level": "INFO"
+ },
+ "swh": {
+ "level": "INFO"
+ },
+ "swh.core.statsd": {
+ "level": "INFO"
+ }
+ },
+ "root": {
+ "handlers": [
+ "console"
+ ],
+ "level": "INFO"
+ },
+ "version": 1
+ }
+---
+# Source: swh/templates/objstorage/configmap.yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ namespace: swh-cassandra-next-version
name: objstorage-rw-pathslicing-configuration-template
data:
config.yml.template: |
objstorage:
+ allow_delete: true
client_max_size: 1073741824
cls: pathslicing
root: /srv/swh/objects
slicing: 0:1/1:5
---
# Source: swh/templates/objstorage/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra-next-version
@@ -2440,21 +2523,21 @@
password: ${CASSANDRA_RW_PASSWORD}
username: ${CASSANDRA_RW_USERNAME}
cls: cassandra
consistency_level: LOCAL_QUORUM
hosts:
- cassandra-cluster-next-version-service
keyspace: swh
objstorage:
cls: remote
enable_requests_retry: true
- url: http://objstorage-rw-pathslicing:5003
+ url: http://objstorage-ro-pathslicing:5003
---
# Source: swh/templates/storage/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra-next-version
name: storage-ro-cassandra-configuration-logging
data:
logging-gunicorn.json: |
{
@@ -4604,20 +4687,35 @@
- kafka-cluster-kafka-brokers:9092
auto_offset_reset: latest
group_id: staging-next-version-archive-webhooks
object_types:
- origin_visit_status
---
# Source: swh/templates/volumes/persistent-volume-claims.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
+ name: objstorage-pathslicing-pvc
+ namespace: swh-cassandra-next-version
+spec:
+ accessModes:
+ - ReadWriteMany
+ resources:
+ requests:
+ storage: 30Gi
+ storageClassName: cephfs
+ volumeMode: Filesystem
+---
+# Source: swh/templates/volumes/persistent-volume-claims.yaml
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
name: swh-graph-grpc-dataset-example-pvc
namespace: swh-cassandra-next-version
labels:
app: graph-grpc-example
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
@@ -4985,20 +5083,37 @@
port: 11211
targetPort: 11211
- name: metrics
port: 9150
targetPort: 9150
---
# Source: swh/templates/objstorage/service.yaml
apiVersion: v1
kind: Service
metadata:
+ name: objstorage-ro-pathslicing
+ namespace: swh-cassandra-next-version
+ labels:
+ app: objstorage-ro-pathslicing
+spec:
+ type: ClusterIP
+ selector:
+ app: objstorage-ro-pathslicing
+ ports:
+ - port: 5003
+ targetPort: 5003
+ name: rpc
+---
+# Source: swh/templates/objstorage/service.yaml
+apiVersion: v1
+kind: Service
+metadata:
name: objstorage-rw-pathslicing
namespace: swh-cassandra-next-version
labels:
app: objstorage-rw-pathslicing
spec:
type: ClusterIP
selector:
app: objstorage-rw-pathslicing
ports:
- port: 5003
@@ -8480,38 +8595,189 @@
path: /metrics
port: 9150
initialDelaySeconds: 5
periodSeconds: 10
---
# Source: swh/templates/objstorage/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: swh-cassandra-next-version
+ name: objstorage-ro-pathslicing
+ labels:
+ app: objstorage-ro-pathslicing
+spec:
+ revisionHistoryLimit: 2
+ selector:
+ matchLabels:
+ app: objstorage-ro-pathslicing
+ strategy:
+ type: RollingUpdate
+ rollingUpdate:
+ maxSurge: 1
+ template:
+ metadata:
+ labels:
+ app: objstorage-ro-pathslicing
+ annotations:
+ checksum/config: 71f02bfaba97500abb88050f40502c4ad09338eae7585ab66c7202fa2838bbe5
+ checksum/config-logging: cd8bff14f691fefde95391e737e2658637337c3b0033bd479f315a822701b316
+ checksum/config-utils: 94d255131467f84bef964a4c72b2b792c5ebaf711bb1c77829d7cd1007a8ac22
+ spec:
+ securityContext:
+ fsGroup: 1005
+ nodeSelector:
+ kubernetes.io/hostname: rancher-node-staging-rke2-metal01
+ affinity:
+ nodeAffinity:
+ requiredDuringSchedulingIgnoredDuringExecution:
+ nodeSelectorTerms:
+ - matchExpressions:
+ - key: swh/objstorage
+ operator: In
+ values:
+ - "true"
+ priorityClassName: swh-cassandra-next-version-frontend-rpc
+ initContainers:
+ - name: prepare-configuration
+ image: container-registry.softwareheritage.org/swh/infra/swh-apps/utils:20250211.1
+ imagePullPolicy: IfNotPresent
+ command:
+ - /entrypoints/prepare-configuration.sh
+ env:
+
+ volumeMounts:
+ - name: configuration
+ mountPath: /etc/swh
+ - name: configuration-template
+ mountPath: /etc/swh/configuration-template
+ - name: config-utils
+ mountPath: /entrypoints
+ readOnly: true
+ containers:
+ - name: objstorage-ro-pathslicing
+ resources:
+ requests:
+ memory: 512Mi
+ cpu: 500m
+ image: container-registry.softwareheritage.org/swh/infra/swh-apps/objstorage:20250211.3
+ imagePullPolicy: IfNotPresent
+ ports:
+ - containerPort: 5003
+ name: rpc
+ readinessProbe:
+ httpGet:
+ path: /
+ port: rpc
+ initialDelaySeconds: 15
+ failureThreshold: 30
+ periodSeconds: 5
+ livenessProbe:
+ tcpSocket:
+ port: rpc
+ initialDelaySeconds: 10
+ periodSeconds: 5
+ command:
+ - /bin/bash
+ args:
+ - -c
+ - /opt/swh/entrypoint.sh
+ env:
+ - name: WORKERS
+ value: "2"
+ - name: THREADS
+ value: "2"
+ - name: TIMEOUT
+ value: "60"
+ - name: STATSD_HOST
+ value: prometheus-statsd-exporter
+ - name: STATSD_PORT
+ value: "9125"
+ - name: STATSD_TAGS
+ value: deployment:objstorage-ro-pathslicing
+ - name: STATSD_SERVICE_TYPE
+ value: objstorage-ro-pathslicing
+ - name: SWH_CONFIG_FILENAME
+ value: /etc/swh/config.yml
+ - name: SWH_LOG_LEVEL
+ value: "INFO"
+ - name: SWH_LOG_CONFIG_JSON
+ value: /etc/swh/logging/logging-gunicorn.json
+ - name: SWH_SENTRY_ENVIRONMENT
+ value: staging
+ - name: SWH_MAIN_PACKAGE
+ value: swh.objstorage
+ - name: SWH_SENTRY_DSN
+ valueFrom:
+ secretKeyRef:
+ name: common-secrets
+ key: objstorage-sentry-dsn
+ # 'name' secret should exist & include key
+ # if the setting doesn't exist, sentry pushes will be disabled
+ optional: true
+ - name: SWH_SENTRY_DISABLE_LOGGING_EVENTS
+ value: "true"
+ volumeMounts:
+ - name: configuration
+ mountPath: /etc/swh
+ - name: configuration-logging
+ mountPath: /etc/swh/logging
+ - name: pathslicing-rw
+ mountPath: /srv/swh/objects
+ readOnly: true
+
+ volumes:
+ - name: configuration
+ emptyDir: {}
+ - name: configuration-template
+ configMap:
+ name: objstorage-ro-pathslicing-configuration-template
+ items:
+ - key: "config.yml.template"
+ path: "config.yml.template"
+ - name: configuration-logging
+ configMap:
+ name: objstorage-ro-pathslicing-configuration-logging
+ items:
+ - key: "logging-gunicorn.json"
+ path: "logging-gunicorn.json"
+ - name: config-utils
+ configMap:
+ name: config-utils
+ defaultMode: 0555
+ - name: pathslicing-rw
+ persistentVolumeClaim:
+ claimName: objstorage-pathslicing-pvc
+---
+# Source: swh/templates/objstorage/deployment.yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ namespace: swh-cassandra-next-version
name: objstorage-rw-pathslicing
labels:
app: objstorage-rw-pathslicing
spec:
revisionHistoryLimit: 2
selector:
matchLabels:
app: objstorage-rw-pathslicing
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: objstorage-rw-pathslicing
annotations:
- checksum/config: 91292ffc846f0c59408d566d3f460483d5cb29c3bda9ee7d121b3be04dc67c68
+ checksum/config: 90e30ca3c2599e3d9dd0814ee18199f3a8327cbc7db090d29bb13a3cb77cf777
checksum/config-logging: 133d6dfe9a941649b0b690857989ce74b7577bc3b58f9bc1e17fcc1784ae85fa
checksum/config-utils: 94d255131467f84bef964a4c72b2b792c5ebaf711bb1c77829d7cd1007a8ac22
spec:
securityContext:
fsGroup: 1005
nodeSelector:
kubernetes.io/hostname: rancher-node-staging-rke2-metal01
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
@@ -8623,32 +8889,22 @@
configMap:
name: objstorage-rw-pathslicing-configuration-logging
items:
- key: "logging-gunicorn.json"
path: "logging-gunicorn.json"
- name: config-utils
configMap:
name: config-utils
defaultMode: 0555
- name: pathslicing-rw
- ephemeral:
- volumeClaimTemplate:
- metadata:
- labels:
- type: persistent-volume
- spec:
- accessModes:
- - ReadWriteOnce
- resources:
- requests:
- storage: 10Gi
- storageClassName: local-persistent
+ persistentVolumeClaim:
+ claimName: objstorage-pathslicing-pvc
---
# Source: swh/templates/provenance/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: swh-cassandra-next-version
name: provenance-graph-granet
labels:
app: provenance-graph-granet
spec:
@@ -9951,21 +10207,21 @@
app: storage-ro-cassandra
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-ro-cassandra
annotations:
- checksum/config: 5da7dff03c99fc7b8b7c7bba9bcca019f750d07da00ae2ac13df5845229529a3
+ checksum/config: 82ddd066ef3b39ba30a9377b2e3d5f34e9c6771d244cb9966170d8821659043e
checksum/config-logging: 539b1f63c51f751ac609a212d1d29b53ccee1fe8861d4b40cf156cabdbbcc9af
checksum/backend-utils: 32b7a7114be63a5fc6c3745bbc5fc4a9722e45283bafbb74fd640ffc9fe052a2
checksum/config-utils: 94d255131467f84bef964a4c72b2b792c5ebaf711bb1c77829d7cd1007a8ac22
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/storage
------------- diff for environment production namespace swh -------------
No differences
------------- diff for environment production namespace swh-cassandra -------------
No differences
[4] testing read/write from objstorage instances
swhstorage@objstorage-read-write-7489cfd58-ddmgd:/opt/swh$ ls /srv/swh/objects/
test
swhstorage@objstorage-read-write-7489cfd58-ddmgd:/opt/swh$ ls -lah /srv/swh/objects/
total 0
drwxrwxrwx 1 root root 8 Mar 7 12:59 .
drwxr-xr-x 1 root root 14 Mar 7 12:56 ..
-rw-r--r-- 1 swhstorage swhstorage 0 Mar 7 12:59 test
swhstorage@objstorage-read-only-564f4bd647-899rq:/opt/swh$ ls -lah /srv/swh/objects/
total 0
drwxrwxrwx 1 root root 8 Mar 7 12:59 .
drwxr-xr-x 1 root root 14 Mar 7 12:57 ..
-rw-r--r-- 1 swhstorage swhstorage 0 Mar 7 12:59 test[5] helm diff on pvc objstorage deployment
tony@kessel:~/work/swh/sysadm-environment/swh-charts (mr/centralize-pv-pvcs) $ make cc-helm-diff | colordiff
Switched to branch 'production'
Switched to branch 'mr/centralize-pv-pvcs'
[cluster-components] Comparing changes between branches production and mr/centralize-pv-pvcs...
Your branch is ahead of 'origin/production' by 10 commits.
(use "git push" to publish your local commits)
[cluster-components] Generate config in production branch for cluster-components/values/admin-rke2.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/archive-production-rke2.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/archive-staging-rke2.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/default.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/gitlab-production.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/gitlab-staging.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/local-cluster.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/rancher.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/test-staging-rke2.yaml...
[cluster-components] Generate config in mr/centralize-pv-pvcs branch for cluster-components/values/admin-rke2.yaml...
[cluster-components] Generate config in mr/centralize-pv-pvcs branch for cluster-components/values/archive-production-rke2.yaml...
[cluster-components] Generate config in mr/centralize-pv-pvcs branch for cluster-components/values/archive-staging-rke2.yaml...
[cluster-components] Generate config in mr/centralize-pv-pvcs branch for cluster-components/values/default.yaml...
[cluster-components] Generate config in mr/centralize-pv-pvcs branch for cluster-components/values/gitlab-production.yaml...
[cluster-components] Generate config in mr/centralize-pv-pvcs branch for cluster-components/values/gitlab-staging.yaml...
[cluster-components] Generate config in mr/centralize-pv-pvcs branch for cluster-components/values/local-cluster.yaml...
[cluster-components] Generate config in mr/centralize-pv-pvcs branch for cluster-components/values/rancher.yaml...
[cluster-components] Generate config in mr/centralize-pv-pvcs branch for cluster-components/values/test-staging-rke2.yaml...
------------- diff for cluster-components/values/admin-rke2.yaml -------------
No differences
------------- diff for cluster-components/values/archive-production-rke2.yaml -------------
No differences
------------- diff for cluster-components/values/archive-staging-rke2.yaml -------------
--- /tmp/swh-chart.cluster-components.zDrXEMJW/archive-staging-rke2.yaml.before 2025-03-07 16:57:41.174804727 +0100
+++ /tmp/swh-chart.cluster-components.zDrXEMJW/archive-staging-rke2.yaml.after 2025-03-07 16:57:41.722787196 +0100
@@ -142,20 +142,35 @@
name: redis-svix-next-version-ext-config
namespace: swh-cassandra-next-version
labels:
app: redis-svix-next-version
data:
redis-additional.conf: |
save 60 500
appendonly yes
appendfsync everysec
---
+# Source: cluster-config/templates/volumes/persistent-volume-claims.yaml
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+ name: objstorage-pathslicing-pvc
+ namespace: swh-cassandra-next-version
+spec:
+ accessModes:
+ - ReadWriteMany
+ resources:
+ requests:
+ storage: 10Gi
+ storageClassName: local-persistent
+ volumeMode: Filesystem
+---
# Source: cluster-config/templates/scrape-cluster-metrics/service.yaml
apiVersion: v1
kind: Service
metadata:
namespace: swh-cassandra-next-version
name: kafka-next-version-internal-servers-service
labels:
app.kubernetes.io/instance: kafka-cluster
app.kubernetes.io/name: kafka-exporter
spec:
------------- diff for cluster-components/values/default.yaml -------------
No differences
------------- diff for cluster-components/values/gitlab-production.yaml -------------
No differences
------------- diff for cluster-components/values/gitlab-staging.yaml -------------
No differences
------------- diff for cluster-components/values/local-cluster.yaml -------------
No differences
------------- diff for cluster-components/values/rancher.yaml -------------
No differences
------------- diff for cluster-components/values/test-staging-rke2.yaml -------------
No differences
tony@kessel:~/work/swh/sysadm-environment/swh-charts (mr/centralize-pv-pvcs) $ make swh-helm-diff | colordiff
Switched to branch 'production'
Switched to branch 'mr/centralize-pv-pvcs'
Switched to branch 'production'
Switched to branch 'mr/centralize-pv-pvcs'
[swh] Comparing changes between branches production and mr/centralize-pv-pvcs (per environment)...
Your branch is ahead of 'origin/production' by 10 commits.
(use "git push" to publish your local commits)
[swh] Generate config in production branch for environment staging, namespace swh...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra...
[swh] Generate config in production branch for environment staging, namespace next-version...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment staging...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment staging...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment staging...
Your branch is ahead of 'origin/production' by 10 commits.
(use "git push" to publish your local commits)
[swh] Generate config in production branch for environment production, namespace swh...
[swh] Generate config in production branch for environment production, namespace swh-cassandra...
[swh] Generate config in production branch for environment production, namespace next-version...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment production...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment production...
[swh] Generate config in mr/centralize-pv-pvcs branch for environment production...
------------- diff for environment staging namespace swh -------------
No differences
------------- diff for environment staging namespace swh-cassandra -------------
No differences
------------- diff for environment staging namespace next-version -------------
--- /tmp/swh-chart.swh.kqsfjQZ0/staging-next-version.before 2025-03-07 17:13:11.285049455 +0100
+++ /tmp/swh-chart.swh.kqsfjQZ0/staging-next-version.after 2025-03-07 17:13:11.949028214 +0100
@@ -4687,35 +4687,20 @@
- kafka-cluster-kafka-brokers:9092
auto_offset_reset: latest
group_id: staging-next-version-archive-webhooks
object_types:
- origin_visit_status
---
# Source: swh/templates/volumes/persistent-volume-claims.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
- name: objstorage-pathslicing-pvc
- namespace: swh-cassandra-next-version
-spec:
- accessModes:
- - ReadWriteMany
- resources:
- requests:
- storage: 10Gi
- storageClassName: local-persistent
- volumeMode: Filesystem
----
-# Source: swh/templates/volumes/persistent-volume-claims.yaml
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
name: swh-graph-grpc-dataset-example-pvc
namespace: swh-cassandra-next-version
labels:
app: graph-grpc-example
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
------------- diff for environment production namespace swh -------------
No differences
------------- diff for environment production namespace swh-cassandra -------------
No differencesEdited by Antoine R. Dumont
Merge request reports
Compare and
Compare changes
- Side-by-side
- Inline
Files
15Loading