Rework the storage configuration template helpers to be yaml-based
1 unresolved thread
1 unresolved thread
Instead of playing a lot of confusing indentation and whitespace games, incrementally build configuration data structures using template functions. Once the structure is built, just output it as yaml, with no indentation.
When including a template in another, instead of having to pass a
brittle indentation number to the generated template, just parse it with
fromYaml, then include it in the data structure.
helm-diff output only consists of key ordering changes:
helm-diff output
[swh] Comparing changes between branches production and mr/refactor-storage-template-helper (per environment)...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment staging, namespace swh...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra-next-version...
Your branch and 'origin/mr/refactor-storage-template-helper' have diverged,
and have 4 and 1 different commits each, respectively.
(use "git pull" if you want to integrate the remote branch with yours)
[swh] Generate config in mr/refactor-storage-template-helper branch for environment staging...
[swh] Generate config in mr/refactor-storage-template-helper branch for environment staging...
[swh] Generate config in mr/refactor-storage-template-helper branch for environment staging...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment production, namespace swh...
[swh] Generate config in production branch for environment production, namespace swh-cassandra...
[swh] Generate config in production branch for environment production, namespace swh-cassandra-next-version...
Your branch and 'origin/mr/refactor-storage-template-helper' have diverged,
and have 4 and 1 different commits each, respectively.
(use "git pull" if you want to integrate the remote branch with yours)
[swh] Generate config in mr/refactor-storage-template-helper branch for environment production...
[swh] Generate config in mr/refactor-storage-template-helper branch for environment production...
[swh] Generate config in mr/refactor-storage-template-helper branch for environment production...
------------- diff for environment staging namespace swh -------------
--- /tmp/swh-chart.swh.qTNYADrG/staging-swh.before 2024-03-29 16:37:08.900561839 +0100
+++ /tmp/swh-chart.swh.qTNYADrG/staging-swh.after 2024-03-29 16:37:09.276569116 +0100
@@ -51,21 +51,22 @@
# Source: swh/templates/scrubber/journal-checker-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: scrubber-journalchecker-directory-template
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber
+ password=${SCRUBBER_POSTGRESQL_PASSWORD}
journal:
brokers:
- journal2.internal.staging.swh.network:9094
batch_size: 100
cls: kafka
group_id: swh-archive-stg-journalchecker
message.max.bytes: 524288000
on_eof: restart
prefix: swh.journal.objects
sasl.mechanism: SCRAM-SHA-512
@@ -76,21 +77,22 @@
# Source: swh/templates/scrubber/journal-checker-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: scrubber-journalchecker-release-template
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber
+ password=${SCRUBBER_POSTGRESQL_PASSWORD}
journal:
brokers:
- journal2.internal.staging.swh.network:9094
batch_size: 200
cls: kafka
group_id: swh-archive-stg-journalchecker
message.max.bytes: 524288000
on_eof: restart
prefix: swh.journal.objects
privileged: true
@@ -102,21 +104,22 @@
# Source: swh/templates/scrubber/journal-checker-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: scrubber-journalchecker-revision-template
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber
+ password=${SCRUBBER_POSTGRESQL_PASSWORD}
journal:
brokers:
- journal2.internal.staging.swh.network:9094
cls: kafka
group_id: swh-archive-stg-journalchecker
message.max.bytes: 524288000
on_eof: restart
prefix: swh.journal.objects
privileged: true
sasl.mechanism: SCRAM-SHA-512
@@ -127,21 +130,22 @@
# Source: swh/templates/scrubber/journal-checker-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: scrubber-journalchecker-snapshot-template
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber
+ password=${SCRUBBER_POSTGRESQL_PASSWORD}
journal:
brokers:
- journal2.internal.staging.swh.network:9094
cls: kafka
group_id: swh-archive-stg-journalchecker
message.max.bytes: 524288000
on_eof: restart
prefix: swh.journal.objects
sasl.mechanism: SCRAM-SHA-512
sasl.password: ${BROKER_USER_PASSWORD}
@@ -151,41 +155,41 @@
# Source: swh/templates/scrubber/storage-checker-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: scrubber-storagechecker-directory-hashes-template
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber
+ password=${SCRUBBER_POSTGRESQL_PASSWORD}
storage:
-
- cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=guest dbname=swh password=${POSTGRESQL_PASSWORD}
+ cls: postgresql
+ db: host=db1.internal.staging.swh.network port=5432 user=guest dbname=swh password=${POSTGRESQL_PASSWORD}
---
# Source: swh/templates/scrubber/storage-checker-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: scrubber-storagechecker-directory-references-template
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber
+ password=${SCRUBBER_POSTGRESQL_PASSWORD}
storage:
-
- cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=guest dbname=swh password=${POSTGRESQL_PASSWORD}
+ cls: postgresql
+ db: host=db1.internal.staging.swh.network port=5432 user=guest dbname=swh password=${POSTGRESQL_PASSWORD}
---
# Source: swh/templates/statsd-exporter/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-statsd-exporter
namespace: swh
data:
config.yml: |
defaults:
@@ -894,33 +898,32 @@
namespace: swh
name: storage-postgresql-read-write-configuration-template
data:
config.yml.template: |
storage:
cls: pipeline
steps:
- cls: record_references
- cls: postgresql
db: host=db1.internal.staging.swh.network port=5432 user=swh dbname=swh password=${POSTGRESQL_PASSWORD}
- objstorage:
- cls: remote
- url: http://objstorage-multiplexer-read-write-rpc-ingress
-
journal_writer:
- cls: kafka
+ anonymize: true
brokers:
- journal2.internal.staging.swh.network
+ clientId: swh.storage.journal_writer.${HOSTNAME}
+ cls: kafka
prefix: swh.journal.objects
- client_id: swh.storage.journal_writer.${HOSTNAME}
- anonymize: true
producer_config:
message.max.bytes: 1000000000
+ objstorage:
+ cls: remote
+ url: http://objstorage-multiplexer-read-write-rpc-ingress
---
# Source: swh/templates/storage/cronjob-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: cronjob-create-object-reference-partitions-configuration-template
data:
config.yml.template: |
storage:
@@ -932,21 +935,22 @@
# Source: swh/templates/toolbox/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: toolbox-scrubber-journal-template
namespace: swh
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber
+ password=${SCRUBBER_POSTGRESQL_PASSWORD}
journal:
brokers:
- journal2.internal.staging.swh.network:9094
cls: kafka
group_id: swh-archive-stg-journalchecker
message.max.bytes: 524288000
on_eof: restart
prefix: swh.journal.objects
sasl.mechanism: SCRAM-SHA-512
sasl.password: ${BROKER_USER_PASSWORD}
@@ -955,26 +959,26 @@
---
# Source: swh/templates/toolbox/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: toolbox-scrubber-storage-template
namespace: swh
data:
config.yml.template: |
storage:
-
- cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=guest dbname=swh password=${POSTGRESQL_PASSWORD}
+ cls: postgresql
+ db: host=db1.internal.staging.swh.network port=5432 user=guest dbname=swh password=${POSTGRESQL_PASSWORD}
scrubber:
cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber
+ password=${SCRUBBER_POSTGRESQL_PASSWORD}
---
# Source: swh/templates/toolbox/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: toolbox-storage-template
namespace: swh
data:
config.yml.template: |
storage:
@@ -1829,21 +1833,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scrubber-journalchecker-directory
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: a142a5207d8d5a158b539d935b0b75953ba9650eadfd91cd4fd8700c9d5f5d4b
+ checksum/config: 481a024c69c683277b7d3d76ee49d95fef7b2c594e0e38ac3f6d2a3c58d14c1e
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scrubber
operator: In
values:
@@ -1970,21 +1974,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scrubber-journalchecker-release
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: 45bb62ef95b6282856c3cef6780aa6d0adc52ceb94972461e554a99a004bc93b
+ checksum/config: cc1c4b15eb9e0ec8a5fa2e7bcfbaa8e19d9bf0c04e33bb9f1920385aebc47e0c
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scrubber
operator: In
values:
@@ -2111,21 +2115,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scrubber-journalchecker-revision
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: 80725683360fcab3686491dad8409f29decbba0b8f9ac03f9e860510a08aec18
+ checksum/config: 01aeb79d3e103482d047d37bf7137d265ff9e49f520866c501f676ac83951169
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scrubber
operator: In
values:
@@ -2252,21 +2256,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scrubber-journalchecker-snapshot
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: 1decc7785aaeacbc3590c0885518b3f62728abd4ecd3bc5b207060acdc6e1e5b
+ checksum/config: 613babb7838529543afd40fbe4e0e4800837b91cae1fb7221cb4dd2c878ff076
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scrubber
operator: In
values:
@@ -2393,21 +2397,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scrubber-storagechecker-directory-hashes
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: a57b35b73615ca40284037a0848fab89d6825e6cfd904a8da85b4cc054adf6b2
+ checksum/config: 633e8bbe45e19c0bb5a207f3e9e76d33a6dde6f7ff9df40cc5ebbe4d3d099534
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scrubber
operator: In
values:
@@ -2546,21 +2550,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scrubber-storagechecker-directory-references
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: 219681c90ac884a0ac06e33d63e00769ace3f27975b29134f6430fe1e487e2d8
+ checksum/config: 4aed08fc3913755045e21f7ebd2b66ccc49323df9ce719fe613fd7c9a89d7fff
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scrubber
operator: In
values:
@@ -4417,21 +4421,21 @@
app: storage-postgresql-read-write
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-postgresql-read-write
annotations:
- checksum/config: 99ed1c8c42e3e126fffe5b57f93461767e6823cc0c04ff905a5e33fbdcbce48b
+ checksum/config: fee18359009625b47b8ad04998eae7dab8d6c02aba3a962af769c69beb0b19d3
checksum/database-utils: 931b7f92b50eae0ecb6203c3e0e48f7c363a99ac0064451187dc12c6eaf0a793
checksum/config-utils: d75ca13b805bce6a8ab59c8e24c938f2283108f6a79134f6e71db86308651dc6
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/storage
operator: In
@@ -4556,21 +4560,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: swh-toolbox
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: f826f7bced54609138a93052d0b80d9634c8fb6640f44c2816664794aedfddad
+ checksum/config: cd8a520b43a4a685a81b1d5d2194fb98897c46b8e62226983d08d762d59cce84
checksum/configScript: 5f0986a71adce81bcdab1a3886cbd18e67d89370b3a20b342248c5f15be406ff
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/toolbox
operator: In
values:
------------- diff for environment staging namespace swh-cassandra -------------
--- /tmp/swh-chart.swh.qTNYADrG/staging-swh-cassandra.before 2024-03-29 16:37:09.088565478 +0100
+++ /tmp/swh-chart.swh.qTNYADrG/staging-swh-cassandra.after 2024-03-29 16:37:09.472572908 +0100
@@ -856,21 +856,22 @@
# Source: swh/templates/indexer-storage/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: indexer-storage-rpc-configuration-template
data:
config.yml.template: |
indexer_storage:
cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=swh-indexer dbname=swh-indexer password=${POSTGRESQL_PASSWORD}
+ db: host=db1.internal.staging.swh.network port=5432 user=swh-indexer dbname=swh-indexer
+ password=${POSTGRESQL_PASSWORD}
journal_writer:
client_id: swh.idx_storage.journal_writer.${HOSTNAME}
cls: kafka
prefix: swh.journal.indexed
producer_config:
message.max.bytes: 1000000000
brokers:
- journal2.internal.staging.swh.network
---
# Source: swh/templates/indexers/configmap-utils.yaml
@@ -4516,21 +4517,22 @@
# Source: swh/templates/scheduler/rpc-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: scheduler-rpc-configuration-template
data:
config.yml.template: |
scheduler:
cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=swh-scheduler dbname=swh-scheduler password=${POSTGRESQL_PASSWORD}
+ db: host=db1.internal.staging.swh.network port=5432 user=swh-scheduler dbname=swh-scheduler
+ password=${POSTGRESQL_PASSWORD}
---
# Source: swh/templates/scheduler/update-metrics-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: scheduler-update-metrics-configuration-template
namespace: swh-cassandra
data:
config.yml.template: |
scheduler:
@@ -4540,33 +4542,34 @@
# Source: swh/templates/scrubber/storage-checker-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: scrubber-storagechecker-directory-references-template
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber
+ password=${SCRUBBER_POSTGRESQL_PASSWORD}
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-ro
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
---
# Source: swh/templates/search/journal-client-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: search-journal-client-indexed-configuration-template
namespace: swh-cassandra
data:
config.yml.template: |
search:
@@ -4694,32 +4697,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-content-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -4739,32 +4742,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-directory-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -4784,32 +4787,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-extid-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -4829,32 +4832,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-metadata-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -4875,32 +4878,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-origin-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -4920,32 +4923,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-origin-visit-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -4965,32 +4968,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-origin-visit-status-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -5010,32 +5013,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-raw-extrinsic-metadata-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -5055,32 +5058,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-release-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -5100,32 +5103,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-revision-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -5145,32 +5148,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-skipped-content-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -5190,32 +5193,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-snapshot-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -5235,108 +5238,110 @@
---
# Source: swh/templates/storage/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-cassandra-configuration-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
- objstorage:
- cls: remote
- url: http://objstorage-multiplexer-read-write-rpc-ingress
-
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
journal_writer:
- cls: kafka
+ anonymize: true
brokers:
- journal2.internal.staging.swh.network
+ clientId: swh.storage-cassandra.journal_writer.${HOSTNAME}
+ cls: kafka
prefix: swh.journal.objects
- client_id: swh.storage-cassandra.journal_writer.${HOSTNAME}
- anonymize: true
producer_config:
message.max.bytes: 1000000000
+ keyspace: swh
+ objstorage:
+ cls: remote
+ url: http://objstorage-multiplexer-read-write-rpc-ingress
---
# Source: swh/templates/toolbox/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: toolbox-indexer-storage-template
namespace: swh-cassandra
data:
config.yml.template: |
indexer_storage:
cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=swh-indexer dbname=swh-indexer password=${POSTGRESQL_PASSWORD}
+ db: host=db1.internal.staging.swh.network port=5432 user=swh-indexer dbname=swh-indexer
+ password=${POSTGRESQL_PASSWORD}
---
# Source: swh/templates/toolbox/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: toolbox-scheduler-template
namespace: swh-cassandra
data:
config.yml.template: |
scheduler:
cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=swh-scheduler dbname=swh-scheduler password=${POSTGRESQL_PASSWORD}
+ db: host=db1.internal.staging.swh.network port=5432 user=swh-scheduler dbname=swh-scheduler
+ password=${POSTGRESQL_PASSWORD}
celery:
task_broker: amqp://swhproducer:${AMQP_PASSWORD}@scheduler0.internal.staging.swh.network:5672/%2f
---
# Source: swh/templates/toolbox/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: toolbox-scrubber-storage-template
namespace: swh-cassandra
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-ro
-
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
scrubber:
cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=db1.internal.staging.swh.network port=5432 user=swh-scrubber dbname=swh-scrubber
+ password=${SCRUBBER_POSTGRESQL_PASSWORD}
---
# Source: swh/templates/toolbox/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: toolbox-vault-template
namespace: swh-cassandra
data:
config.yml.template: |
vault:
cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=swh-vault dbname=swh-vault password=${POSTGRESQL_PASSWORD}
+ db: host=db1.internal.staging.swh.network port=5432 user=swh-vault dbname=swh-vault
+ password=${POSTGRESQL_PASSWORD}
---
# Source: swh/templates/toolbox/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: toolbox-webhooks-template
namespace: swh-cassandra
data:
config.yml.template: |
webhooks:
@@ -5698,21 +5703,22 @@
# Source: swh/templates/vault/rpc-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: vault-rpc-configuration-template
data:
config.yml.template: |
vault:
cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=swh-vault dbname=swh-vault password=${POSTGRESQL_PASSWORD}
+ db: host=db1.internal.staging.swh.network port=5432 user=swh-vault dbname=swh-vault
+ password=${POSTGRESQL_PASSWORD}
storage:
cls: pipeline
steps:
- cls: retry
- cls: remote
url: http://storage-postgresql-read-only-rpc-ingress
scheduler:
cls: remote
url: http://scheduler.internal.staging.swh.network
objstorage:
@@ -8223,21 +8229,21 @@
app: indexer-storage-rpc
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: indexer-storage-rpc
annotations:
- checksum/config: bfa86ff57a3c5ce8420e51d66b742b91336a8785f5c84091bbbf2bf9fc10c965
+ checksum/config: 5ce708e81c9dbf7d47e360b2206348a3ef041a7c4fcbd75e269ad85901f9c725
checksum/database-utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/rpc
operator: In
values:
@@ -15689,21 +15695,21 @@
app: scheduler-rpc
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scheduler-rpc
annotations:
- checksum/config: 0701be2312409903eb6f77d9b2adfd469be7f8b94a853ec764283066b486022e
+ checksum/config: 7703e348e073fc0114bc9bea025180b2a85ebf8f2bd7b326e58ad9baa3f62344
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scheduler
operator: In
values:
- "true"
@@ -15816,21 +15822,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scrubber-storagechecker-directory-references
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: 0b0695d74ddf47fce9241311f8070c7845d019bb4ff30d883866b4cf518c574b
+ checksum/config: e6399effa44239bbe882531cc8c285d0a4656e5041e57d773d4a143318615bcc
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scrubber
operator: In
values:
@@ -16327,21 +16333,21 @@
app: storage-replayer-content
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-content
annotations:
- checksum/config: 74673c4289fe0881b0b2ff369c979626472ac305473d264da86e9c0c6a241336
+ checksum/config: d5680e6ca19e544abde136e93d270f30716699eb148965aa3ebdcd48b3b8b78d
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -16455,21 +16461,21 @@
app: storage-replayer-directory
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-directory
annotations:
- checksum/config: 995564bd84a3cefb2634627bd77721b7c8ed519ca2b851301c9d36bd56a630e4
+ checksum/config: 4fb0ef79bdd5ce6ff3e8a44abf4f74c31391d3a45ef43b3e0918399ff380ff07
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -16583,21 +16589,21 @@
app: storage-replayer-extid
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-extid
annotations:
- checksum/config: d731e2c666d2c82555abe96706c14f7426bf1c78ddf83e7ee8ffee741a149cd6
+ checksum/config: e3094379b0ee7982a27a1201d147cba3e171fa174a13fbd7b63f9bfc82842110
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -16711,21 +16717,21 @@
app: storage-replayer-metadata
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-metadata
annotations:
- checksum/config: 99fff6f6daca34620efdb430c0662825cc9c7e3e44e03d85b99e0bc4c24ab3c4
+ checksum/config: ee24107408cd362c96945d5d847128a3d4b2b8f94b44de51b5c99f375ca9bb07
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -16839,21 +16845,21 @@
app: storage-replayer-origin
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-origin
annotations:
- checksum/config: 9980410c4c1ed3bbbde5a6a777e78f100fc09ea68dcc8d6f46892d5cf6eaae05
+ checksum/config: 5fce86dfaa53eb1ef7927017751c823579bd3c96d7f9e0d46f6a2f8ba5fd8cef
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -16967,21 +16973,21 @@
app: storage-replayer-origin-visit
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-origin-visit
annotations:
- checksum/config: 79c3887276b574a89abb234555bc21cdeb699916c95edca63e8d6a52fa3390d7
+ checksum/config: 477853a0fd3ad21dcae5667e635e5c6c8e9a191521b1037939d59f8c6755165f
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -17095,21 +17101,21 @@
app: storage-replayer-origin-visit-status
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-origin-visit-status
annotations:
- checksum/config: 4b08ab81b404ad278188e8904229c6e68d19de9def463369e62432aa046f0dbb
+ checksum/config: 67a771e899841d09d44fe3db548c63dd33e52554f83dcc8866b65a9442ffcb60
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -17223,21 +17229,21 @@
app: storage-replayer-raw-extrinsic-metadata
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-raw-extrinsic-metadata
annotations:
- checksum/config: b957874c1c20e156da22f4967bfabb84cbf58f369a8df7ea8be0d02a454c0a7b
+ checksum/config: abf36180324ee584007ec0eba4951749584bdc68524496baa24bc9eb7e7419b4
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -17351,21 +17357,21 @@
app: storage-replayer-release
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-release
annotations:
- checksum/config: ed8166eae4a49809de794a4ea3b4d769d5b321f8064bdc6c6dad6411cb22faf4
+ checksum/config: e0e5e283a58d4e5d50ca446271a47ce134adc5fb5448bb762c2a361ff664e319
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -17479,21 +17485,21 @@
app: storage-replayer-revision
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-revision
annotations:
- checksum/config: bdbba198748e5ce312963daaa57fcb5a451215761243d27475708d26a22a3152
+ checksum/config: 3085e6cf15602aa5a7c58aa8ca29a3cd8ab69b2dfd2dc5d36451c2a9584e7365
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -17607,21 +17613,21 @@
app: storage-replayer-skipped-content
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-skipped-content
annotations:
- checksum/config: 71d056e260a8f43476b198183c0e63e0914cc4f9a5d10ba1dbbc1090464e07c6
+ checksum/config: ac73aee02acfdb4f04fe33dceb1226c8a603754b5c601f3253670934b750d3fb
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -17735,21 +17741,21 @@
app: storage-replayer-snapshot
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-snapshot
annotations:
- checksum/config: 699850c7fa8d3940d1e6850e3c0cce8d91798cb8a40767b16503905314836466
+ checksum/config: 51143c9b98771e2d1fa6682472e5934d6ab343bd9de6864ff4369b5acdf736e7
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -17863,21 +17869,21 @@
app: storage-cassandra
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-cassandra
annotations:
- checksum/config: d36a1e69ab3e06eca9f2efc20e67f01944777580c960a5220478f1962946ae0a
+ checksum/config: 868790d74a6b6b725d5be5383c9e5869769c4a404876631002244e9557e0e099
checksum/database-utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
checksum/config-utils: 13a26f6add17e96ce01550153c77dcd48de60241a3f4db3c93d5467234be2a7f
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/storage
operator: In
@@ -18015,21 +18021,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: swh-toolbox
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: df5d1aa136684d49d96cb247e608bd3c30b74943b7fd8ae9d0e30fca3c736d2a
+ checksum/config: 13e3e922e0047d68e91df2e4184b7c372247116d52b49a6bcf8e6c089663aa65
checksum/configScript: ac6ee536c8e0fc712036a0f9b01dca9d0aec5289436804e6dcdf7a4dcf548de3
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/toolbox
operator: In
values:
@@ -18280,21 +18286,21 @@
app: vault-rpc
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: vault-rpc
annotations:
- checksum/config: 246b99ea64f7ad0594498496794b1cdcc479dad9e6b582997834d783d29a43d1
+ checksum/config: e896fe6cbe9d38d03b6f487df1d81f73953e7398cc2bf2ce008877f4ebfa4eb4
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/rpc
operator: In
values:
- "true"
------------- diff for environment staging namespace swh-cassandra-next-version -------------
--- /tmp/swh-chart.swh.qTNYADrG/staging-swh-cassandra-next-version.before 2024-03-29 16:37:09.180567257 +0100
+++ /tmp/swh-chart.swh.qTNYADrG/staging-swh-cassandra-next-version.after 2024-03-29 16:37:09.568574766 +0100
@@ -861,32 +861,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra-next-version
name: storage-replayer-configuration-content-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -906,32 +906,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra-next-version
name: storage-replayer-configuration-directory-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -951,32 +951,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra-next-version
name: storage-replayer-configuration-extid-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -996,32 +996,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra-next-version
name: storage-replayer-configuration-metadata-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -1042,32 +1042,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra-next-version
name: storage-replayer-configuration-origin-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -1087,32 +1087,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra-next-version
name: storage-replayer-configuration-origin-visit-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -1132,32 +1132,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra-next-version
name: storage-replayer-configuration-origin-visit-status-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -1177,32 +1177,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra-next-version
name: storage-replayer-configuration-raw-extrinsic-metadata-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -1222,32 +1222,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra-next-version
name: storage-replayer-configuration-release-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -1267,32 +1267,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra-next-version
name: storage-replayer-configuration-revision-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -1312,32 +1312,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra-next-version
name: storage-replayer-configuration-skipped-content-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -1357,32 +1357,32 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra-next-version
name: storage-replayer-configuration-snapshot-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- journal2.internal.staging.swh.network:9094
sasl.username: swh-cassandra-stg
sasl.password: ${BROKER_USER_PASSWORD}
security.protocol: sasl_ssl
@@ -1402,45 +1402,44 @@
---
# Source: swh/templates/storage/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra-next-version
name: storage-cassandra-configuration-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra1.internal.staging.swh.network
- - cassandra2.internal.staging.swh.network
- - cassandra3.internal.staging.swh.network
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
- objstorage:
- cls: remote
- url: http://objstorage-multiplexer-read-write-rpc-ingress
-
+ hosts:
+ - cassandra1.internal.staging.swh.network
+ - cassandra2.internal.staging.swh.network
+ - cassandra3.internal.staging.swh.network
journal_writer:
- cls: kafka
+ anonymize: true
brokers:
- journal2.internal.staging.swh.network
+ clientId: swh.storage-cassandra.journal_writer.${HOSTNAME}
+ cls: kafka
prefix: swh.journal.objects
- client_id: swh.storage-cassandra.journal_writer.${HOSTNAME}
- anonymize: true
producer_config:
message.max.bytes: 1000000000
+ keyspace: swh
+ objstorage:
+ cls: remote
+ url: http://objstorage-multiplexer-read-write-rpc-ingress
---
# Source: swh/templates/utils/config-utils.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: config-utils
namespace: swh-cassandra-next-version
data:
prepare-configuration.sh: |
#!/bin/bash
@@ -1607,21 +1606,22 @@
# Source: swh/templates/vault/rpc-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra-next-version
name: vault-rpc-configuration-template
data:
config.yml.template: |
vault:
cls: postgresql
- db: host=db1.internal.staging.swh.network port=5432 user=swh-vault dbname=swh-vault password=${POSTGRESQL_PASSWORD}
+ db: host=db1.internal.staging.swh.network port=5432 user=swh-vault dbname=swh-vault
+ password=${POSTGRESQL_PASSWORD}
storage:
cls: pipeline
steps:
- cls: retry
- cls: remote
url: http://storage-postgresql-read-only-rpc-ingress
scheduler:
cls: remote
url: http://scheduler.internal.staging.swh.network
objstorage:
@@ -3372,21 +3372,21 @@
app: storage-replayer-content
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-content
annotations:
- checksum/config: dbd1de02f5095d8798d6123b1ccc7e46373e2535b30064559ca0a98cbb26be5d
+ checksum/config: f5eec2092742fe85cc1c257edb74589e0c028f8e210d0fdf0c923713e933dcd6
checksum/config_utils: dd7d83d547731ea7598fa298565f3b1fe41eb657ae93bab75e15b722dbcc71ae
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -3500,21 +3500,21 @@
app: storage-replayer-directory
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-directory
annotations:
- checksum/config: b59da477dccdaad4129be94e104df5855b3b550c7df5d5c51cc804a0ec99ea4e
+ checksum/config: 809bab36ca0dc72d6ad97eaf35d10c76bb46e20bd6e46dea4cd2ed77a4f9fa70
checksum/config_utils: dd7d83d547731ea7598fa298565f3b1fe41eb657ae93bab75e15b722dbcc71ae
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -3628,21 +3628,21 @@
app: storage-replayer-extid
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-extid
annotations:
- checksum/config: b4f71667efae23004d3806e40754074af8ecfc2cce3e977587e33cdb558651e8
+ checksum/config: 730c81eca19d3e1b5c7992b65dd8c025c153dd6ee72b630500d2bf9ba34a5523
checksum/config_utils: dd7d83d547731ea7598fa298565f3b1fe41eb657ae93bab75e15b722dbcc71ae
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -3756,21 +3756,21 @@
app: storage-replayer-metadata
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-metadata
annotations:
- checksum/config: 5a61f8b9e28e625a7b262c1fc9bde58b7b101df00adb4c928ca0e774cdb32ab0
+ checksum/config: 00488a2cbecaf6caa2e834c2a939859559d593dc7c92585dc7afa25e4215484d
checksum/config_utils: dd7d83d547731ea7598fa298565f3b1fe41eb657ae93bab75e15b722dbcc71ae
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -3884,21 +3884,21 @@
app: storage-replayer-origin
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-origin
annotations:
- checksum/config: 0a7546e3af563a2ad84e3bb7bb09747f0db36a48204afe59f178f8dd29c78e8f
+ checksum/config: f2d3e1743e69612cb49351e467652ebab2dea407862a0b4231d9b316011a68f3
checksum/config_utils: dd7d83d547731ea7598fa298565f3b1fe41eb657ae93bab75e15b722dbcc71ae
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -4012,21 +4012,21 @@
app: storage-replayer-origin-visit
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-origin-visit
annotations:
- checksum/config: a9aa6fae2152989e375278357dba57bb57a263bf97fa3753b7308b69043d9b16
+ checksum/config: 0fd1ff57cea4c21913cc327c4f914fdd83daf8bf910ffc2b0711a250f576bff0
checksum/config_utils: dd7d83d547731ea7598fa298565f3b1fe41eb657ae93bab75e15b722dbcc71ae
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -4140,21 +4140,21 @@
app: storage-replayer-origin-visit-status
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-origin-visit-status
annotations:
- checksum/config: 8c75c9a3a55fe1e0bb758b2e9657465a5e740e58a761fc411b00f30359dee883
+ checksum/config: 838684529c6bd42fe87bd2858c6e41cab59eadb8168ec0db97d98e75fb1ad697
checksum/config_utils: dd7d83d547731ea7598fa298565f3b1fe41eb657ae93bab75e15b722dbcc71ae
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -4268,21 +4268,21 @@
app: storage-replayer-raw-extrinsic-metadata
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-raw-extrinsic-metadata
annotations:
- checksum/config: b172657e96cd82ebad8eaa8236db4fa2eb1356540a390037d9579748c3b88b29
+ checksum/config: b939073d989afe88a411110a6410d38fb1bd8b55ed285d4e97c9b4ebb5852fea
checksum/config_utils: dd7d83d547731ea7598fa298565f3b1fe41eb657ae93bab75e15b722dbcc71ae
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -4396,21 +4396,21 @@
app: storage-replayer-release
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-release
annotations:
- checksum/config: a63f069b8a80bd2b7fac1f806595d67c418d280de0ce9f7bf326865369f8cdbf
+ checksum/config: f8e15a3d8ce7784ae7b0831b4dc0dcecdc8b40df81c3f8fe510b57dc21d27e20
checksum/config_utils: dd7d83d547731ea7598fa298565f3b1fe41eb657ae93bab75e15b722dbcc71ae
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -4524,21 +4524,21 @@
app: storage-replayer-revision
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-revision
annotations:
- checksum/config: 8df6295db49ba64d9573fc6df43fddd8ee02261e577e37443797141f72a8fe91
+ checksum/config: 98ce591a68d6cf980d3f0ec318f6184fec96f22dfd5847c92612de594a534ea8
checksum/config_utils: dd7d83d547731ea7598fa298565f3b1fe41eb657ae93bab75e15b722dbcc71ae
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -4652,21 +4652,21 @@
app: storage-replayer-skipped-content
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-skipped-content
annotations:
- checksum/config: c557fabf9b20b68407c1f631175e9fce60d9206004d8f2f42c4ad49d8f80a540
+ checksum/config: 2e06843bc1245901371688391ed10dbe8d88b22ab127575254e0120c99dec8a5
checksum/config_utils: dd7d83d547731ea7598fa298565f3b1fe41eb657ae93bab75e15b722dbcc71ae
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -4780,21 +4780,21 @@
app: storage-replayer-snapshot
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-snapshot
annotations:
- checksum/config: cd90396600ab7d855e6a4162109c755fe25924f9eb23c4b9235cc1054231b8f7
+ checksum/config: 315c528fbcfedb699dbef895306ea973684a86c8a8e33581512f0f1adb8980e3
checksum/config_utils: dd7d83d547731ea7598fa298565f3b1fe41eb657ae93bab75e15b722dbcc71ae
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -4908,21 +4908,21 @@
app: storage-cassandra
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-cassandra
annotations:
- checksum/config: a5ed10fed892156c66d77df6cbbd4e06411fa1b104e0e36b47115cbbdee6971b
+ checksum/config: 71831b8763188f2a40d1f8cb6152583ef40c62febe0580978588522e1ce6ddc4
checksum/database-utils: dd7d83d547731ea7598fa298565f3b1fe41eb657ae93bab75e15b722dbcc71ae
checksum/config-utils: 94d255131467f84bef964a4c72b2b792c5ebaf711bb1c77829d7cd1007a8ac22
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/storage
operator: In
@@ -5060,21 +5060,21 @@
app: vault-rpc
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: vault-rpc
annotations:
- checksum/config: f5b5a6fa10a0b85c2bb41374736ade4d3bcb65ad8b2ec7f73df4dcefd1ad5e2e
+ checksum/config: 29a404a93e2c55f1ff8d1be1a18f5d385b9956a88149478fdc7b41853ab5e54b
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/rpc
operator: In
values:
- "true"
------------- diff for environment production namespace swh -------------
--- /tmp/swh-chart.swh.qTNYADrG/production-swh.before 2024-03-29 16:37:09.816579566 +0100
+++ /tmp/swh-chart.swh.qTNYADrG/production-swh.after 2024-03-29 16:37:10.176586532 +0100
@@ -736,33 +736,35 @@
# Source: swh/templates/indexer-storage/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: indexer-storage-read-only-configuration-template
data:
config.yml.template: |
indexer_storage:
cls: postgresql
- db: host=postgresql-indexer-rw.internal.softwareheritage.org port=5432 user=guest dbname=softwareheritage-indexer password=${POSTGRESQL_PASSWORD}
+ db: host=postgresql-indexer-rw.internal.softwareheritage.org port=5432 user=guest
+ dbname=softwareheritage-indexer password=${POSTGRESQL_PASSWORD}
---
# Source: swh/templates/indexer-storage/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: indexer-storage-read-write-configuration-template
data:
config.yml.template: |
indexer_storage:
cls: postgresql
- db: host=postgresql-indexer-rw.internal.softwareheritage.org port=5432 user=swhstorage dbname=softwareheritage-indexer password=${POSTGRESQL_PASSWORD}
+ db: host=postgresql-indexer-rw.internal.softwareheritage.org port=5432 user=swhstorage
+ dbname=softwareheritage-indexer password=${POSTGRESQL_PASSWORD}
journal_writer:
client_id: swh.idx_storage.journal_writer.${HOSTNAME}
cls: kafka
prefix: swh.journal.indexed
producer_config:
message.max.bytes: 1000000000
brokers:
- kafka1.internal.softwareheritage.org
- kafka2.internal.softwareheritage.org
- kafka3.internal.softwareheritage.org
@@ -6587,21 +6589,22 @@
# Source: swh/templates/scheduler/rpc-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: scheduler-rpc-configuration-template
data:
config.yml.template: |
scheduler:
cls: postgresql
- db: host=postgresql-scheduler-rw.internal.softwareheritage.org port=5432 user=swhscheduler dbname=softwareheritage-scheduler password=${POSTGRESQL_PASSWORD}
+ db: host=postgresql-scheduler-rw.internal.softwareheritage.org port=5432 user=swhscheduler
+ dbname=softwareheritage-scheduler password=${POSTGRESQL_PASSWORD}
---
# Source: swh/templates/scheduler/update-metrics-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: scheduler-update-metrics-configuration-template
namespace: swh
data:
config.yml.template: |
scheduler:
@@ -6611,21 +6614,22 @@
# Source: swh/templates/scrubber/journal-checker-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: scrubber-journalchecker-release-template
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber
+ dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
journal:
brokers:
- kafka1.internal.softwareheritage.org:9094
- kafka2.internal.softwareheritage.org:9094
- kafka3.internal.softwareheritage.org:9094
- kafka4.internal.softwareheritage.org:9094
batch_size: 200
cls: kafka
group_id: swh-archive-prod-journalchecker
message.max.bytes: "524288000"
@@ -6640,21 +6644,22 @@
# Source: swh/templates/scrubber/journal-checker-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: scrubber-journalchecker-revision-template
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber
+ dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
journal:
brokers:
- kafka1.internal.softwareheritage.org:9094
- kafka2.internal.softwareheritage.org:9094
- kafka3.internal.softwareheritage.org:9094
- kafka4.internal.softwareheritage.org:9094
cls: kafka
group_id: swh-archive-prod-journalchecker
message.max.bytes: "524288000"
on_eof: restart
@@ -6668,21 +6673,22 @@
# Source: swh/templates/scrubber/journal-checker-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: scrubber-journalchecker-snapshot-template
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber
+ dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
journal:
brokers:
- kafka1.internal.softwareheritage.org:9094
- kafka2.internal.softwareheritage.org:9094
- kafka3.internal.softwareheritage.org:9094
- kafka4.internal.softwareheritage.org:9094
cls: kafka
group_id: swh-archive-prod-journalchecker
message.max.bytes: "524288000"
on_eof: restart
@@ -6695,105 +6701,111 @@
# Source: swh/templates/scrubber/storage-checker-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: scrubber-storagechecker-primary-directory-hashes-template
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber
+ dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
storage:
-
- cls: postgresql
- db: host=postgresql-storage-rw.internal.softwareheritage.org port=5432 user=guest dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
+ cls: postgresql
+ db: host=postgresql-storage-rw.internal.softwareheritage.org port=5432 user=guest
+ dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
---
# Source: swh/templates/scrubber/storage-checker-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: scrubber-storagechecker-primary-directory-references-template
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber
+ dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
storage:
-
- cls: postgresql
- db: host=postgresql-storage-rw.internal.softwareheritage.org port=5432 user=guest dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
+ cls: postgresql
+ db: host=postgresql-storage-rw.internal.softwareheritage.org port=5432 user=guest
+ dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
---
# Source: swh/templates/scrubber/storage-checker-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: scrubber-storagechecker-secondary-directory-hashes-template
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber
+ dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
storage:
-
- cls: postgresql
- db: host=postgresql-storage-replica.internal.softwareheritage.org port=5432 user=guest dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
+ cls: postgresql
+ db: host=postgresql-storage-replica.internal.softwareheritage.org port=5432 user=guest
+ dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
---
# Source: swh/templates/scrubber/storage-checker-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: scrubber-storagechecker-secondary-directory-references-template
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber
+ dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
storage:
-
- cls: postgresql
- db: host=postgresql-storage-replica.internal.softwareheritage.org port=5432 user=guest dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
+ cls: postgresql
+ db: host=postgresql-storage-replica.internal.softwareheritage.org port=5432 user=guest
+ dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
---
# Source: swh/templates/scrubber/storage-checker-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: scrubber-storagechecker-secondary-snapshot-hashes-template
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber
+ dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
storage:
-
- cls: postgresql
- db: host=postgresql-storage-replica.internal.softwareheritage.org port=5432 user=guest dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
+ cls: postgresql
+ db: host=postgresql-storage-replica.internal.softwareheritage.org port=5432 user=guest
+ dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
---
# Source: swh/templates/scrubber/storage-checker-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: scrubber-storagechecker-secondary-snapshot-references-template
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber
+ dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
storage:
-
- cls: postgresql
- db: host=postgresql-storage-replica.internal.softwareheritage.org port=5432 user=guest dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
+ cls: postgresql
+ db: host=postgresql-storage-replica.internal.softwareheritage.org port=5432 user=guest
+ dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
---
# Source: swh/templates/search/journal-client-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: search-journal-client-indexed-configuration-template
namespace: swh
data:
config.yml.template: |
search:
@@ -6934,21 +6946,22 @@
kind: ConfigMap
metadata:
namespace: swh
name: storage-postgresql-azure-readonly-configuration-template
data:
config.yml.template: |
storage:
cls: pipeline
steps:
- cls: postgresql
- db: host=postgresql-storage-rw.internal.softwareheritage.org port=5432 user=guest dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
+ db: host=postgresql-storage-rw.internal.softwareheritage.org port=5432 user=guest
+ dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
objstorage:
cls: multiplexer
objstorages:
- cls: http
compression: gzip
timeout: 120
url: https://softwareheritage.s3.amazonaws.com/content/
- cls: remote
url: http://objstorage-ro-saam-zfs-rpc-ingress
- cls: filtered
@@ -7034,21 +7047,34 @@
metadata:
namespace: swh
name: storage-postgresql-saam-zfs-configuration-template
data:
config.yml.template: |
storage:
cls: pipeline
steps:
- cls: record_references
- cls: postgresql
- db: host=postgresql-storage-rw.internal.softwareheritage.org port=5432 user=swhstorage dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
+ db: host=postgresql-storage-rw.internal.softwareheritage.org port=5432 user=swhstorage
+ dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
+ journal_writer:
+ anonymize: true
+ brokers:
+ - kafka1.internal.softwareheritage.org
+ - kafka2.internal.softwareheritage.org
+ - kafka3.internal.softwareheritage.org
+ - kafka4.internal.softwareheritage.org
+ clientId: swh.storage.journal_writer.${HOSTNAME}
+ cls: kafka
+ prefix: swh.journal.objects
+ producer_config:
+ message.max.bytes: 1000000000
objstorage:
cls: multiplexer
objstorages:
- cls: pathslicing
compression: none
root: /srv/softwareheritage/objects
slicing: 0:2/0:5
- accounts:
"0":
account_name: ${ACCOUNT_NAME_0}
@@ -7108,85 +7134,76 @@
container_name: contents
e:
account_name: ${ACCOUNT_NAME_14}
api_secret_key: ${API_SECRET_KEY_14}
container_name: contents
f:
account_name: ${ACCOUNT_NAME_15}
api_secret_key: ${API_SECRET_KEY_15}
container_name: contents
cls: azure-prefixed
-
- journal_writer:
- cls: kafka
- brokers:
- - kafka1.internal.softwareheritage.org
- - kafka2.internal.softwareheritage.org
- - kafka3.internal.softwareheritage.org
- - kafka4.internal.softwareheritage.org
- prefix: swh.journal.objects
- client_id: swh.storage.journal_writer.${HOSTNAME}
- anonymize: true
- producer_config:
- message.max.bytes: 1000000000
---
# Source: swh/templates/storage/cronjob-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: cronjob-create-object-reference-partitions-configuration-template
data:
config.yml.template: |
storage:
cls: pipeline
steps:
- cls: postgresql
- db: host=postgresql-storage-rw.internal.softwareheritage.org port=5432 user=swhstorage dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
+ db: host=postgresql-storage-rw.internal.softwareheritage.org port=5432 user=swhstorage
+ dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
---
# Source: swh/templates/toolbox/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: toolbox-indexer-storage-template
namespace: swh
data:
config.yml.template: |
indexer_storage:
cls: postgresql
- db: host=postgresql-indexer-rw.internal.softwareheritage.org port=5432 user=swhstorage dbname=softwareheritage-indexer password=${POSTGRESQL_PASSWORD}
+ db: host=postgresql-indexer-rw.internal.softwareheritage.org port=5432 user=swhstorage
+ dbname=softwareheritage-indexer password=${POSTGRESQL_PASSWORD}
---
# Source: swh/templates/toolbox/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: toolbox-scheduler-template
namespace: swh
data:
config.yml.template: |
scheduler:
cls: postgresql
- db: host=postgresql-scheduler-rw.internal.softwareheritage.org port=5432 user=swhscheduler dbname=softwareheritage-scheduler password=${POSTGRESQL_PASSWORD}
+ db: host=postgresql-scheduler-rw.internal.softwareheritage.org port=5432 user=swhscheduler
+ dbname=softwareheritage-scheduler password=${POSTGRESQL_PASSWORD}
celery:
task_broker: amqp://swhproducer:${AMQP_PASSWORD}@rabbitmq.internal.softwareheritage.org:5672/%2f
---
# Source: swh/templates/toolbox/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: toolbox-scrubber-journal-template
namespace: swh
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber
+ dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
journal:
brokers:
- kafka1.internal.softwareheritage.org:9094
- kafka2.internal.softwareheritage.org:9094
- kafka3.internal.softwareheritage.org:9094
- kafka4.internal.softwareheritage.org:9094
cls: kafka
group_id: swh-archive-prod-journalchecker
message.max.bytes: "524288000"
prefix: swh.journal.objects
@@ -7197,66 +7214,70 @@
---
# Source: swh/templates/toolbox/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: toolbox-scrubber-storage-primary-template
namespace: swh
data:
config.yml.template: |
storage:
-
- cls: postgresql
- db: host=postgresql-storage-rw.internal.softwareheritage.org port=5432 user=guest dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
+ cls: postgresql
+ db: host=postgresql-storage-rw.internal.softwareheritage.org port=5432 user=guest
+ dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
scrubber:
cls: postgresql
- db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber
+ dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
---
# Source: swh/templates/toolbox/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: toolbox-scrubber-storage-secondary-template
namespace: swh
data:
config.yml.template: |
storage:
-
- cls: postgresql
- db: host=postgresql-storage-replica.internal.softwareheritage.org port=5432 user=guest dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
+ cls: postgresql
+ db: host=postgresql-storage-replica.internal.softwareheritage.org port=5432 user=guest
+ dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
scrubber:
cls: postgresql
- db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber
+ dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
---
# Source: swh/templates/toolbox/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: toolbox-storage-template
namespace: swh
data:
config.yml.template: |
storage:
cls: postgresql
- db: host=postgresql-storage-rw.internal.softwareheritage.org port=5432 user=swhstorage dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
+ db: host=postgresql-storage-rw.internal.softwareheritage.org port=5432 user=swhstorage
+ dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
---
# Source: swh/templates/toolbox/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: toolbox-storage-secondary-template
namespace: swh
data:
config.yml.template: |
storage:
cls: postgresql
- db: host=postgresql-storage-replica.internal.softwareheritage.org port=5432 user=swhstorage dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
+ db: host=postgresql-storage-replica.internal.softwareheritage.org port=5432 user=swhstorage
+ dbname=softwareheritage password=${POSTGRESQL_PASSWORD}
---
# Source: swh/templates/toolbox/script-utils-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: toolbox-script-utils
namespace: swh
data:
register-task-types.sh: |
#!/bin/bash
@@ -7634,21 +7655,22 @@
# Source: swh/templates/vault/rpc-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh
name: vault-rpc-configuration-template
data:
config.yml.template: |
vault:
cls: postgresql
- db: host=postgresql-vault-rw.internal.softwareheritage.org port=5432 user=swh-vault dbname=swh-vault password=${POSTGRESQL_PASSWORD}
+ db: host=postgresql-vault-rw.internal.softwareheritage.org port=5432 user=swh-vault
+ dbname=swh-vault password=${POSTGRESQL_PASSWORD}
storage:
cls: pipeline
steps:
- cls: retry
- cls: remote
url: http://storage-azure-read-only-rpc-ingress
scheduler:
cls: remote
url: http://scheduler.internal.softwareheritage.org
objstorage:
@@ -10039,21 +10061,21 @@
app: indexer-storage-read-only
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: indexer-storage-read-only
annotations:
- checksum/config: dd6f822f2061639286c3a4405e71ee570528b8068871a0b2f8873721a7dd2a3d
+ checksum/config: ea070fbf964af6768231f7af3c7c7bc5f0766ba5a9ea4ad2af0c547c102b8301
checksum/database-utils: 931b7f92b50eae0ecb6203c3e0e48f7c363a99ac0064451187dc12c6eaf0a793
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/rpc
operator: In
values:
@@ -10187,21 +10209,21 @@
app: indexer-storage-read-write
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: indexer-storage-read-write
annotations:
- checksum/config: e12584e748caaa07b7ba5e521cdfd26f50cbbaa08573c63d1cc8901fc6689a98
+ checksum/config: d9df641d5e27737c4dbc8f73df98bd87a33da54e708d9b03d2ac627825b369f1
checksum/database-utils: 931b7f92b50eae0ecb6203c3e0e48f7c363a99ac0064451187dc12c6eaf0a793
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/rpc
operator: In
values:
@@ -21197,21 +21219,21 @@
app: scheduler-rpc
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scheduler-rpc
annotations:
- checksum/config: d83adbbd0a90fe39e2a7773833bfb7913f3548df3236539319994a3d734640d6
+ checksum/config: 2a05194a8beda075c834220f944ee6d93b21035b1e5852c7a9b08f955460f909
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scheduler
operator: In
values:
- "true"
@@ -21324,21 +21346,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scrubber-journalchecker-release
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: cd0a1cee59b9f8bc72fb93a2d4c607377e88751cb4e85abb0b47a2f592051711
+ checksum/config: 66796b946f73e43abaaabf416fd8ad1aeb6dc610f2efeb838e99edb8c65b9039
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scrubber
operator: In
values:
@@ -21472,21 +21494,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scrubber-journalchecker-revision
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: 84513e943956d5146668540e38c8e6b673e84bb969e8a9536dd2d21aece1071d
+ checksum/config: 342ace2aac070afa370be22bcea9c0c59ba699ee97b39026d7666a5af81ae190
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scrubber
operator: In
values:
@@ -21620,21 +21642,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scrubber-journalchecker-snapshot
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: cd4d24b648de17139c423cc02ae29aa100a22bdfd6a7a26fdca91183bb06e82d
+ checksum/config: c6a43b504cc727c0ce7a00ea795f733e1b51b4a64f6961a655b1c64140e5ac4c
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scrubber
operator: In
values:
@@ -21768,21 +21790,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scrubber-storagechecker-primary-directory-hashes
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: c126d6a89c619233fc8f87bfd955eedf4d292c7d71835d211383a274a989288d
+ checksum/config: 0d8e1c78891d2697ecdb562f0317978f37a3371de9e1edd4c04a50ae0a199a9e
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scrubber
operator: In
values:
@@ -21921,21 +21943,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scrubber-storagechecker-primary-directory-references
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: 6f0007c03907ff90d0756a8dc8fddfa7da2da9e6d69c4c010aebd081d7b3f25b
+ checksum/config: 5675e574af3a9aeedd0ad1170d930f5595a5691a0db1b9609d78df4d65227847
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scrubber
operator: In
values:
@@ -22074,21 +22096,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scrubber-storagechecker-secondary-directory-hashes
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: 0c324fec0473ad16eafa86c2e23de077b8e071bfd625d0a659493e21ddb9552b
+ checksum/config: 9d67c9591cf27e153fa0df08bbcb2b680d0c4e87866ce24bb6dce3e942c12b54
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scrubber
operator: In
values:
@@ -22227,21 +22249,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scrubber-storagechecker-secondary-directory-references
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: 5ae33eb49a8ae57ca0640ccb76991e6ece23bf6c3583c2067eb8cb868e6166ab
+ checksum/config: fde034a5742a50c92b65c1037d7405120820bdbe8b9423cfad846cba0c8732be
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scrubber
operator: In
values:
@@ -22380,21 +22402,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scrubber-storagechecker-secondary-snapshot-hashes
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: 496c0c477dd7fff2410a21d2cee031c5ef9649c875497aaf2a004e5a8560f06e
+ checksum/config: 7ee8e7600f4903f7074c5d6fc37945071a806254930318ffe6adbe16e83722b1
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scrubber
operator: In
values:
@@ -22533,21 +22555,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scrubber-storagechecker-secondary-snapshot-references
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: 51f6d84628202678308237991c87aab0cf6d822271f2e76325a46c59720c7816
+ checksum/config: 3f702f6c6555b8496847d61959efc42d7d96260fa47e81a66b57be8453483526
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scrubber
operator: In
values:
@@ -23045,21 +23067,21 @@
app: storage-postgresql-azure-readonly
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-postgresql-azure-readonly
annotations:
- checksum/config: 0df08fe07b48ebfcfd5668efeb13e0dc3c7abb0805f26e9a539a5b881829ea55
+ checksum/config: 01f435377fa75cddf09f230de070347c446a4a41c02390cda60f3b9419aa5b64
checksum/database-utils: 931b7f92b50eae0ecb6203c3e0e48f7c363a99ac0064451187dc12c6eaf0a793
checksum/config-utils: d75ca13b805bce6a8ab59c8e24c938f2283108f6a79134f6e71db86308651dc6
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/storage
operator: In
@@ -23407,21 +23429,21 @@
app: storage-postgresql-saam-zfs
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-postgresql-saam-zfs
annotations:
- checksum/config: 8011d082d6e7a1c2dbb9e7a4cdd9dc38012d002ed8792f32d189c472e8890b07
+ checksum/config: 3acccc25a5b75b667e3b95ab9a30a78a134a112c078d4ebd966a9a3a8da7e5fb
checksum/database-utils: 931b7f92b50eae0ecb6203c3e0e48f7c363a99ac0064451187dc12c6eaf0a793
checksum/config-utils: d75ca13b805bce6a8ab59c8e24c938f2283108f6a79134f6e71db86308651dc6
spec:
nodeSelector:
kubernetes.io/hostname: saam
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
@@ -23789,21 +23811,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: swh-toolbox
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: 7f5c16452ec35bd838368c66e83ace493b8d65905ae1c1a8df7679703a5f7428
+ checksum/config: 1d035bc274174eaa8d4ff914fab3dddc39f418701233002fa71227ebfab63ecb
checksum/configScript: 617a3bf1b7cc114666253d98def7fe1fb764db4c48d5398e09cc7f506c5f4cf6
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/toolbox
operator: In
values:
@@ -24158,21 +24180,21 @@
app: vault-rpc
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: vault-rpc
annotations:
- checksum/config: b174b0fb0962f9f4cef470cfb61fae496409553672228e0049c9e8563a52a566
+ checksum/config: 32f8985609f65524bf136ef5ade40c5fec3a64f5472c070b1705ca13f32c70ab
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/rpc
operator: In
values:
- "true"
------------- diff for environment production namespace swh-cassandra -------------
--- /tmp/swh-chart.swh.qTNYADrG/production-swh-cassandra.before 2024-03-29 16:37:09.916581501 +0100
+++ /tmp/swh-chart.swh.qTNYADrG/production-swh-cassandra.after 2024-03-29 16:37:10.284588622 +0100
@@ -441,71 +441,73 @@
# Source: swh/templates/scrubber/storage-checker-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: scrubber-storagechecker-directory-hashes-template
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber
+ dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
storage:
- cls: cassandra
- hosts:
- - cassandra01.internal.softwareheritage.org
- - cassandra02.internal.softwareheritage.org
- - cassandra03.internal.softwareheritage.org
- - cassandra04.internal.softwareheritage.org
- - cassandra05.internal.softwareheritage.org
- - cassandra06.internal.softwareheritage.org
- - cassandra07.internal.softwareheritage.org
- - cassandra08.internal.softwareheritage.org
- - cassandra09.internal.softwareheritage.org
- - cassandra10.internal.softwareheritage.org
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-ro
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
+ hosts:
+ - cassandra01.internal.softwareheritage.org
+ - cassandra02.internal.softwareheritage.org
+ - cassandra03.internal.softwareheritage.org
+ - cassandra04.internal.softwareheritage.org
+ - cassandra05.internal.softwareheritage.org
+ - cassandra06.internal.softwareheritage.org
+ - cassandra07.internal.softwareheritage.org
+ - cassandra08.internal.softwareheritage.org
+ - cassandra09.internal.softwareheritage.org
+ - cassandra10.internal.softwareheritage.org
+ keyspace: swh
---
# Source: swh/templates/scrubber/storage-checker-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: scrubber-storagechecker-directory-references-template
data:
config.yml.template: |
scrubber:
cls: postgresql
- db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber
+ dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
storage:
- cls: cassandra
- hosts:
- - cassandra01.internal.softwareheritage.org
- - cassandra02.internal.softwareheritage.org
- - cassandra03.internal.softwareheritage.org
- - cassandra04.internal.softwareheritage.org
- - cassandra05.internal.softwareheritage.org
- - cassandra06.internal.softwareheritage.org
- - cassandra07.internal.softwareheritage.org
- - cassandra08.internal.softwareheritage.org
- - cassandra09.internal.softwareheritage.org
- - cassandra10.internal.softwareheritage.org
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-ro
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
+ hosts:
+ - cassandra01.internal.softwareheritage.org
+ - cassandra02.internal.softwareheritage.org
+ - cassandra03.internal.softwareheritage.org
+ - cassandra04.internal.softwareheritage.org
+ - cassandra05.internal.softwareheritage.org
+ - cassandra06.internal.softwareheritage.org
+ - cassandra07.internal.softwareheritage.org
+ - cassandra08.internal.softwareheritage.org
+ - cassandra09.internal.softwareheritage.org
+ - cassandra10.internal.softwareheritage.org
+ keyspace: swh
---
# Source: swh/templates/statsd-exporter/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-statsd-exporter
namespace: swh-cassandra
data:
config.yml: |
defaults:
@@ -566,39 +568,39 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-content-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra01.internal.softwareheritage.org
- - cassandra02.internal.softwareheritage.org
- - cassandra03.internal.softwareheritage.org
- - cassandra04.internal.softwareheritage.org
- - cassandra05.internal.softwareheritage.org
- - cassandra06.internal.softwareheritage.org
- - cassandra07.internal.softwareheritage.org
- - cassandra08.internal.softwareheritage.org
- - cassandra09.internal.softwareheritage.org
- - cassandra10.internal.softwareheritage.org
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra01.internal.softwareheritage.org
+ - cassandra02.internal.softwareheritage.org
+ - cassandra03.internal.softwareheritage.org
+ - cassandra04.internal.softwareheritage.org
+ - cassandra05.internal.softwareheritage.org
+ - cassandra06.internal.softwareheritage.org
+ - cassandra07.internal.softwareheritage.org
+ - cassandra08.internal.softwareheritage.org
+ - cassandra09.internal.softwareheritage.org
+ - cassandra10.internal.softwareheritage.org
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- kafka1.internal.softwareheritage.org:9094
- kafka2.internal.softwareheritage.org:9094
- kafka3.internal.softwareheritage.org:9094
- kafka4.internal.softwareheritage.org:9094
@@ -621,39 +623,39 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-extid-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra01.internal.softwareheritage.org
- - cassandra02.internal.softwareheritage.org
- - cassandra03.internal.softwareheritage.org
- - cassandra04.internal.softwareheritage.org
- - cassandra05.internal.softwareheritage.org
- - cassandra06.internal.softwareheritage.org
- - cassandra07.internal.softwareheritage.org
- - cassandra08.internal.softwareheritage.org
- - cassandra09.internal.softwareheritage.org
- - cassandra10.internal.softwareheritage.org
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra01.internal.softwareheritage.org
+ - cassandra02.internal.softwareheritage.org
+ - cassandra03.internal.softwareheritage.org
+ - cassandra04.internal.softwareheritage.org
+ - cassandra05.internal.softwareheritage.org
+ - cassandra06.internal.softwareheritage.org
+ - cassandra07.internal.softwareheritage.org
+ - cassandra08.internal.softwareheritage.org
+ - cassandra09.internal.softwareheritage.org
+ - cassandra10.internal.softwareheritage.org
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- kafka1.internal.softwareheritage.org:9094
- kafka2.internal.softwareheritage.org:9094
- kafka3.internal.softwareheritage.org:9094
- kafka4.internal.softwareheritage.org:9094
@@ -676,39 +678,39 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-metadata-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra01.internal.softwareheritage.org
- - cassandra02.internal.softwareheritage.org
- - cassandra03.internal.softwareheritage.org
- - cassandra04.internal.softwareheritage.org
- - cassandra05.internal.softwareheritage.org
- - cassandra06.internal.softwareheritage.org
- - cassandra07.internal.softwareheritage.org
- - cassandra08.internal.softwareheritage.org
- - cassandra09.internal.softwareheritage.org
- - cassandra10.internal.softwareheritage.org
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra01.internal.softwareheritage.org
+ - cassandra02.internal.softwareheritage.org
+ - cassandra03.internal.softwareheritage.org
+ - cassandra04.internal.softwareheritage.org
+ - cassandra05.internal.softwareheritage.org
+ - cassandra06.internal.softwareheritage.org
+ - cassandra07.internal.softwareheritage.org
+ - cassandra08.internal.softwareheritage.org
+ - cassandra09.internal.softwareheritage.org
+ - cassandra10.internal.softwareheritage.org
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- kafka1.internal.softwareheritage.org:9094
- kafka2.internal.softwareheritage.org:9094
- kafka3.internal.softwareheritage.org:9094
- kafka4.internal.softwareheritage.org:9094
@@ -732,39 +734,39 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-origin-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra01.internal.softwareheritage.org
- - cassandra02.internal.softwareheritage.org
- - cassandra03.internal.softwareheritage.org
- - cassandra04.internal.softwareheritage.org
- - cassandra05.internal.softwareheritage.org
- - cassandra06.internal.softwareheritage.org
- - cassandra07.internal.softwareheritage.org
- - cassandra08.internal.softwareheritage.org
- - cassandra09.internal.softwareheritage.org
- - cassandra10.internal.softwareheritage.org
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra01.internal.softwareheritage.org
+ - cassandra02.internal.softwareheritage.org
+ - cassandra03.internal.softwareheritage.org
+ - cassandra04.internal.softwareheritage.org
+ - cassandra05.internal.softwareheritage.org
+ - cassandra06.internal.softwareheritage.org
+ - cassandra07.internal.softwareheritage.org
+ - cassandra08.internal.softwareheritage.org
+ - cassandra09.internal.softwareheritage.org
+ - cassandra10.internal.softwareheritage.org
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- kafka1.internal.softwareheritage.org:9094
- kafka2.internal.softwareheritage.org:9094
- kafka3.internal.softwareheritage.org:9094
- kafka4.internal.softwareheritage.org:9094
@@ -787,39 +789,39 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-origin-visit-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra01.internal.softwareheritage.org
- - cassandra02.internal.softwareheritage.org
- - cassandra03.internal.softwareheritage.org
- - cassandra04.internal.softwareheritage.org
- - cassandra05.internal.softwareheritage.org
- - cassandra06.internal.softwareheritage.org
- - cassandra07.internal.softwareheritage.org
- - cassandra08.internal.softwareheritage.org
- - cassandra09.internal.softwareheritage.org
- - cassandra10.internal.softwareheritage.org
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra01.internal.softwareheritage.org
+ - cassandra02.internal.softwareheritage.org
+ - cassandra03.internal.softwareheritage.org
+ - cassandra04.internal.softwareheritage.org
+ - cassandra05.internal.softwareheritage.org
+ - cassandra06.internal.softwareheritage.org
+ - cassandra07.internal.softwareheritage.org
+ - cassandra08.internal.softwareheritage.org
+ - cassandra09.internal.softwareheritage.org
+ - cassandra10.internal.softwareheritage.org
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- kafka1.internal.softwareheritage.org:9094
- kafka2.internal.softwareheritage.org:9094
- kafka3.internal.softwareheritage.org:9094
- kafka4.internal.softwareheritage.org:9094
@@ -842,39 +844,39 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-origin-visit-status-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra01.internal.softwareheritage.org
- - cassandra02.internal.softwareheritage.org
- - cassandra03.internal.softwareheritage.org
- - cassandra04.internal.softwareheritage.org
- - cassandra05.internal.softwareheritage.org
- - cassandra06.internal.softwareheritage.org
- - cassandra07.internal.softwareheritage.org
- - cassandra08.internal.softwareheritage.org
- - cassandra09.internal.softwareheritage.org
- - cassandra10.internal.softwareheritage.org
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra01.internal.softwareheritage.org
+ - cassandra02.internal.softwareheritage.org
+ - cassandra03.internal.softwareheritage.org
+ - cassandra04.internal.softwareheritage.org
+ - cassandra05.internal.softwareheritage.org
+ - cassandra06.internal.softwareheritage.org
+ - cassandra07.internal.softwareheritage.org
+ - cassandra08.internal.softwareheritage.org
+ - cassandra09.internal.softwareheritage.org
+ - cassandra10.internal.softwareheritage.org
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- kafka1.internal.softwareheritage.org:9094
- kafka2.internal.softwareheritage.org:9094
- kafka3.internal.softwareheritage.org:9094
- kafka4.internal.softwareheritage.org:9094
@@ -897,39 +899,39 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-raw-extrinsic-metadata-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra01.internal.softwareheritage.org
- - cassandra02.internal.softwareheritage.org
- - cassandra03.internal.softwareheritage.org
- - cassandra04.internal.softwareheritage.org
- - cassandra05.internal.softwareheritage.org
- - cassandra06.internal.softwareheritage.org
- - cassandra07.internal.softwareheritage.org
- - cassandra08.internal.softwareheritage.org
- - cassandra09.internal.softwareheritage.org
- - cassandra10.internal.softwareheritage.org
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra01.internal.softwareheritage.org
+ - cassandra02.internal.softwareheritage.org
+ - cassandra03.internal.softwareheritage.org
+ - cassandra04.internal.softwareheritage.org
+ - cassandra05.internal.softwareheritage.org
+ - cassandra06.internal.softwareheritage.org
+ - cassandra07.internal.softwareheritage.org
+ - cassandra08.internal.softwareheritage.org
+ - cassandra09.internal.softwareheritage.org
+ - cassandra10.internal.softwareheritage.org
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- kafka1.internal.softwareheritage.org:9094
- kafka2.internal.softwareheritage.org:9094
- kafka3.internal.softwareheritage.org:9094
- kafka4.internal.softwareheritage.org:9094
@@ -952,39 +954,39 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-release-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra01.internal.softwareheritage.org
- - cassandra02.internal.softwareheritage.org
- - cassandra03.internal.softwareheritage.org
- - cassandra04.internal.softwareheritage.org
- - cassandra05.internal.softwareheritage.org
- - cassandra06.internal.softwareheritage.org
- - cassandra07.internal.softwareheritage.org
- - cassandra08.internal.softwareheritage.org
- - cassandra09.internal.softwareheritage.org
- - cassandra10.internal.softwareheritage.org
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra01.internal.softwareheritage.org
+ - cassandra02.internal.softwareheritage.org
+ - cassandra03.internal.softwareheritage.org
+ - cassandra04.internal.softwareheritage.org
+ - cassandra05.internal.softwareheritage.org
+ - cassandra06.internal.softwareheritage.org
+ - cassandra07.internal.softwareheritage.org
+ - cassandra08.internal.softwareheritage.org
+ - cassandra09.internal.softwareheritage.org
+ - cassandra10.internal.softwareheritage.org
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- kafka1.internal.softwareheritage.org:9094
- kafka2.internal.softwareheritage.org:9094
- kafka3.internal.softwareheritage.org:9094
- kafka4.internal.softwareheritage.org:9094
@@ -1007,39 +1009,39 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-revision-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra01.internal.softwareheritage.org
- - cassandra02.internal.softwareheritage.org
- - cassandra03.internal.softwareheritage.org
- - cassandra04.internal.softwareheritage.org
- - cassandra05.internal.softwareheritage.org
- - cassandra06.internal.softwareheritage.org
- - cassandra07.internal.softwareheritage.org
- - cassandra08.internal.softwareheritage.org
- - cassandra09.internal.softwareheritage.org
- - cassandra10.internal.softwareheritage.org
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra01.internal.softwareheritage.org
+ - cassandra02.internal.softwareheritage.org
+ - cassandra03.internal.softwareheritage.org
+ - cassandra04.internal.softwareheritage.org
+ - cassandra05.internal.softwareheritage.org
+ - cassandra06.internal.softwareheritage.org
+ - cassandra07.internal.softwareheritage.org
+ - cassandra08.internal.softwareheritage.org
+ - cassandra09.internal.softwareheritage.org
+ - cassandra10.internal.softwareheritage.org
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- kafka1.internal.softwareheritage.org:9094
- kafka2.internal.softwareheritage.org:9094
- kafka3.internal.softwareheritage.org:9094
- kafka4.internal.softwareheritage.org:9094
@@ -1062,39 +1064,39 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-skipped-content-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra01.internal.softwareheritage.org
- - cassandra02.internal.softwareheritage.org
- - cassandra03.internal.softwareheritage.org
- - cassandra04.internal.softwareheritage.org
- - cassandra05.internal.softwareheritage.org
- - cassandra06.internal.softwareheritage.org
- - cassandra07.internal.softwareheritage.org
- - cassandra08.internal.softwareheritage.org
- - cassandra09.internal.softwareheritage.org
- - cassandra10.internal.softwareheritage.org
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra01.internal.softwareheritage.org
+ - cassandra02.internal.softwareheritage.org
+ - cassandra03.internal.softwareheritage.org
+ - cassandra04.internal.softwareheritage.org
+ - cassandra05.internal.softwareheritage.org
+ - cassandra06.internal.softwareheritage.org
+ - cassandra07.internal.softwareheritage.org
+ - cassandra08.internal.softwareheritage.org
+ - cassandra09.internal.softwareheritage.org
+ - cassandra10.internal.softwareheritage.org
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- kafka1.internal.softwareheritage.org:9094
- kafka2.internal.softwareheritage.org:9094
- kafka3.internal.softwareheritage.org:9094
- kafka4.internal.softwareheritage.org:9094
@@ -1117,39 +1119,39 @@
---
# Source: swh/templates/storage-replayer/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-replayer-configuration-snapshot-template
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra01.internal.softwareheritage.org
- - cassandra02.internal.softwareheritage.org
- - cassandra03.internal.softwareheritage.org
- - cassandra04.internal.softwareheritage.org
- - cassandra05.internal.softwareheritage.org
- - cassandra06.internal.softwareheritage.org
- - cassandra07.internal.softwareheritage.org
- - cassandra08.internal.softwareheritage.org
- - cassandra09.internal.softwareheritage.org
- - cassandra10.internal.softwareheritage.org
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-rw
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
directory_entries_insert_algo: batch
+ hosts:
+ - cassandra01.internal.softwareheritage.org
+ - cassandra02.internal.softwareheritage.org
+ - cassandra03.internal.softwareheritage.org
+ - cassandra04.internal.softwareheritage.org
+ - cassandra05.internal.softwareheritage.org
+ - cassandra06.internal.softwareheritage.org
+ - cassandra07.internal.softwareheritage.org
+ - cassandra08.internal.softwareheritage.org
+ - cassandra09.internal.softwareheritage.org
+ - cassandra10.internal.softwareheritage.org
+ keyspace: swh
objstorage:
cls: noop
journal_client:
cls: kafka
brokers:
- kafka1.internal.softwareheritage.org:9094
- kafka2.internal.softwareheritage.org:9094
- kafka3.internal.softwareheritage.org:9094
- kafka4.internal.softwareheritage.org:9094
@@ -1175,39 +1177,38 @@
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-cassandra-readonly-configuration-template
data:
config.yml.template: |
storage:
cls: pipeline
steps:
- cls: retry
- - cls: cassandra
+ - auth_provider:
+ cls: cassandra.auth.PlainTextAuthProvider
+ password: ${CASSANDRA_PASSWORD}
+ username: swh-ro
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
hosts:
- cassandra01.internal.softwareheritage.org
- cassandra02.internal.softwareheritage.org
- cassandra03.internal.softwareheritage.org
- cassandra04.internal.softwareheritage.org
- cassandra05.internal.softwareheritage.org
- cassandra06.internal.softwareheritage.org
- cassandra07.internal.softwareheritage.org
- cassandra08.internal.softwareheritage.org
- cassandra09.internal.softwareheritage.org
- cassandra10.internal.softwareheritage.org
keyspace: swh
- consistency_level: LOCAL_QUORUM
- auth_provider:
- cls: cassandra.auth.PlainTextAuthProvider
- password: ${CASSANDRA_PASSWORD}
- username: swh-ro
-
objstorage:
cls: multiplexer
objstorages:
- cls: http
compression: gzip
timeout: 120
url: https://softwareheritage.s3.amazonaws.com/content/
- cls: remote
url: http://objstorage-ro-saam-zfs-rpc-ingress
- cls: filtered
@@ -1292,39 +1293,38 @@
kind: ConfigMap
metadata:
namespace: swh-cassandra
name: storage-cassandra-readonly-internal-configuration-template
data:
config.yml.template: |
storage:
cls: pipeline
steps:
- cls: retry
- - cls: cassandra
+ - auth_provider:
+ cls: cassandra.auth.PlainTextAuthProvider
+ password: ${CASSANDRA_PASSWORD}
+ username: swh-ro
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
hosts:
- cassandra01.internal.softwareheritage.org
- cassandra02.internal.softwareheritage.org
- cassandra03.internal.softwareheritage.org
- cassandra04.internal.softwareheritage.org
- cassandra05.internal.softwareheritage.org
- cassandra06.internal.softwareheritage.org
- cassandra07.internal.softwareheritage.org
- cassandra08.internal.softwareheritage.org
- cassandra09.internal.softwareheritage.org
- cassandra10.internal.softwareheritage.org
keyspace: swh
- consistency_level: LOCAL_QUORUM
- auth_provider:
- cls: cassandra.auth.PlainTextAuthProvider
- password: ${CASSANDRA_PASSWORD}
- username: swh-ro
-
objstorage:
cls: multiplexer
objstorages:
- cls: http
compression: gzip
timeout: 120
url: https://softwareheritage.s3.amazonaws.com/content/
- cls: remote
url: http://objstorage-ro-saam-zfs-rpc-ingress
- cls: filtered
@@ -1406,42 +1406,42 @@
---
# Source: swh/templates/toolbox/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: toolbox-scrubber-storage-template
namespace: swh-cassandra
data:
config.yml.template: |
storage:
- cls: cassandra
- hosts:
- - cassandra01.internal.softwareheritage.org
- - cassandra02.internal.softwareheritage.org
- - cassandra03.internal.softwareheritage.org
- - cassandra04.internal.softwareheritage.org
- - cassandra05.internal.softwareheritage.org
- - cassandra06.internal.softwareheritage.org
- - cassandra07.internal.softwareheritage.org
- - cassandra08.internal.softwareheritage.org
- - cassandra09.internal.softwareheritage.org
- - cassandra10.internal.softwareheritage.org
- keyspace: swh
- consistency_level: LOCAL_QUORUM
auth_provider:
cls: cassandra.auth.PlainTextAuthProvider
password: ${CASSANDRA_PASSWORD}
username: swh-ro
-
+ cls: cassandra
+ consistency_level: LOCAL_QUORUM
+ hosts:
+ - cassandra01.internal.softwareheritage.org
+ - cassandra02.internal.softwareheritage.org
+ - cassandra03.internal.softwareheritage.org
+ - cassandra04.internal.softwareheritage.org
+ - cassandra05.internal.softwareheritage.org
+ - cassandra06.internal.softwareheritage.org
+ - cassandra07.internal.softwareheritage.org
+ - cassandra08.internal.softwareheritage.org
+ - cassandra09.internal.softwareheritage.org
+ - cassandra10.internal.softwareheritage.org
+ keyspace: swh
scrubber:
cls: postgresql
- db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
+ db: host=postgresql-scrubber-rw.internal.softwareheritage.org port=5432 user=swh-scrubber
+ dbname=swh-scrubber password=${SCRUBBER_POSTGRESQL_PASSWORD}
---
# Source: swh/templates/toolbox/script-utils-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: toolbox-script-utils
namespace: swh-cassandra
data:
register-task-types.sh: |
#!/bin/bash
@@ -3220,21 +3220,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scrubber-storagechecker-directory-hashes
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: 6cfe4d2f19cbddc54651c2affd8e8bdeab1232cd01581609a604fce54ffa46ed
+ checksum/config: 38708b457599700ce41e6a719744198bf0acdbad88e05292d7e7d404a125fd87
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scrubber
operator: In
values:
@@ -3373,21 +3373,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: scrubber-storagechecker-directory-references
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: f5a7865a1d7bea72762c1d8def72b0c451b2df3c6cf9a0e742de9e0cd9a85502
+ checksum/config: 7f4d111d203912f0bfed538e8bd4cffc1be864f59e3b283507607a8f0e8aa711
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/scrubber
operator: In
values:
@@ -3562,21 +3562,21 @@
app: storage-replayer-content
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-content
annotations:
- checksum/config: 982bf6ab2629e85309f349a923803f5382b4c05be095a14233eafd9839cb79e3
+ checksum/config: cdb30362f2e6619de7af7411b812644759dae47896085efcae39ab08d1a74bd1
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -3677,21 +3677,21 @@
app: storage-replayer-extid
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-extid
annotations:
- checksum/config: 6ad25ad0f54ca8e77c63186a685f59b41b80780669512a8336acb1dd7a90df84
+ checksum/config: 48bf68b1e842996fc0f1dd39565354863572190716809d109c745663b51c9014
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -3792,21 +3792,21 @@
app: storage-replayer-metadata
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-metadata
annotations:
- checksum/config: 7198c6b7943856fbc5a1296fb94fbd696c068d6cdee8f829194dbf476c656f8d
+ checksum/config: 58e85924f4702d1e0d94ef447f49529edd518c4d05d7a7afec8f02114165bcd7
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -3907,21 +3907,21 @@
app: storage-replayer-origin
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-origin
annotations:
- checksum/config: 758a60730a13e61590b7b9bc7c99bd9498e1e68afe42a6696b1d3736d1b84083
+ checksum/config: 45e92c8dd52b5a3df5eea028c96e771cd30f3eaa0bb2397fc38d341d7e7a4202
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -4022,21 +4022,21 @@
app: storage-replayer-origin-visit
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-origin-visit
annotations:
- checksum/config: 281301d748200ea812a8135a1d93a8c13b71d250b7bb6dc0dab40b605516937b
+ checksum/config: 403e92088f4f42ea9cd4370840b5896d219f1325e1051caf93590fca84952365
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -4137,21 +4137,21 @@
app: storage-replayer-origin-visit-status
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-origin-visit-status
annotations:
- checksum/config: a4c0f2fa314b76a158a475ddd2133f0d899d486f225b3739b6dabde2dbac79a8
+ checksum/config: 95c32618ce1f6eac1687070254ff9d822c2053589ac4c30f99d571bd96456804
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -4252,21 +4252,21 @@
app: storage-replayer-raw-extrinsic-metadata
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-raw-extrinsic-metadata
annotations:
- checksum/config: 99fec5e27d8458a4a33a263eecfd30e9705a7333db13628ebe62d86d891662fc
+ checksum/config: 292fded243de82312509c49b0a285090662edc8cb4723d467a7951a77ba87c73
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -4367,21 +4367,21 @@
app: storage-replayer-release
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-release
annotations:
- checksum/config: 5c4292f79bcd72d378f7690c72de8cc6043fd6720d3f45bf4d511e0235feed45
+ checksum/config: 8faebf68549b5834755d4af7a6619e6595e03aff61eab6c0dabebae295816dac
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -4482,21 +4482,21 @@
app: storage-replayer-revision
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-revision
annotations:
- checksum/config: f5c7f904e21bc14e4947d7d0d1eac4c6cd6bbf7579e88561657537b0eee2bc11
+ checksum/config: e915cfe2f0b4772ffdc47e0c933c02ebcb3933e1106f6ad2acf4ad8862eecda0
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -4597,21 +4597,21 @@
app: storage-replayer-skipped-content
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-skipped-content
annotations:
- checksum/config: ba59bb68a01cb1873eda5d1f18cde9283919be9d0821c9d7a69d990bb43a3d1b
+ checksum/config: 4eb7f7197c4b9529dc12d8533feb5d0bbc878b16c7cee35062506c072e93c221
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -4712,21 +4712,21 @@
app: storage-replayer-snapshot
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-replayer-snapshot
annotations:
- checksum/config: cf842d561ae79f7cfb8826a83843a1923ccd07136b9cd4be13cad38169c0a33d
+ checksum/config: 94f2ab304922804ad361d2afb2b8a447dff2926d5a0f055c4edfc6638a21bdad
checksum/config_utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/replayer
operator: In
values:
@@ -4827,21 +4827,21 @@
app: storage-cassandra-readonly
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-cassandra-readonly
annotations:
- checksum/config: a462042990677f75734be67b3ae604b38c439ce883d4c0afc394417a4aa2da59
+ checksum/config: 075016b72b56c3e6d42486c63683e2bc8c43eabb10ed6f3a379677e9cf61cded
checksum/database-utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
checksum/config-utils: 13a26f6add17e96ce01550153c77dcd48de60241a3f4db3c93d5467234be2a7f
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/storage
operator: In
@@ -5183,21 +5183,21 @@
app: storage-cassandra-readonly-internal
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: storage-cassandra-readonly-internal
annotations:
- checksum/config: 90453719fecf0db009df97c6ae33273fbe133d3e8266763c31b70f7227805a25
+ checksum/config: f927652c28d189cb3f336457645e5477f3b23a87f546a7a8deb100967c16352b
checksum/database-utils: 367f28d8854b3b188d7a45015752b10be8e691f0c3b2a21db02501e655b932f0
checksum/config-utils: 13a26f6add17e96ce01550153c77dcd48de60241a3f4db3c93d5467234be2a7f
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/storage
operator: In
@@ -5540,21 +5540,21 @@
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: swh-toolbox
annotations:
# Force a rollout upgrade if the configuration changes
- checksum/config: 88d71f4d0110718e59c2c354b59e119df7d2fbd8d5169bdc9217c880083c99cd
+ checksum/config: e10bab347d0c53b041c8e104700dda7139a8048829e5a46f74a912f314416f48
checksum/configScript: ef8d5ad9dd2b20fc89e96d4042a4354752947bd26cec17d125fc7e8d648fdeea
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: swh/toolbox
operator: In
values:
Edited by Nicolas Dandrimont
Merge request reports
Activity
assigned to @olasd
added 4 commits
-
b263c0e2...1f6cdd44 - 3 commits from branch
production - 17b9b945 - Rework the storage configuration template helpers to be yaml-based
-
b263c0e2...1f6cdd44 - 3 commits from branch
IINM, this part of the diff looks off in terms of indentation. The rest looks ok though.
- cls: cassandra - hosts: - - cassandra01.internal.softwareheritage.org - - cassandra02.internal.softwareheritage.org - - cassandra03.internal.softwareheritage.org - - cassandra04.internal.softwareheritage.org - - cassandra05.internal.softwareheritage.org - - cassandra06.internal.softwareheritage.org - - cassandra07.internal.softwareheritage.org - - cassandra08.internal.softwareheritage.org - - cassandra09.internal.softwareheritage.org - - cassandra10.internal.softwareheritage.org - keyspace: swh - consistency_level: LOCAL_QUORUM auth_provider: cls: cassandra.auth.PlainTextAuthProvider password: ${CASSANDRA_PASSWORD} username: swh-rw + cls: cassandra + consistency_level: LOCAL_QUORUM directory_entries_insert_algo: batch + hosts: + - cassandra01.internal.softwareheritage.org + - cassandra02.internal.softwareheritage.org + - cassandra03.internal.softwareheritage.org + - cassandra04.internal.softwareheritage.org + - cassandra05.internal.softwareheritage.org + - cassandra06.internal.softwareheritage.org + - cassandra07.internal.softwareheritage.org + - cassandra08.internal.softwareheritage.org + - cassandra09.internal.softwareheritage.org + - cassandra10.internal.softwareheritage.org + keyspace: swh
Final comment: yaml allows splitting strings indented over multiple lines (that's what happens with the postgresql connection strings), it just concatenates them back into a space-separated string
In [6]: document = """\ ...: --- ...: a: long string ...: broken up into two lines ...: """ In [7]: yaml.safe_load(document) Out[7]: {'a': 'long string broken up into two lines'}-
it's right there in the diff:
@@ -894,33 +898,32 @@ namespace: swh name: storage-postgresql-read-write-configuration-template data: config.yml.template: | storage: cls: pipeline steps: - cls: record_references - cls: postgresql db: host=db1.internal.staging.swh.network port=5432 user=swh dbname=swh password=${POSTGRESQL_PASSWORD} - objstorage: - cls: remote - url: http://objstorage-multiplexer-read-write-rpc-ingress - journal_writer: - cls: kafka + anonymize: true brokers: - journal2.internal.staging.swh.network + clientId: swh.storage.journal_writer.${HOSTNAME} + cls: kafka prefix: swh.journal.objects - client_id: swh.storage.journal_writer.${HOSTNAME} - anonymize: true producer_config: message.max.bytes: 1000000000 + objstorage: + cls: remote + url: http://objstorage-multiplexer-read-write-rpc-ingress
Please register or sign in to reply