[cassandra] Configure staging cassandra and populate backend

Cassandra service is basically configured in puppet. It will need more tuning.

Problematic: Test Cassandra as main backend in staging. We need in parallel a postgresql backend which needs up-to-date with the cassandra backend.

staging:

• Install data model in cassandra (create keyspace with the proper replication factor and strategy) [1]
• Install redis stack to capture replayer problems
• Trigger journal replayer to fill in cassandra at postgresql's current level
• Install reaper (in charge of triggering repairs regularly)
  • Install keyspace
  • Install cluster seed in reaper ui
  • Install reaper application in staging cluster
• Add storage replayer redis configuration so they can report replaying issues
• Reset topics consumption so they replay from scratch
• (ongoing) replayer fill-in cassandra
• scrubber: Configure for cassandra
• Checks

[1]

CREATE KEYSPACE swh WITH replication = {'class': 'NetworkTopologyStrategy', 'sesi_rocquencourt_staging': '3'}  AND durable_writes = true;

changed milestone to %Cassandra in production as primary storage [Roadmap - Tooling and infrastructure]

• Basic checks regarding jmx connection [1]
• Create keyspace with the proper replication factor and strategy [2]

[1]

root@cassandra1:~# PASSWD=$(cat /etc/cassandra/jmxremote.password | awk '{print $2}')
root@cassandra1:~# for n in $(seq 1 3); do /opt/cassandra/bin/nodetool -h cassandra$n -u cassandra --password $PASSWD  status; done
Datacenter: sesi_rocquencourt_staging
=====================================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address          Load        Tokens  Owns (effective)  Host ID                               Rack
UN  192.168.130.182  128.05 KiB  16      59.3%             76e43f7a-e1fc-405b-b3ef-361bb7e76b72  rack1
UN  192.168.130.181  123.43 KiB  16      64.7%             91e480eb-8f3c-4f8b-b4d4-1b5fe377e3f7  rack1
UN  192.168.130.183  129.64 KiB  16      76.0%             4f4fddc0-b7ce-468d-80dc-58cbfe23d030  rack1

Datacenter: sesi_rocquencourt_staging
=====================================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address          Load        Tokens  Owns (effective)  Host ID                               Rack
UN  192.168.130.182  128.05 KiB  16      59.3%             76e43f7a-e1fc-405b-b3ef-361bb7e76b72  rack1
UN  192.168.130.181  123.43 KiB  16      64.7%             91e480eb-8f3c-4f8b-b4d4-1b5fe377e3f7  rack1
UN  192.168.130.183  129.64 KiB  16      76.0%             4f4fddc0-b7ce-468d-80dc-58cbfe23d030  rack1

Datacenter: sesi_rocquencourt_staging
=====================================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address          Load        Tokens  Owns (effective)  Host ID                               Rack
UN  192.168.130.182  128.05 KiB  16      59.3%             76e43f7a-e1fc-405b-b3ef-361bb7e76b72  rack1
UN  192.168.130.181  123.43 KiB  16      64.7%             91e480eb-8f3c-4f8b-b4d4-1b5fe377e3f7  rack1
UN  192.168.130.183  129.64 KiB  16      76.0%             4f4fddc0-b7ce-468d-80dc-58cbfe23d030  rack1

[2]

root@cassandra1:~# /opt/cassandra/bin/cqlsh $(hostname -f)
Connected to archive_staging at cassandra1.internal.staging.swh.network:9042
[cqlsh 6.0.0 | Cassandra 4.0.8 | CQL spec 3.4.5 | Native protocol v5]
Use HELP for help.
cqlsh> CREATE KEYSPACE swh WITH replication = {'class': 'NetworkTopologyStrategy', 'sesi_rocquencourt_staging': '3'}  AND durable_writes = true;
cqlsh> DESCRIBE KEYSPACES

swh     system_auth         system_schema  system_views
system  system_distributed  system_traces  system_virtual_schema

marked the checklist item Install data model in cassandra (create keyspace with the proper replication factor and strategy) [1] as completed

changed the description

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@8eaac6ee

Prepare redis installation:

$ cat ./redis-operator.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: archive-staging-rke2-redis-operator
  namespace: argocd
spec:
  project: default
  source:
    chart: redis-operator
    repoURL: 'https://ot-container-kit.github.io/helm-charts/'
    targetRevision: 0.14.3
    helm:
      releaseName: redis-operator
  destination:
    server: https://rancher.euwest.azure.internal.softwareheritage.org/k8s/clusters/c-m-9n5h9nrf
    namespace: redis
syncPolicy:
  automated:
    prune: true
    selfHeal: true
$ kubectl --kubeconfig ~/.kube/config.d/archive-staging-rke2.yaml create namespace redis
namespace/redis created
$ kubectl --kubeconfig ~/.kube/config.d/cluster-admin.yaml apply -f ./redis-operator.yaml
application.argoproj.io/archive-staging-rke2-redis-operator created

marked the checklist item Install redis stack to capture replayer problems as completed

Install the redis instance (for the replayer to push their issues if any) [1]

$ cd k8s-cluster-config
$ git pull
$ git checkout staging-cassandra
$ kubectl --context cluster-admin -f local-persistent-provisioner-application.yaml
application.argoproj.io/archive-staging-rke2-local-persistent-provisioner created
$ kubectl --context cluster-admin -f redis.yaml
application.argoproj.io/archive-staging-rke2-redis configured

changed the description

added cassandra state:wip labels

added 4h of time spent at 2023-03-21

mentioned in merge request swh/infra/ci-cd/swh-charts!39 (merged)

mentioned in merge request swh/infra/ci-cd/k8s-clusters-conf!18 (merged)

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@d3b000fb

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@d182b83e

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@cf194185

marked the checklist item Trigger journal replayer to fill in cassandra at postgresql's current level as completed

• replayers are deployed and running on the staging cluster (namespace swh-cassandra)
• cassandra's monitoring is deployed, the grafana dashboards needed to be updated to select the production or staging nodes

Some "adjustement" are still needed:

• redis configuration to store the replayer failures
• repear installation and configuration

mentioned in commit swh/infra/ci-cd/swh-charts@84cd39c3

mentioned in commit swh/infra/ci-cd/swh-charts@9eefae89

Cluster configuration in reaper:

changed the description

• Install the keyspace for reaper in cassandra backend [1]

[1]

cqlsh> CREATE KEYSPACE reaper_db WITH replication = {'class': 'NetworkTopologyStrategy', 'sesi_rocquencourt_staging': 3};
cqlsh>

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@f3c5c607

mentioned in commit swh/infra/puppet/puppet-swh-site@d9b8f095

mentioned in commit swh/infra/puppet/puppet-swh-site@f85c99c1

@guillaume is working on this too.

mentioned in commit swh/infra/ci-cd/swh-charts@53bac30d

mentioned in commit swh/infra/ci-cd/swh-charts@ee3b778c

marked the checklist item Install reaper application in staging cluster as completed

marked the checklist item Install reaper (in charge of triggering repairs regularly) as completed

mentioned in commit swh/infra/ci-cd/swh-charts@3f627084

mentioned in commit swh/infra/ci-cd/swh-charts@510b86a9

changed the description

Reset (well delete here) on storage1.staging:

root@storage1:~# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --list --all-topics | grep replayer | xargs -t -n1 /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --delete --all-topics --execute --group
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server storage1:9092 --delete --all-topics --execute --group swh-cassandra-stg-cassandra-replayer-origin
Deletion of requested consumer groups ('swh-cassandra-stg-cassandra-replayer-origin') was successful.
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server storage1:9092 --delete --all-topics --execute --group swh-cassandra-stg-cassandra-replayer-content
Deletion of requested consumer groups ('swh-cassandra-stg-cassandra-replayer-content') was successful.
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server storage1:9092 --delete --all-topics --execute --group swh-cassandra-stg-cassandra-replayer-metadata
Deletion of requested consumer groups ('swh-cassandra-stg-cassandra-replayer-metadata') was successful.
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server storage1:9092 --delete --all-topics --execute --group swh-cassandra-stg-cassandra-replayer-origin-visit
Deletion of requested consumer groups ('swh-cassandra-stg-cassandra-replayer-origin-visit') was successful.
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server storage1:9092 --delete --all-topics --execute --group swh-cassandra-stg-cassandra-replayer-release
Deletion of requested consumer groups ('swh-cassandra-stg-cassandra-replayer-release') was successful.
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server storage1:9092 --delete --all-topics --execute --group swh-cassandra-stg-cassandra-replayer-extid
Deletion of requested consumer groups ('swh-cassandra-stg-cassandra-replayer-extid') was successful.
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server storage1:9092 --delete --all-topics --execute --group swh-cassandra-stg-cassandra-replayer-origin-visit-status
Deletion of requested consumer groups ('swh-cassandra-stg-cassandra-replayer-origin-visit-status') was successful.
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server storage1:9092 --delete --all-topics --execute --group swh-cassandra-stg-cassandra-replayer-directory
Deletion of requested consumer groups ('swh-cassandra-stg-cassandra-replayer-directory') was successful.
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server storage1:9092 --delete --all-topics --execute --group swh-cassandra-stg-cassandra-replayer-revision
Deletion of requested consumer groups ('swh-cassandra-stg-cassandra-replayer-revision') was successful.
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server storage1:9092 --delete --all-topics --execute --group swh-cassandra-stg-cassandra-replayer-skipped-content
Deletion of requested consumer groups ('swh-cassandra-stg-cassandra-replayer-skipped-content') was successful.
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server storage1:9092 --delete --all-topics --execute --group swh-cassandra-stg-cassandra-replayer-snapshot
Deletion of requested consumer groups ('swh-cassandra-stg-cassandra-replayer-snapshot') was successful.
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server storage1:9092 --delete --all-topics --execute --group swh-cassandra-stg-cassandra-replayer-raw-extrinsic-metadata
Deletion of requested consumer groups ('swh-cassandra-stg-cassandra-replayer-raw-extrinsic-metadata') was successful.

mentioned in commit swh/infra/ci-cd/swh-charts@4ba124fa

marked the checklist item Reset topics consumption so they replay from scratch as completed

changed the description

changed the description

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@6304eb64

mentioned in merge request swh/infra/puppet/puppet-swh-site!615 (merged)