Upgrade cassandra in all environments
The version of cassandra we are using (4.0.8) is a couple of version late:
- in the 4.0 branch: 4.0.13 (unsupported since 2024-05-20)
- in the 4.1 branch: 4.1.7
- in the 5.0 branch: 5.0.2 (GA)
It could be a good idea to test an upgrade before the switch to cassandra in production.
We should update and validate the migration is going fine through our various testing environments first. Then finalize in the production environment if everything is fine.
1st:
-
dockerdocker environment is already in 4.1 branch - next-version: 4.0.1 -> 4.0.8 -> 4.0.15
- staging: 4.0.8 -> 4.0.15
- production: 4.0.8 -> 4.0.15
2nd:
- next-version: 4.0.15 -> 4.1.7
- staging: 4.0.15 -> 4.1.7
- production: 4.0.15 -> 4.1.7
3rd:
- next-version: 4.1 -> 5.0
- staging: 4.1 -> 5.0
- production: 4.1 -> 5.0
What are the important changes:
- 4.0.8 -> 4.0.15:
- https://github.com/apache/cassandra/blob/cassandra-4.0/NEWS.txt
- https://github.com/apache/cassandra/blob/cassandra-4.0/CHANGES.txt
- CASSANDRA-18773: compaction speedup
- CASSANDRA-18736: transaction log issue on startup (we had some issues that look like this (#5297 (closed)))
- 4.0 -> 4.1:
- https://github.com/apache/cassandra/blob/cassandra-4.1/NEWS.txt
- https://github.com/apache/cassandra/blob/cassandra-4.1/CHANGES.txt
- A lot of paxos improvements, cf upgrade procedure
-
CASSANDRA-15234 Unit change
Units suffixes should be provided for all rates(B/s|KiB/s|MiB/s), memory (B|KiB|MiB|GiB) and duration(d|h|m|s|ms|us|µs|ns) parameters.
- Some deprecations of mbeans methods, we should ensure the monitoring will still work as expected
- https://cassandra.apache.org/_/blog/Apache-Cassandra-4.1-New-SSTable-Identifiers.html
- 4.1 -> 5.0:
The recommanded upgrade path is to upgrade to the last patch available for the current Major.Minor version so in our case 4.0.8 -> 4.0.15 -> 4.1.7 -> 5.0.2
Upgrade procedure:
- 4.0.x -> 4.0.15
- Check reaper
- 4.0 -> 4.1
- Steps for upgrading Paxos:
- Set paxos_variant: v2 across the cluster. This may be set via JMX, but should also be written persistently to any yaml.
- Ensure paxos repairs are running regularly, either as part of normal incremental repair workflows or on their own separate schedule. These operations are cheap and better to run frequently (e.g. once per hour)
- Set paxos_state_purging: repaired across the cluster. This may be set via JMX, but should also be written persistently to any yaml. NOTE: once this has been set, you must not restore paxos_state_purging: legacy. If this setting must be disabled you must instead set paxos_state_purging: gc_grace. This may be necessary if paxos repairs must be disabled for some reason on an extended basis, but in this case your applications must restore default commit consistency to ensure correctness.
- Applications may now safely be updated to use ANY commit consistency level (or LOCAL_QUORUM, as preferred). Uncontended writes should now take 2 round-trips, and uncontended reads should typically take one round-trip.
- Steps for upgrading Paxos:
- 4.1 -> 5.0.2
Interesting documentation:
Activity
assigned to @vsellier
Check available cassandra 4.x releases:
ᐅ curl -s https://archive.apache.org/dist/cassandra/ | awk -F ">" '/4\.[01]\./{split($3,a,"/");print a[1]}' 4.0.0 4.0.1 4.0.10 4.0.11 4.0.12 4.0.13 4.0.14 4.0.15 4.0.2 4.0.3 4.0.4 4.0.5 4.0.6 4.0.7 4.0.8 4.0.9 4.1.0 4.1.1 4.1.2 4.1.3 4.1.4 4.1.5 4.1.6 4.1.7
next-version is managed through the k8ssandra-operator [1] We did not find the matrix table of k8ssandra-operator of cassandra version.
We'll start this by checking the behavior in the next-version environment. The impact in case of problem will be less heavy.
The current cassandra version is 4.0.1 in that environment [2] (it's hard-coded in the cluster-components chart). We'll do the adaptation so we can declare the version we want to use.
[2] We can also check the K8ssandraCluster kind serverVersion.
$ nodetool -u $USER -pw $PASS version ReleaseVersion: 4.0.1[1]
tony@kessel:~ $ helm search repo k8ssandra-operator -l | head -10 NAME CHART VERSION APP VERSION DESCRIPTION k8ssandra/k8ssandra-operator 1.20.3 1.20.3 Kubernetes operator which handles the provision... k8ssandra/k8ssandra-operator 1.20.2 1.20.2 Kubernetes operator which handles the provision... k8ssandra/k8ssandra-operator 1.20.1 1.20.1 Kubernetes operator which handles the provision... k8ssandra/k8ssandra-operator 1.20.0 1.20.0 Kubernetes operator which handles the provision... k8ssandra/k8ssandra-operator 1.19.0 1.19.0 Kubernetes operator which handles the provision... k8ssandra/k8ssandra-operator 1.18.0 1.18.0 Kubernetes operator which handles the provision... k8ssandra/k8ssandra-operator 1.17.0 1.17.0 Kubernetes operator which handles the provision... <------------------------ current version k8ssandra/k8ssandra-operator 1.16.0 1.16.0 Kubernetes operator which handles the provision... k8ssandra/k8ssandra-operator 1.15.0 1.15.0 Kubernetes operator which handles the provision...Edited by Antoine R. Dumontmentioned in commit swh/infra/ci-cd/swh-charts@7a6391fe
mentioned in commit swh/infra/ci-cd/swh-charts@56f0586c
mentioned in commit swh/infra/ci-cd/swh-charts@60f966d3
mentioned in commit swh/infra/ci-cd/swh-charts@600543b5
assigned to @guillaume and unassigned @vsellier
mentioned in commit swh/infra/ci-cd/swh-charts@bfa2710c
mentioned in commit swh/infra/ci-cd/swh-charts@563cd79e
mentioned in commit swh/infra/ci-cd/swh-charts@1caf02eb
mentioned in commit swh/infra/ci-cd/swh-charts@18a83b33
