Upgrade cassandra from 4.0.5 to 4.0.8
The previous cassandra version are not available in the default directory, only the last versions.
In order to install the new servers, we could get the package from the archive or upgrade to the latest lts version.
Let's try the upgrade, it will allow to test the upgrade procedure before having real load on the cluster.
Designs
- #4737Cassandra in production as primary storage [Roadmap - Tooling and infrastructure]
Activity
changed milestone to %Cassandra in production as primary storage [Roadmap - Tooling and infrastructure]
added activity::Deployment cassandra state:wip labels
assigned to @vsellier
marked this issue as related to #4737 (closed)
mentioned in commit swh/infra/puppet/puppet-swh-site@f9bc97fb
Upgrade plan:
- disable the compactions
- make a snapshot of the data (for the fun)
- apply puppet on all the nodes (cassandra is not restarted by puppet to be able to control the cluster restart)
- perform a rolling restart of the cluster
- check the replayers are still working
- restart the compactions
- remove the snapshot
Edited by Vincent Sellier1/ disabling autocompaction
root@pergamon:~# clush -b -w @cassandra "/opt/cassandra/bin/nodetool -u cassandra --password [redacted] disableautocompaction" root@pergamon:~# clush -b -w @cassandra "/opt/cassandra/bin/nodetool -u cassandra --password [redacted] statusautocompaction" --------------- cassandra[01-06] (6) --------------- not running2/ create a snapshot
Snapshot creation and status
vsellier@cassandra01 ~ % /opt/cassandra/bin/nodetool -u cassandra --password [redacted] listsnapshots Snapshot Details: There are no snapshots vsellier@cassandra01 ~ % /opt/cassandra/bin/nodetool -u cassandra --password [redacted] snapshot Requested creating snapshot(s) for [all keyspaces] with snapshot name [1678368683998] and options {skipFlush=false} Snapshot directory: 1678368683998 vsellier@cassandra01 ~ % /opt/cassandra/bin/nodetool -u cassandra --password [redacted] listsnapshots Snapshot Details: Snapshot name Keyspace name Column family name True size Size on disk 1678368683998 system_schema columns 0 bytes 29.49 KiB 1678368683998 system_schema types 0 bytes 5.93 KiB 1678368683998 system_schema indexes 0 bytes 13 bytes 1678368683998 system_schema keyspaces 0 bytes 6.06 KiB 1678368683998 system_schema dropped_columns 0 bytes 16.21 KiB 1678368683998 system_schema aggregates 0 bytes 11.03 KiB 1678368683998 system_schema triggers 0 bytes 13 bytes 1678368683998 system_schema tables 0 bytes 21.52 KiB 1678368683998 system_schema views 0 bytes 13 bytes 1678368683998 system_schema functions 0 bytes 11.48 KiB 1678368683998 system compaction_history 0 bytes 245.26 KiB 1678368683998 system IndexInfo 0 bytes 13 bytes 1678368683998 system repairs 0 bytes 74.38 KiB 1678368683998 system size_estimates 0 bytes 64.61 KiB 1678368683998 system table_estimates 0 bytes 115.39 KiB 1678368683998 system paxos 0 bytes 377.47 MiB 1678368683998 system built_views 0 bytes 13 bytes 1678368683998 system peer_events 0 bytes 13 bytes 1678368683998 system peers_v2 0 bytes 20.17 KiB 1678368683998 system peers 0 bytes 19.79 KiB 1678368683998 system peer_events_v2 0 bytes 13 bytes 1678368683998 system batches 0 bytes 13 bytes 1678368683998 system transferred_ranges 0 bytes 13 bytes 1678368683998 system transferred_ranges_v2 0 bytes 13 bytes 1678368683998 system view_builds_in_progress 0 bytes 13 bytes 1678368683998 system local 0 bytes 12.11 KiB 1678368683998 system sstable_activity 0 bytes 62.69 KiB 1678368683998 system available_ranges_v2 0 bytes 13 bytes 1678368683998 system available_ranges 0 bytes 13 bytes 1678368683998 system prepared_statements 0 bytes 15.65 KiB 1678368683998 system_auth roles 0 bytes 5.09 KiB 1678368683998 system_auth role_members 0 bytes 13 bytes 1678368683998 system_auth resource_role_permissons_index 0 bytes 13 bytes 1678368683998 system_auth network_permissions 0 bytes 13 bytes 1678368683998 system_auth role_permissions 0 bytes 13 bytes 1678368683998 swh raw_extrinsic_metadata_by_id 0 bytes 2.01 GiB 1678368683998 swh skipped_content_by_sha1_git 0 bytes 7.87 MiB 1678368683998 swh directory_entry 0 bytes 6.89 TiB 1678368683998 swh content_by_sha256 0 bytes 339.52 GiB 1678368683998 swh content 0 bytes 648.56 GiB 1678368683998 swh content_by_sha1 0 bytes 262.35 GiB 1678368683998 swh origin_visit 0 bytes 19.28 GiB 1678368683998 swh metadata_authority 0 bytes 17.4 KiB 1678368683998 swh skipped_content_by_sha256 0 bytes 10.24 MiB 1678368683998 swh directory 0 bytes 309.22 GiB 1678368683998 swh origin 0 bytes 10.27 GiB 1678368683998 swh metadata_fetcher 0 bytes 27.88 KiB 1678368683998 swh extid_by_target 0 bytes 3.84 GiB 1678368683998 swh origin_visit_status 0 bytes 33.27 GiB 1678368683998 swh raw_extrinsic_metadata 0 bytes 36.13 GiB 1678368683998 swh snapshot 0 bytes 6.1 GiB 1678368683998 swh extid 0 bytes 3.93 GiB 1678368683998 swh snapshot_branch 0 bytes 118.9 GiB 1678368683998 swh release 0 bytes 3.79 GiB 1678368683998 swh revision_parent 0 bytes 112.81 GiB 1678368683998 swh content_by_sha1_git 0 bytes 261.9 GiB 1678368683998 swh skipped_content_by_sha1 0 bytes 7.65 MiB 1678368683998 swh revision 0 bytes 412.98 GiB 1678368683998 swh skipped_content 0 bytes 29.11 MiB 1678368683998 swh skipped_content_by_blake2s256 0 bytes 10.25 MiB 1678368683998 swh content_by_blake2s256 0 bytes 349.99 GiB 1678368683998 system_distributed parent_repair_history 0 bytes 4.41 MiB 1678368683998 system_distributed repair_history 0 bytes 5.74 MiB 1678368683998 system_distributed view_build_status 0 bytes 13 bytes 1678368683998 reaper_db repair_run_by_cluster_v2 0 bytes 71.35 KiB 1678368683998 reaper_db repair_run_by_cluster 0 bytes 964 bytes 1678368683998 reaper_db schema_migration 0 bytes 19.83 KiB 1678368683998 reaper_db repair_schedule_by_cluster_and_keyspace 0 bytes 12.84 KiB 1678368683998 reaper_db repair_schedule_v1 0 bytes 15.01 KiB 1678368683998 reaper_db schema_migration_leader 0 bytes 964 bytes 1678368683998 reaper_db node_metrics_v3 0 bytes 1.36 KiB 1678368683998 reaper_db leader 0 bytes 11.68 KiB 1678368683998 reaper_db snapshot 0 bytes 984 bytes 1678368683998 reaper_db node_metrics_v1 0 bytes 1.23 KiB 1678368683998 reaper_db running_repairs 0 bytes 11.58 KiB 1678368683998 reaper_db cluster 0 bytes 12.54 KiB 1678368683998 reaper_db running_reapers 0 bytes 946 bytes 1678368683998 reaper_db percent_repaired_by_schedule 0 bytes 39.58 KiB 1678368683998 reaper_db repair_unit_v1 0 bytes 7.9 KiB 1678368683998 reaper_db repair_run_by_unit 0 bytes 39.23 KiB 1678368683998 reaper_db repair_run 0 bytes 3.16 MiB 1678368683998 reaper_db diagnostic_event_subscription 0 bytes 1.03 KiB 1678368683998 reaper_db node_operations 0 bytes 1.13 KiB 1678368683998 system_traces sessions 0 bytes 13 bytes 1678368683998 system_traces events 0 bytes 13 bytesTotal TrueDiskSpaceUsed: 0 bytes
Edited by Vincent Sellier3/ apply puppet and 4/ perform a rolling restart
- check the current version
root@pergamon:~# clush -b -w @cassandra "/opt/cassandra/bin/nodetool -u cassandra --password [redacted] version" --------------- cassandra[01-06] (6) --------------- ReleaseVersion: 4.0.5- rolling restart of the cluster (in progress)- apply puppet
root@pergamon:~# clush -b -w @cassandra "puppet agent --test" --------------- cassandra01 --------------- Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Retrieving locales Info: Loading facts Info: Caching catalog for cassandra01.internal.softwareheritage.org Info: Applying configuration version '1678381985' Notice: /Stage[main]/Profile::Cassandra/File[/opt/cassandra-4.0.8]/ensure: created Notice: /Stage[main]/Profile::Cassandra/Archive[cassandra]/ensure: download archive from https://dlcdn.apache.org/cassandra/4.0.8/apache-cassandra-4.0.8-bin.tar.gz to /opt/apache-cassandra-4.0.8-bin.tar.gz and extracted in /opt/cassandra-4.0.8 with cleanup (corrective) Notice: /Stage[main]/Profile::Cassandra/File[/opt/cassandra]/target: target changed '/opt/cassandra-4.0.5' to '/opt/cassandra-4.0.8' Notice: /Stage[main]/Profile::Cassandra/File[/etc/cassandra/jmx_exporter.yml]/content: Info: Computing checksum on file /etc/cassandra/jmx_exporter.yml Info: /Stage[main]/Profile::Cassandra/File[/etc/cassandra/jmx_exporter.yml]: Filebucketed /etc/cassandra/jmx_exporter.yml to puppet with sum 877cf6491dfef559ebc434ac5c849a87 Notice: /Stage[main]/Profile::Cassandra/File[/etc/cassandra/jmx_exporter.yml]/content: content changed '{mtime}2023-03-09 16:55:21 +0000' to '{mtime}2023-03-09 17:13:25 +0000' Notice: Applied catalog in 10.19 seconds ... --------------- cassandra06 --------------- Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Retrieving locales Info: Loading facts Info: Caching catalog for cassandra06.internal.softwareheritage.org Info: Applying configuration version '1678381985' Notice: /Stage[main]/Profile::Cassandra/File[/opt/cassandra-4.0.8]/ensure: created Notice: /Stage[main]/Profile::Cassandra/Archive[cassandra]/ensure: download archive from https://dlcdn.apache.org/cassandra/4.0.8/apache-cassandra-4.0.8-bin.tar.gz to /opt/apache-cassandra-4.0.8-bin.tar.gz and extracted in /opt/cassandra-4.0.8 with cleanup (corrective) Notice: /Stage[main]/Profile::Cassandra/File[/opt/cassandra]/target: target changed '/opt/cassandra-4.0.5' to '/opt/cassandra-4.0.8' Notice: /Stage[main]/Profile::Cassandra/File[/etc/cassandra/jmx_exporter.yml]/content:Info: Computing checksum on file /etc/cassandra/jmx_exporter.yml Info: /Stage[main]/Profile::Cassandra/File[/etc/cassandra/jmx_exporter.yml]: Filebucketed /etc/cassandra/jmx_exporter.yml to puppet with sum 877cf6491dfef559ebc434ac5c849a87 Notice: /Stage[main]/Profile::Cassandra/File[/etc/cassandra/jmx_exporter.yml]/content: content changed '{mtime}2023-03-09 17:07:22 +0000' to '{mtime}2023-03-09 17:13:23 +0000' Notice: Applied catalog in 10.15 seconds clush: cassandra[01-06] (6): exited with exit code 2
root@pergamon:/home/vsellier# cd root@pergamon:~# /home/vsellier/restart-cassandra-cluster.sh restarting cassandra01.internal.softwareheritage.org... Connection to cassandra01.internal.softwareheritage.org closed. Waiting for the node to be down.Node cassandra01.internal.softwareheritage.org was correctly stopped Waiting for node cassandra01.internal.softwareheritage.org to be up.................................................................................................... ...............................................root@pergamon:~# clush -b -w @cassandra "/opt/cassandra/bin/nodetool -u cassandra --password [redacted] version" --------------- cassandra[02-06] (5) --------------- ReleaseVersion: 4.0.5 --------------- cassandra01 --------------- ReleaseVersion: 4.0.8Edited by Vincent Selliercluster fully restarted after quite a wild:
root@pergamon:~# /home/vsellier/restart-cassandra-cluster.sh restarting cassandra01.internal.softwareheritage.org... Connection to cassandra01.internal.softwareheritage.org closed. Waiting for the node to be down.Node cassandra01.internal.softwareheritage.org was correctly stopped Waiting for node cassandra01.internal.softwareheritage.org to be up.........................................................................................................................................................................................Node cassandra01.internal.softwareheritage.org successfully restarted Waiting 60 before restarting next node restarting cassandra02.internal.softwareheritage.org... Connection to cassandra02.internal.softwareheritage.org closed. Waiting for the node to be down.Node cassandra02.internal.softwareheritage.org was correctly stopped Waiting for node cassandra02.internal.softwareheritage.org to be up.................................................................................................................................................................. .......................Node cassandra02.internal.softwareheritage.org successfully restarted Waiting 60 before restarting next node restarting cassandra03.internal.softwareheritage.org... Connection to cassandra03.internal.softwareheritage.org closed. Waiting for the node to be down.Node cassandra03.internal.softwareheritage.org was correctly stopped Waiting for node cassandra03.internal.softwareheritage.org to be up....................................................................................................................................................... ........................................Node cassandra03.internal.softwareheritage.org successfully restarted Waiting 60 before restarting next node restarting cassandra04.internal.softwareheritage.org... Connection to cassandra04.internal.softwareheritage.org closed. Waiting for the node to be down.Node cassandra04.internal.softwareheritage.org was correctly stopped Waiting for node cassandra04.internal.softwareheritage.org to be up....................................................................................................................................................... .....................................................Node cassandra04.internal.softwareheritage.org successfully restarted Waiting 60 before restarting next node restarting cassandra05.internal.softwareheritage.org... Connection to cassandra05.internal.softwareheritage.org closed. Waiting for the node to be down.Node cassandra05.internal.softwareheritage.org was correctly stopped Waiting for node cassandra05.internal.softwareheritage.org to be up....................................................................................................................................................... ..............................................Node cassandra05.internal.softwareheritage.org successfully restarted Waiting 60 before restarting next node restarting cassandra06.internal.softwareheritage.org... Connection to cassandra06.internal.softwareheritage.org closed. Waiting for the node to be down.Node cassandra06.internal.softwareheritage.org was correctly stopped Waiting for node cassandra06.internal.softwareheritage.org to be up....................................................................................................................................................... ..............................................Node cassandra06.internal.softwareheritage.org successfully restarted Waiting 60 before restarting next node Done, cluster completely restartedroot@pergamon:~# clush -b -w @cassandra "/opt/cassandra/bin/nodetool -u cassandra --password [redacted] version" --------------- cassandra[01-06] (6) --------------- ReleaseVersion: 4.0.8Edited by Vincent Sellier5/ Check the replayers
- There are no errors on the replayers, the content and directory replayers are restarted to recover the lag after their consumer group were removed. the reads look quite fast.
6/ Restart the compactions The autocompaction was automatically reactivated after the nodes restart because they were not deactivated in the configuration (taking note for later)
7/ delete snapshots
root@pergamon:~# clush -b -w @cassandra "/opt/cassandra/bin/nodetool -u cassandra --password [redacted] clearsnapshot --all" --------------- cassandra[01-06] (6) --------------- Requested clearing snapshot(s) for [all keyspaces] with [all snapshots] root@pergamon:~# clush -b -w @cassandra "/opt/cassandra/bin/nodetool -u cassandra --password [redacted] listsnapshots" --------------- cassandra[01-06] (6) --------------- Snapshot Details: There are no snapshotsremoved state:wip label
Please register or sign in to reply