Decommission dbreplica01.euwest.azure
It's no longer used by other services [1] as the replication has not been working for a while. As the vm is still running on the azure infra, it still costs azure credits though.
Let's clean up.
-
Deactivate puppet on impacted node -
Decommission certificate on puppet master [2] -
Ensure no one is using it [1] -
Stop node and wait for alerts if any services got missed -
Decommission node (click delete
on azure portal as the node is not currently terraform declared) -
(azure specifics) Drop no longer necessary resources [3] -
Drop swh-site (puppet) references
[1]
root@pergamon:~# clush -b -w @indexer-workers grep -riA2 "indexer_storage:" /etc/softwareheritage/indexer/
---------------
indexer-worker[01-03].euwest.azure (3)
---------------
/etc/softwareheritage/indexer/extrinsic_metadata.yml:indexer_storage:
/etc/softwareheritage/indexer/extrinsic_metadata.yml- cls: remote
/etc/softwareheritage/indexer/extrinsic_metadata.yml- url: http://saam.internal.softwareheritage.org:5007/
--
/etc/softwareheritage/indexer/content_mimetype.yml:indexer_storage:
/etc/softwareheritage/indexer/content_mimetype.yml- cls: remote
/etc/softwareheritage/indexer/content_mimetype.yml- url: http://saam.internal.softwareheritage.org:5007/
--
/etc/softwareheritage/indexer/content_fossology_license.yml:indexer_storage:
/etc/softwareheritage/indexer/content_fossology_license.yml- cls: remote
/etc/softwareheritage/indexer/content_fossology_license.yml- url: http://saam.internal.softwareheritage.org:5007/
---------------
indexer-worker[04-06].euwest.azure (3)
---------------
/etc/softwareheritage/indexer/origin_intrinsic_metadata.yml:indexer_storage:
/etc/softwareheritage/indexer/origin_intrinsic_metadata.yml- cls: remote
/etc/softwareheritage/indexer/origin_intrinsic_metadata.yml- url: http://saam.internal.softwareheritage.org:5007/
--
/etc/softwareheritage/indexer/extrinsic_metadata.yml:indexer_storage:
/etc/softwareheritage/indexer/extrinsic_metadata.yml- cls: remote
/etc/softwareheritage/indexer/extrinsic_metadata.yml- url: http://saam.internal.softwareheritage.org:5007/
root@pergamon:~# clush -b -w @vault-workers grep -riA2 "storage:" /etc/softwareheritage/vault_cooker.yml
---------------
vault-worker[01-02].euwest.azure (2)
---------------
storage:
cls: retry
storage:
cls: remote
url: http://saam.internal.softwareheritage.org:5002/
[2]
root@pergamon:~# swh-puppet-master-decommission dbreplica1.euwest.azure.internal.softwareheritage.org
+ puppet node deactivate dbreplica1.euwest.azure.internal.softwareheritage.org
Submitted 'deactivate node' for dbreplica1.euwest.azure.internal.softwareheritage.org with UUID 97be66ae-e4f5-48b7-807f-a12194c7bf4d
+ puppet node clean dbreplica1.euwest.azure.internal.softwareheritage.org
Notice: Revoked certificate with serial 138
Notice: Removing file Puppet::SSL::Certificate dbreplica1.euwest.azure.internal.softwareheritage.org at '/var/lib/puppet/ssl/ca/signed/dbreplica1.euwest.azure.internal.softwa
reheritage.org.pem'
dbreplica1.euwest.azure.internal.softwareheritage.org
+ puppet cert clean dbreplica1.euwest.azure.internal.softwareheritage.org
Warning: `puppet cert` is deprecated and will be removed in a future release.
(location: /usr/lib/ruby/vendor_ruby/puppet/application.rb:370:in `run')
Notice: Revoked certificate with serial 137
Notice: Revoked certificate with serial 138
+ systemctl restart apache2
root@pergamon:~# puppet agent --test Info: Using configured environment 'production' Info: Retrieving pluginfacts
Info: Retrieving plugin Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for pergamon.softwareheritage.org
Info: Applying configuration version '1674479896' Info: Computing checksum on file /etc/ssh/puppet_known_hosts/dbreplica1.euwest.azure.internal.softwareheritage.org.keys
Info: /Stage[main]/Profile::Ssh::Server/File[/etc/ssh/puppet_known_hosts/dbreplica1.euwest.azure.internal.softwareheritage.org.keys]: Filebucketed /etc/ssh/puppet_known_hosts/dbreplica1.euwest.azure.internal.softwareheritage.org.keys to puppet with sum e96992ea03d8e2fa781113de64f3a455
Notice: /Stage[main]/Profile::Ssh::Server/File[/etc/ssh/puppet_known_hosts/dbreplica1.euwest.azure.internal.softwareheritage.org.keys]/ensure: removed
Info: /etc/ssh/puppet_known_hosts: Scheduling refresh of Exec[update ssh_known_hosts] Notice: /Stage[main]/Profile::Ssh::Server/Exec[update ssh_known_hosts]: Triggered 'refresh' from 1 event
Info: Computing checksum on file /etc/icinga2/zones.d/master/dbreplica1.euwest.azure.internal.softwareheritage.org.conf Info: /Stage[main]/Profile::Icinga2::Master/File[/etc/icinga2/zones.d/master/dbreplica1.euwest.azure.internal.softwareheritage.org.conf]: Filebucketed /etc/icinga2/zones.d/master/dbreplica1.euwest.azure.internal.softwareheritage.org.conf to puppet with sum da698691244e1e0283dff70419f1bf41
Notice: /Stage[main]/Profile::Icinga2::Master/File[/etc/icinga2/zones.d/master/dbreplica1.euwest.azure.internal.softwareheritage.org.conf]/ensure: removed Info: /etc/icinga2/zones.d/master: Scheduling refresh of Class[Icinga2::Service] Info: Computing checksum on file /etc/prometheus/exported-configs/dbreplica1.euwest.azure_node.yaml
Info: /Stage[main]/Profile::Prometheus::Server/File[/etc/prometheus/exported-configs/dbreplica1.euwest.azure_node.yaml]: Filebucketed /etc/prometheus/exported-configs/dbreplica1.euwest.azure_node.yaml to puppet with sum 08786915f3712a73b6ab5547edcf3e26
Notice: /Stage[main]/Profile::Prometheus::Server/File[/etc/prometheus/exported-configs/dbreplica1.euwest.azure_node.yaml]/ensure: removed
Info: Computing checksum on file /etc/prometheus/exported-configs/dbreplica1.euwest.azure_sql.yaml
Info: /Stage[main]/Profile::Prometheus::Server/File[/etc/prometheus/exported-configs/dbreplica1.euwest.azure_sql.yaml]: Filebucketed /etc/prometheus/exported-configs/dbreplica1.euwest.azure_sql.yaml to puppet with sum 112d7c6f6072d8d6fb86e785468e5d20
Notice: /Stage[main]/Profile::Prometheus::Server/File[/etc/prometheus/exported-configs/dbreplica1.euwest.azure_sql.yaml]/ensure: removed
Info: Computing checksum on file /etc/prometheus/exported-configs/dbreplica1.euwest.azure_statsd.yaml
Info: /Stage[main]/Profile::Prometheus::Server/File[/etc/prometheus/exported-configs/dbreplica1.euwest.azure_statsd.yaml]: Filebucketed /etc/prometheus/exported-configs/dbreplica1.euwest.azure_statsd.yaml to puppet with sum 0b7ea1345dc93319ec43100ce37f9a94
Notice: /Stage[main]/Profile::Prometheus::Server/File[/etc/prometheus/exported-configs/dbreplica1.euwest.azure_statsd.yaml]/ensure: removed
Info: /etc/prometheus/exported-configs: Scheduling refresh of Exec[update-prometheus-config]
Notice: /Stage[main]/Profile::Bind_server::Primary/Resource_record[k8s-rancher-app/CNAME]/ensure: created
Notice: /Stage[main]/Profile::Prometheus::Server/Exec[update-prometheus-config]: Triggered 'refresh' from 1 event
Info: /Stage[main]/Profile::Prometheus::Server/Exec[update-prometheus-config]: Scheduling refresh of Service[prometheus]
Notice: /Stage[main]/Profile::Prometheus::Server/Service[prometheus]: Triggered 'refresh' from 1 event
Info: Class[Icinga2::Service]: Scheduling refresh of Service[icinga2]
Notice: /Stage[main]/Icinga2::Service/Service[icinga2]: Triggered 'refresh' from 1 event
Notice: Applied catalog in 62.03 seconds
Edited by Antoine R. Dumont