Migrate rke "admin" cluster to rke2 cluster

changed milestone to %Dynamic infrastructure [Roadmap - Tooling and infrastructure]

added System administration activity::MRO labels

assigned to @vsellier

marked this issue as related to #5021 (closed)

mentioned in issue #5021 (closed)

changed the description

mentioned in commit swh/infra/puppet/puppet-swh-site@ed6df6f1

mentioned in merge request swh/infra/puppet/puppet-swh-site!634 (merged)

mentioned in commit swh/infra/ci-cd/swh-charts@c7ea1d91

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@77c1018c

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@aafab933

mentioned in commit swh-sysadmin-provisioning@706aca29

marked the checklist item Declare the IPs on the inventory as completed

marked the checklist item Declare and Create a new cluster admin-rke2 with terraform [1] as completed

marked the checklist item Create the thanos configuration for the cluster in puppet and azure as completed

marked the checklist item Declare the cluster in the firewall aliases as completed

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@9dbf787f

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@ae197782

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@27ae0613

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@3dd7d224

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@e2ed9280

mentioned in commit swh/infra/puppet/puppet-swh-site@cff4f5c8

marked the checklist item Migrate minio vhost to the new cluster as completed

marked the checklist item Create the argocd configuration (Cluster, secrets, configurations (metallb, monitoring, alerting, blackbox, ...) as completed

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@d53ed186

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@e4842772

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@a2cfebe3

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@d0d786e7

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@f1868431

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@0f673a3c

mentioned in commit swh/infra/puppet/puppet-swh-site@74722147

mentioned in commit swh/infra/ci-cd/swh-charts@584f1631

Argocd is now fully active in the new cluster. I've changed the main application (argocd-applications) to disable pruning and self healing to avoid any side effect during the double deployment on the server.

All the admin configuration can be decommissioned now.

mentioned in commit swh-sysadmin-provisioning@fe443b13

former nodes decommissioned in puppet

root@pergamon:~# swh-puppet-master-decommission rancher-node-admin-mgmt1.internal.admin.swh.network
+ puppet node deactivate rancher-node-admin-mgmt1.internal.admin.swh.network
Submitted 'deactivate node' for rancher-node-admin-mgmt1.internal.admin.swh.network with UUID 82abbb5c-6571-4d76-860f-b49b5add9986
+ puppet node clean rancher-node-admin-mgmt1.internal.admin.swh.network
Notice: Revoked certificate with serial 379
Notice: Removing file Puppet::SSL::Certificate rancher-node-admin-mgmt1.internal.admin.swh.network at '/var/lib/puppet/ssl/ca/signed/rancher-node-admin-mgmt1.internal.admin.swh.network.pem'
rancher-node-admin-mgmt1.internal.admin.swh.network
+ puppet cert clean rancher-node-admin-mgmt1.internal.admin.swh.network
Warning: `puppet cert` is deprecated and will be removed in a future release.
   (location: /usr/lib/ruby/vendor_ruby/puppet/application.rb:370:in `run')
Notice: Revoked certificate with serial 376
Notice: Revoked certificate with serial 379
+ systemctl restart apache2
root@pergamon:~# swh-puppet-master-decommission rancher-node-admin-node01.internal.admin.swh.network
+ puppet node deactivate rancher-node-admin-node01.internal.admin.swh.network
Submitted 'deactivate node' for rancher-node-admin-node01.internal.admin.swh.network with UUID 25bdfa94-2557-4038-8c1c-beecfdab24c6
+ puppet node clean rancher-node-admin-node01.internal.admin.swh.network
Notice: Revoked certificate with serial 380
Notice: Removing file Puppet::SSL::Certificate rancher-node-admin-node01.internal.admin.swh.network at '/var/lib/puppet/ssl/ca/signed/rancher-node-admin-node01.internal.admin.swh.network.pem'
rancher-node-admin-node01.internal.admin.swh.network
+ puppet cert clean rancher-node-admin-node01.internal.admin.swh.network
Warning: `puppet cert` is deprecated and will be removed in a future release.
   (location: /usr/lib/ruby/vendor_ruby/puppet/application.rb:370:in `run')
Notice: Revoked certificate with serial 377
Notice: Revoked certificate with serial 380
+ systemctl restart apache2
root@pergamon:~# swh-puppet-master-decommission rancher-node-admin-node02.internal.admin.swh.network
+ puppet node deactivate rancher-node-admin-node02.internal.admin.swh.network
Submitted 'deactivate node' for rancher-node-admin-node02.internal.admin.swh.network with UUID 49dd06b7-5cf9-474a-b4f2-b2d7aa8fc282
+ puppet node clean rancher-node-admin-node02.internal.admin.swh.network
Notice: Revoked certificate with serial 378
Notice: Removing file Puppet::SSL::Certificate rancher-node-admin-node02.internal.admin.swh.network at '/var/lib/puppet/ssl/ca/signed/rancher-node-admin-node02.internal.admin.swh.network.pem'
rancher-node-admin-node02.internal.admin.swh.network
+ puppet cert clean rancher-node-admin-node02.internal.admin.swh.network
Warning: `puppet cert` is deprecated and will be removed in a future release.
   (location: /usr/lib/ruby/vendor_ruby/puppet/application.rb:370:in `run')
Notice: Revoked certificate with serial 378
+ systemctl restart apache2
root@pergamon:~# swh-puppet-master-decommission rancher-node-admin-node03.internal.admin.swh.network
+ puppet node deactivate rancher-node-admin-node03.internal.admin.swh.network
Submitted 'deactivate node' for rancher-node-admin-node03.internal.admin.swh.network with UUID 11c4c748-0693-4468-883b-699dbefb09b7
+ puppet node clean rancher-node-admin-node03.internal.admin.swh.network
Notice: Revoked certificate with serial 381
Notice: Removing file Puppet::SSL::Certificate rancher-node-admin-node03.internal.admin.swh.network at '/var/lib/puppet/ssl/ca/signed/rancher-node-admin-node03.internal.admin.swh.network.pem'
rancher-node-admin-node03.internal.admin.swh.network
+ puppet cert clean rancher-node-admin-node03.internal.admin.swh.network
Warning: `puppet cert` is deprecated and will be removed in a future release.
   (location: /usr/lib/ruby/vendor_ruby/puppet/application.rb:370:in `run')
Notice: Revoked certificate with serial 381
+ systemctl restart apache2
root@pergamon:~# puppet agent --test

cluster and associated resources destroyed by terraform:

terraform plan --destroy --target=rancher2_cluster.cluster-admin --out=/tmp/terraform.out                                                                                            16:56:00
rancher2_cluster.cluster-admin: Refreshing state... [id=c-q2wd4]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # rancher2_app_v2.cluster-admin-rancher-monitoring will be destroyed
...
  Plan: 0 to add, 0 to change, 7 to destroy.
╷
│ Warning: Resource targeting is in effect