CoreDNS configuration was reset during rke2 upgrade on cluster archive-production-rke2

During the first rke2 upgrade (v1.26.15-rke2r1) on cluster archive-production-rke2, the customized CoreDNS configuration was reset. This explain the 503 errors on archive.softwareheritage.org.

1. I restart the pods web-archive.
2. I remove the svix endpoint archive.softwareheritage.org of event-type origin.visit and restart the pods web-archive.

swh@webhooks-origin-visit-status-6d9d6c4855-bfn6b:~$ swh webhooks endpoint delete origin.visit https://archive.softwareheritage.org/save/origin/visit/webhook/

3. Finally after re-apply terraform with the modifications due to the rke2 upgrades, everything works fine again.

Terraform will perform the following actions:

  # rancher2_cluster_v2.archive-production-rke2 will be updated in-place
  ~ resource "rancher2_cluster_v2" "archive-production-rke2" {
        id                         = "fleet-default/archive-production-rke2"
        name                       = "archive-production-rke2"
        # (9 unchanged attributes hidden)

      ~ rke_config {
          ~ chart_values          = <<-EOT
                rke2-calico: {}
              + rke2-coredns:
              +   autoscaler:
              +     coresPerReplica: 64
              +     max: 5
              +     preventSinglePointFailure: true
              +   resources:
              +     limits:
              +       cpu: 8
              +     requests:
              +       cpu: 500m
              +       memory: 128Mi
            EOT
            # (2 unchanged attributes hidden)

          ~ machine_selector_config {
              + config = {
                  + "cloud-provider-name" = null
                }
            }

            # (4 unchanged blocks hidden)
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

/cc @teams/sysadmin

assigned to @guillaume

mentioned in commit swh-sysadmin-provisioning@1c557dd9

I've recreated the webhooks endpoint:

swh@webhooks-origin-visit-status-6d9d6c4855-bfn6b:~$ swh webhooks -C $SWH_CONFIG_FILENAME endpoint create origin.visit https://archive.softwareheritage.org/save/origin/visit/webhook/ --secret ${TOKEN}
swh@webhooks-origin-visit-status-6d9d6c4855-bfn6b:~$ swh webhooks -C $SWH_CONFIG_FILENAME endpoint list origin.visit
https://archive.softwareheritage.org/save/origin/visit/webhook/

closed