Argocd can't synchronize staging applications

This is present in the logs:

time="2024-04-03T07:18:03Z" level=error msg="Failed to sync cluster" error="failed to load initial state of resource Redis.redis.redis.opstreelabs.in: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=Redis failed: Post \"https://webhook-service.redis-operator.svc:443/convert?timeout=30s\": service \"webhook-service\" not found" server="https://rancher.euwest.azure.internal.softwareheritage.org/k8s/clusters/c-m-9n5h9nrf"
time="2024-04-03T07:18:03Z" level=error msg="Failed to cache app resources: error getting resource tree: failed to get app hosts: error synchronizing cache state : failed to sync cluster https://rancher.euwest.azure.internal.softwareheritage.org/k8s/clusters/c-m-9n5h9nrf: failed to load initial state of resource Redis.redis.redis.opstreelabs.in: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=Redis failed: Post \"https://webhook-service.redis-operator.svc:44
time="2024-04-03T07:18:03Z" level=info msg="Skipping auto-sync: application status is Unknown" application=argocd/swh-swh-next-version-377
time="2024-04-03T07:18:03Z" level=error msg="Failed to cache app resources: error getting resource tree: failed to get app hosts: error synchronizing cache state : failed to sync cluster https://rancher.euwest.azure.internal.softwareheritage.org/k8s/clusters/c-m-9n5h9nrf: failed to load initial state of resource Redis.redis.redis.opstreelabs.in: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=Redis failed: Post \"https://webhook-service.redis-operator.svc:44
time="2024-04-03T07:18:03Z" level=info msg="Skipping auto-sync: application status is Unknown" application=argocd/archive-staging-rke2-swh-cassandra
time="2024-04-03T07:18:03Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/archive-staging-rke2-swh
time="2024-04-03T07:18:03Z" level=warning msg="Failed to get cached managed resources for tree reconciliation, fall back to full reconciliation" application=argocd/archive-staging-rke2-swh dest-name= dest-namespace=swh dest-server="https://rancher.euwest.azure.internal.softwareheritage.org/k8s/clusters/c-m-9n5h9nrf" fields.level=0
time="2024-04-03T07:18:03Z" level=info msg="Comparing app state (cluster: https://rancher.euwest.azure.internal.softwareheritage.org/k8s/clusters/c-m-9n5h9nrf, namespace: swh)" application=argocd/archive-staging-rke2-swh
...
time="2024-04-03T07:18:03Z" level=info msg="Comparing app state (cluster: https://rancher.euwest.azure.internal.softwareheritage.org/k8s/clusters/c-m-682nvssh, namespace: argocd)" application=argocd/admin-rke2-argocd-applications
time="2024-04-03T07:18:03Z" level=warning msg="Failed to get cached managed resources for tree reconciliation, fall back to full reconciliation" application=argocd/swh-swh-next-version-377 dest-name= dest-namespace=swh-cassandra-next-version dest-server="https://rancher.euwest.azure.internal.softwareheritage.org/k8s/clusters/c-m-9n5h9nrf" fields.level=0
time="2024-04-03T07:18:03Z" level=info msg="Comparing app state (cluster: https://rancher.euwest.azure.internal.softwareheritage.org/k8s/clusters/c-m-9n5h9nrf, namespace: swh-cassandra-next-version)" application=argocd/swh-swh-next-version-377
time="2024-04-03T07:18:03Z" level=info msg="Normalized app spec: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2024-04-02T22:30:26Z\",\"message\":\"Failed to load live state: failed to get cluster info for \\\"https://rancher.euwest.azure.internal.softwareheritage.org/k8s/clusters/c-m-9n5h9nrf\\\": error synchronizing cache state : failed to sync cluster https://rancher.euwest.azure.internal.softwareheritage.org/k8s/clusters/c-m-9n5h9nrf: failed to load initial sta
time="2024-04-03T07:18:03Z" level=error msg="Failed to cache app resources: error getting resource tree: failed to get app hosts: error synchronizing cache state : failed to sync cluster https://rancher.euwest.azure.internal.softwareheritage.org/k8s/clusters/c-m-9n5h9nrf: failed to load initial state of resource Redis.redis.redis.opstreelabs.in: conversion webhook for redis.redis.opstreelabs.in/v1beta1, Kind=Redis failed: Post \"https://webhook-service.redis-operator.svc:44
time="2024-04-03T07:18:03Z" level=info msg="Skipping auto-sync: application status is Unknown" application=argocd/archive-staging-rke2-swh

added argocd kubernetes labels

assigned to @vsellier

First attempt, the argocd server cache was invalidated without success

The redis operator upgrade attempted yesterday failed due to a too long annotation on the CRD objects.

The CRDs were manually replaced with

kubectl --context archive-staging-rke2 replace -f redis.yaml
kubectl --context archive-staging-rke2 replace -f redis-cluster.yaml
...

It unstucked the arcgocd synchronization but didn't solved to redis application.

To allow the application sync, the server-side apply option needs to be set to true for it.

closed

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@1404fe08

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@9d1a9ddd