Skip to content

staging: Autoscaling of the indexers not properly configured

Seen in the archive-staging cluster's events:

swh         0s          Warning   KEDAScalerFailed             scaledobject/indexer-origin-extrinsic-scaledobject            error creating kafka client: kafka: client has run out of available brokers to talk to: dial tcp: address journal1.internal.staging.swh.network: missing port in address
swh         0s          Warning   ScaledObjectCheckFailed      scaledobject/indexer-origin-extrinsic-scaledobject            Failed to ensure HPA is correctly created for ScaledObject
swh         0s          Warning   KEDAScalerFailed             scaledobject/indexer-content-mimetype-scaledobject            error creating kafka client: kafka: client has run out of available brokers to talk to: dial tcp: address journal1.internal.staging.swh.network: missing port in address
swh         0s          Warning   ScaledObjectCheckFailed      scaledobject/indexer-content-mimetype-scaledobject            Failed to ensure HPA is correctly created for ScaledObject
swh         0s          Warning   KEDAScalerFailed             scaledobject/indexer-content-fossology-license-scaledobject   error creating kafka client: kafka: client has run out of available brokers to talk to: dial tcp: address journal1.internal.staging.swh.network: missing port in address
swh         0s          Warning   ScaledObjectCheckFailed      scaledobject/indexer-content-fossology-license-scaledobject   Failed to ensure HPA is correctly created for ScaledObject
swh         0s          Warning   KEDAScalerFailed             scaledobject/indexer-origin-intrinsic-scaledobject            error creating kafka client: kafka: client has run out of available brokers to talk to: dial tcp: address journal1.internal.staging.swh.network: missing port in address
❯ kubectl get scaledobject --context archive-staging -n swh | grep -e "^NAME" -e indexer                                                                                                    10:09:58
NAME                                             SCALETARGETKIND      SCALETARGETNAME                     MIN   MAX   TRIGGERS   AUTHENTICATION                                 READY   ACTIVE    FALLBACK   AGE
indexer-content-fossology-license-scaledobject   apps/v1.Deployment   indexer-content-fossology-license   1     2     kafka      keda-storage-replayer-trigger-authentication   False   Unknown   Unknown    14d
indexer-content-mimetype-scaledobject            apps/v1.Deployment   indexer-content-mimetype            1     2     kafka      keda-storage-replayer-trigger-authentication   False   Unknown   Unknown    14d
indexer-origin-extrinsic-scaledobject            apps/v1.Deployment   indexer-origin-extrinsic            1     2     kafka      keda-storage-replayer-trigger-authentication   False   Unknown   Unknown    14d
indexer-origin-intrinsic-scaledobject            apps/v1.Deployment   indexer-origin-intrinsic            1     2     kafka      keda-storage-replayer-trigger-authentication   False   Unknown   Unknown    14d

^ Ready: false Active:Unkown

Apparently, keda allows to monitor it's status through PodMonitor objects, we need to check how to activate it and to alert on this

Related to #4540 (closed)


Migrated from T4621 (view on Phabricator)

Edited by Vincent Sellier