staging: Autoscaling of the indexers not properly configured
Seen in the archive-staging cluster's events:
swh 0s Warning KEDAScalerFailed scaledobject/indexer-origin-extrinsic-scaledobject error creating kafka client: kafka: client has run out of available brokers to talk to: dial tcp: address journal1.internal.staging.swh.network: missing port in address
swh 0s Warning ScaledObjectCheckFailed scaledobject/indexer-origin-extrinsic-scaledobject Failed to ensure HPA is correctly created for ScaledObject
swh 0s Warning KEDAScalerFailed scaledobject/indexer-content-mimetype-scaledobject error creating kafka client: kafka: client has run out of available brokers to talk to: dial tcp: address journal1.internal.staging.swh.network: missing port in address
swh 0s Warning ScaledObjectCheckFailed scaledobject/indexer-content-mimetype-scaledobject Failed to ensure HPA is correctly created for ScaledObject
swh 0s Warning KEDAScalerFailed scaledobject/indexer-content-fossology-license-scaledobject error creating kafka client: kafka: client has run out of available brokers to talk to: dial tcp: address journal1.internal.staging.swh.network: missing port in address
swh 0s Warning ScaledObjectCheckFailed scaledobject/indexer-content-fossology-license-scaledobject Failed to ensure HPA is correctly created for ScaledObject
swh 0s Warning KEDAScalerFailed scaledobject/indexer-origin-intrinsic-scaledobject error creating kafka client: kafka: client has run out of available brokers to talk to: dial tcp: address journal1.internal.staging.swh.network: missing port in address
❯ kubectl get scaledobject --context archive-staging -n swh | grep -e "^NAME" -e indexer 10:09:58
NAME SCALETARGETKIND SCALETARGETNAME MIN MAX TRIGGERS AUTHENTICATION READY ACTIVE FALLBACK AGE
indexer-content-fossology-license-scaledobject apps/v1.Deployment indexer-content-fossology-license 1 2 kafka keda-storage-replayer-trigger-authentication False Unknown Unknown 14d
indexer-content-mimetype-scaledobject apps/v1.Deployment indexer-content-mimetype 1 2 kafka keda-storage-replayer-trigger-authentication False Unknown Unknown 14d
indexer-origin-extrinsic-scaledobject apps/v1.Deployment indexer-origin-extrinsic 1 2 kafka keda-storage-replayer-trigger-authentication False Unknown Unknown 14d
indexer-origin-intrinsic-scaledobject apps/v1.Deployment indexer-origin-intrinsic 1 2 kafka keda-storage-replayer-trigger-authentication False Unknown Unknown 14d
^ Ready: false Active:Unkown
Apparently, keda allows to monitor it's status through PodMonitor objects, we need to check how to activate it and to alert on this
Related to #4540 (closed)
Migrated from T4621 (view on Phabricator)
Edited by Vincent Sellier