next-version: Ensure kafka is not running on db1 & storage1

Follow up of [1]

db1 and storage1 are dedicated nodes with lots of disk space and a few pods running on them on purpose. But due to this, the kube scheduler keep on making run backends pods on those nodes which needs lots of resources.

The last service is kafka. Enforcing where to make those run should help in mitigating this issue. Let's do it.

[1] #5506 (closed)

Our cluster-components kafka chart has been adapted so we can put affinity on kafkaClusterNode so it runs on labelled node with swh/backend=true. This effectively excludes db1 and storage1 from the scheduling.

A bunch of other adaptations has been made to it too.

1. drop zookeeper from the charts (after migration to Kraft) [1]
2. migrate from zookeeper-ed cluster to Kraft (Kafka Raft metadata) [2]
3. metrics have been activated on the cluster

[1] strimzi operator mentioned it won't be supported by them nor by upstream.

[2] https://strimzi.io/docs/operators/latest/full/deploying.html#proc-deploy-migrate-kraft-str

mentioned in commit swh/infra/ci-cd/swh-charts@26231680

mentioned in commit swh/infra/ci-cd/swh-charts@83be6fa1

mentioned in commit swh/infra/ci-cd/swh-charts@172cf6cb

mentioned in issue #5515 (closed)

metrics have been activated on the cluster

But they are not scraped properly for now. [1] is empty where there should be data.

[1] http://thanos.internal.admin.swh.network:19191/graph?g0.expr=kafka_topic_partitions&g0.tab=1&g0.stacked=0&g0.range_input=1h&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D

mentioned in commit swh/infra/ci-cd/swh-charts@99987df6

mentioned in commit swh/infra/ci-cd/swh-charts@3ce8081f

And now we can see a wee bit more what's happening there too [1]

[1] https://grafana.softwareheritage.org/goto/GOgsVpIHz?orgId=1

kafka is no longer running on those nodes. They are dedicated to objstorages (and dbs for db1).

closed