Backend scripts don't support running on multiple nodes
Many start scripts don't support running on multiple nodes, even if the --nodelist
argument is passed.
For example, for the command
"${KAFKA_BIN_PATH}/kafka-broker-api-versions.sh" \
--bootstrap-server "${KAFKA_SLURM_NODELIST}:${KAFKA_BROKER_PORT}"
for now, the KAFKA_SLURM_NODELIST
comes from SLURM and is in a format like:
c[1262,1266]
which is not valid for most of the Kafka's --bootstrap-server
parameter,
so Kafka sees this literal string:
c[1262,1266]:9092
Kafka will fail to resolve this as a valid hostname.
We need to convert KAFKA_SLURM_NODELIST
into a comma-separated list of actual hostnames, like:
c1262:9092,c1266:9092
A possible fix to this can be
NODES=$(scontrol show hostname "$KAFKA_SLURM_NODELIST")
# Append port and join into comma-separated list
BROKER_LIST=$(printf "%s:${KAFKA_BROKER_PORT}," $NODES | sed 's/,$//')
...
"\"${KAFKA_BIN_PATH}/kafka-broker-api-versions.sh\" --bootstrap-server "$BROKER_LIST" >/dev/null 2>&1"