debian-upgrade.rst

Upgrade Procedure for Debian Nodes in Kafka Cluster

Intended audience

sysadm staff members

Purpose

This page documents the steps to upgrade Debian nodes running in a Kafka cluster. The upgrade process involves various commands and checks before and after rebooting the node.

Prerequisites

• Familiarity with SSH and CLI-based command execution
• Out-of-band Access to the node (IDRAC/ILO) for reboot
• Access to the node through SSH (requires the vpn)

Step 0: Initial Steps

The kafka nodes are mostly running on bare metal machines. So, ensuring the out of band access to the machine is ok. This definitely helps when something goes wrong during a reboot (disk order or names change, network, ...).

Note: For (kubernetes) pods running kafka, this kind is dealt with automatically for kafka operator so this documentation does not cover this part.

Step 1: Migrate to the next debian suite

Update the Debian version of the node (e.g. bullseye to bookworm) using the following command:

root@node:~# /usr/local/bin/migrate-to-${NEXT_CODENAME}.sh

Note: The script should be present on the machine (installed through puppet).

Step 2: Run Puppet Agent

Once the upgrade procedure happened, run the puppet agent to apply any necessary configuration changes (e.g. /etc/apt/sources.list change, etc...)

root@node:~# puppet agent -t

Step 2: Stop Puppet Agent

As we will stop the service, we don't want the agent to start it back again.

root@node:~# puppet agent --disable "Ongoing debian upgrade"

Step 4: Autoremove and Purge

Perform autoremove to remove unnecessary packages left-over from the migration.

root@node:~# apt autoremove

Step 5: Stop the kafka service

The cluster can support one non-responding node so it's ok to stop the service (and let the node leave the cluster for maintenance operation).

We can check the cluster's status which should stay green after the elasticsearch service is stopped.

root@node:~# systemctl stop kafka
root@node:~# kcat -b $server

Note: $server if of the form hostname:9092 (with the hostname another cluster node than the one we are currently upgrading). kcat is a program from the kafkacat debian package.

Step 6: Reboot the node

We are ready to reboot the node:

root@node:~# reboot

You can connect to the serial console of the machine to follow through the reboot (and unstuck it if any problem arises).

Step 7: Clean up some more

Once the machine is restarted, some cleanup might be necessary.

root@node:~# apt autopurge

Step 8: Activate puppet agent

Activate back the puppet agent and make it run. This will start back the disabled service again.

root@node:~# puppet agent --enable && puppet agent --test

Step 8: Join back the cluster

After the service restarted, check the node joined back the cluster.

root@node:~# kcat -b $server

Example:

root@kafka1:~# kcat -L -b $server | head
Metadata for all topics (from broker -1: kafka1:9092/bootstrap):
2 brokers:
 broker 3 at kafka3.internal.staging.swh.network:9092
 broker 1 at journal2.internal.staging.swh.network:9092 (controller)