staging/kafka3: Install new bare metal kafka server

Installation procedure: https://docs.softwareheritage.org/sysadm/server-architecture/howto-install-new-physical-server.html

Inventory: https://inventory.internal.admin.swh.network/dcim/devices/290/

Environment: staging

Summary:

• Management address (DNS): 128.93.134.53
• VLAN configuration: VLAN443
• Internal IP: 192.168.130.202
• Public IP: 128.93.166.41
• Internal DNS name: kafka3.internal.staging.swh.network
• Public DNS name: kafka3.journal.staging.swh.network

Tasks:

• Declare the servers in the inventory
• Add the management info in the credential store (root and idrac/ilo access)
• Check Out-of-band access is ok
• Install the OS
• swh/infra/puppet/puppet-swh-site!724 (closed): Adapt puppet configuration
• Update firewall rules with the new machine's ip (e.g. swh_$environment_kube_workers, ...)
• Record the dns entry broker3.journal.staging.swh.network to resolve
• Request to the DSI to make this machine publicly reachable

added activity::Deployment label

changed the description

marked the checklist item Add the management info in the credential store (root and idrac/ilo access) as completed

mentioned in commit ipxe@1301aefa

Work has been started.

But currently the install fails to see the disk [1]

Time to take some rest. I'll double check this next week.

[1]

changed title from Install new bare metal kafka2.staging to staging: Install new bare metal kafka server

There may be a clash name with the kafka2 fqdn. So we need to determine the proper fqdn to use. @vsellier suggested kafka3 (but i fear that'd clash all the same with the kafka3 production node)

As another point, we need to make this node reachable from internet (for mirror use too).

changed title from staging: Install new bare metal kafka server to staging/kafka0: Install new bare metal kafka server

changed title from staging/kafka0: Install new bare metal kafka server to staging/kafka3: Install new bare metal kafka server

changed the description

mentioned in issue #5530 (closed)

marked the checklist item Install the OS as completed

[x] Install the os

Connecting through ssh from pergamon:

root@kafka3:~# uptime
 11:07:55 up 0 min,  1 user,  load average: 0.14, 0.05, 0.02
root@kafka3:~# uname -a
Linux kafka3 6.1.0-30-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.124-1 (2025-01-12) x86_64 GNU/Linux
root@kafka3:~# hostname -f
kafka3
root@kafka3:~# hostname
kafka3
root@kafka3:~# cat /etc/hosts
127.0.0.1       localhost
::1             localhost ip6-localhost ip6-loopback
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters

127.0.1.1 kafka3 kafka3.internal.staging.swh.network
root@kafka3:~# lsb_release -c
No LSB modules are available.
Codename:       bookworm

marked the checklist item Declare the servers in the inventory as completed

mentioned in commit ipxe@f0431025

mentioned in commit ipxe@e76a4584

I've made a basic run with puppet so we can connect to it (through its ip, puppet agent is disabled on pergamon).

Since pergamon got upgraded the certification coming from the puppet agent --test worked immediately.

• Puppet is off-limit for now (internal migration), deactivate puppet agent
• Add non-free source in debian to access the zfs-dkms dependency
• Install zfs-dkmk
• Identify disks to use as zpool [1]
• Create zpool with proper setup [2]
• Activate puppet agent
• swh/infra/puppet/puppet-swh-site!725 (merged): Adapt configuration to add the new kafka node in the cluster
• Let puppet install kafka (to use the proper mountpoint)

[1]

root@kafka3:~# ls -lah /dev/disk/by-id/
...
lrwxrwxrwx 1 root root  13 Jan 17 11:07 nvme-eui.37305230584026360025384e00000002 -> ../../nvme0n1
lrwxrwxrwx 1 root root  13 Jan 17 11:07 nvme-eui.37305230584026410025384e00000002 -> ../../nvme1n1
...

[2]

root@kafka3:~# zpool create \
  -o ashift=12 \
  -O atime=off \
  -O relatime=on \
  -O mountpoint=none\
  -O compression=lz4 \
  data \
  mirror \
  /dev/disk/by-id/nvme-eui.37305230584026*
root@kafka3:~# zpool status
  pool: data
 state: ONLINE
config:

        NAME                                           STATE     READ WRITE CKSUM
        data                                           ONLINE       0     0     0
          mirror-0                                     ONLINE       0     0     0
            nvme-eui.37305230584026360025384e00000002  ONLINE       0     0     0
            nvme-eui.37305230584026410025384e00000002  ONLINE       0     0     0

errors: No known data errors
root@kafka3:~# zfs list
NAME   USED  AVAIL     REFER  MOUNTPOINT
data   408K  1.68T       96K  none
root@kafka3:~# zfs create -o mountpoint=/srv/kafka data/kafka
root@kafka3:~# mount | grep /srv/kafka
data/kafka on /srv/kafka type zfs (rw,noatime,xattr,noacl)
root@kafka3:~# df -h /srv/kafka
Filesystem      Size  Used Avail Use% Mounted on
data/kafka      1.7T  128K  1.7T   1% /srv/kafka

changed the description