Install the new bare metal server(s) for cassandra staging cluster

Out of the configuration from the inventory [1], define the configuration [2] and build the iso [3]

[1] https://inventory.internal.admin.swh.network/dcim/devices/207/interfaces/

[2] cassandra1.yaml:

---
VLAN_ID: 443
IPADDRESS: 192.168.130.181
NETMASK: 255.255.255.0
GATEWAY: 192.168.130.1
NAMESERVER: 192.168.100.29
HOSTNAME: cassandra1
DOMAINNAME: internal.staging.swh.network
DEPLOYMENT: staging
SUBNET: sesi_rocquencourt_staging
BOOT_DISK_ID_PATTERN: "*_Boot_Controller_*"

[3]

$ cd ipxe/configs
$ ./build_iso.sh cassandra1
...
Makefile.efi:54: warning: pattern recipe did not update peer target 'bin-x86_64-efi/ipxe.usb'.
rm bin-x86_64-efi/version.ipxe.efi.o bin-x86_64-efi/ipxe.efi
make: Leaving directory '/home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/src'
built iso image /home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/configs/cassandra1.iso
Generated preseeding config in /home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/configs/preseeding/cassandra1.txt and /home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/configs/preseeding/finish_install/cassandra1.sh.

changed the description

Install the resulting generated files for cassandra1 [1] [2].

[1]

$ HOSTNAME=cassandra1
$ rsync -av --include="*/" --include="${HOSTNAME}.sh" --include="${HOSTNAME}.txt" \
--exclude="*" configs/preseeding/ \
pergamon.internal.softwareheritage.org:/srv/softwareheritage/preseeding/
sending incremental file list
rsync: failed to set times on "/srv/softwareheritage/preseeding/.": Operation not permitted (1)
./
rsync: failed to set times on "/srv/softwareheritage/preseeding/finish_install": Operation not permitted (1)
finish_install/
finish_install/cassandra1.sh

sent 1,931 bytes  received 259 bytes  4,380.00 bytes/sec
total size is 20,332  speedup is 9.28
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1333) [sender=3.2.3]

[2]

root@pergamon:/srv/softwareheritage/preseeding# find -type f -iname "cassandra1.*"
./finish_install/cassandra1.sh
./cassandra1.txt

Proceed with the installation (ongoing)

It's up [1]

[1] From serial console:

Debian GNU/Linux 11 cassandra1 ttyS0

cassandra1 login: root
Password:
Linux cassandra1 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@cassandra1:~# uptime
 10:49:03 up 1 min,  1 user,  load average: 0.25, 0.16, 0.06

changed the description

mentioned in commit swh/devel/swh-docs@918664db

mentioned in merge request swh/devel/swh-docs!339 (merged)

mentioned in commit ipxe@48f9de46

Create configuration for cassandra2 [1]
Generate iso including ipxe [2]
Install the necessary preseeding files [3]
Connect to its idrac
- Activate ipmi remote access
- install the iso as cd-dvd
- connect to serial console to check the installation is running
ensure the install is ok and we can log in [4]

[1] https://inventory.internal.admin.swh.network/dcim/devices/208/interfaces/

---
VLAN_ID: 443
IPADDRESS: 192.168.130.182
NETMASK: 255.255.255.0
GATEWAY: 192.168.130.1
NAMESERVER: 192.168.100.29
HOSTNAME: cassandra2
DOMAINNAME: internal.staging.swh.network
DEPLOYMENT: staging
SUBNET: sesi_rocquencourt_staging
BOOT_DISK_ID_PATTERN: "*_Boot_Controller_*"

[2]

./build_iso.sh cassandra2
make: Entering directory '/home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/src'
  [DEPS] image/embedded.c
  [BUILD] bin-x86_64-efi/embedded.o
  [VERSION] bin-x86_64-efi/version.ipxe.efi.o
  [AR] bin-x86_64-efi/blib.a
ar: creating bin-x86_64-efi/blib.a
  [LD] bin-x86_64-efi/ipxe.efi.tmp
  [FINISH] bin-x86_64-efi/ipxe.efi
  [GENFSIMG] bin-x86_64-efi/ipxe.iso
Makefile.efi:54: warning: pattern recipe did not update peer target 'bin-x86_64-efi/ipxe.usb'.
rm bin-x86_64-efi/version.ipxe.efi.o bin-x86_64-efi/ipxe.efi
make: Leaving directory '/home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/src'
built iso image /home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/configs/cassandra2.iso
Generated preseeding config in /home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/configs/preseeding/cassandra2.txt and /home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/configs/preseeding/finish_install/cassandra2.sh.

[3]

$ HOSTNAME=cassandra2
rsync -av --include="*/" --include="${HOSTNAME}.sh" --include="${HOSTNAME}.txt" \
--exclude="*" configs/preseeding/ \
pergamon.internal.softwareheritage.org:/srv/softwareheritage/preseeding/

sending incremental file list
rsync: failed to set times on "/srv/softwareheritage/preseeding/.": Operation not permitted (1)
./
cassandra2.txt
rsync: failed to set times on "/srv/softwareheritage/preseeding/finish_install": Operation not permitted (1)
finish_install/
finish_install/cassandra2.sh

sent 20,584 bytes  received 278 bytes  41,724.00 bytes/sec
total size is 20,332  speedup is 0.97
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1333) [sender=3.2.3]

$ ssh pergamon.internal.softwareheritage.org find /srv/softwareheritage/preseeding -type f -iname '\*cassandra2.\*'
/srv/softwareheritage/preseeding/cassandra2.txt
/srv/softwareheritage/preseeding/finish_install/cassandra2.sh

[4] with the root password (installed in credentials store now)

Debian GNU/Linux 11 cassandra2 ttyS0

cassandra2 login: root
Password:
Linux cassandra2 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@cassandra2:~# uptime
 13:42:57 up 3 min,  1 user,  load average: 0.08, 0.23, 0.12

Create configuration for cassandra3 [1]
Generate iso (which includes ipxe) [2]
Install the necessary preseeding files [3]
Connect to its idrac
- Activate ipmi remote access (reset server to take setup change into account)
- install the iso as cd-dvd
- connect to serial console to check the installation is running ok

[1] https://inventory.internal.admin.swh.network/dcim/devices/20/interfaces/

---
VLAN_ID: 443
IPADDRESS: 192.168.130.183
NETMASK: 255.255.255.0
GATEWAY: 192.168.130.1
NAMESERVER: 192.168.100.29
HOSTNAME: cassandra3
DOMAINNAME: internal.staging.swh.network
DEPLOYMENT: staging
SUBNET: sesi_rocquencourt_staging
BOOT_DISK_ID_PATTERN: "*_Boot_Controller_*"

[2]

$ ./build_iso.sh cassandra3
make: Entering directory '/home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/src'
  [DEPS] image/embedded.c
  [BUILD] bin-x86_64-efi/embedded.o
  [VERSION] bin-x86_64-efi/version.ipxe.efi.o
  [AR] bin-x86_64-efi/blib.a
ar: creating bin-x86_64-efi/blib.a
  [LD] bin-x86_64-efi/ipxe.efi.tmp
  [FINISH] bin-x86_64-efi/ipxe.efi
  [GENFSIMG] bin-x86_64-efi/ipxe.iso
Makefile.efi:54: warning: pattern recipe did not update peer target 'bin-x86_64-efi/ipxe.usb'.
rm bin-x86_64-efi/version.ipxe.efi.o bin-x86_64-efi/ipxe.efi
make: Leaving directory '/home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/src'
built iso image /home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/configs/cassandra3.iso
Generated preseeding config in /home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/configs/preseeding/cassandra3.txt and /home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/configs/preseeding/finish_install/cassandra3.sh.

[3] Install generated files and checks it's properly installed.

$ HOSTNAME=cassandra3
rsync -av --include="*/" --include="${HOSTNAME}.sh" --include="${HOSTNAME}.txt" \
--exclude="*" configs/preseeding/ \
pergamon.internal.softwareheritage.org:/srv/softwareheritage/preseeding/

$ ssh pergamon.internal.softwareheritage.org find /srv/softwareheritage/preseeding -type f -iname '\*cassandra3.\*'
/srv/softwareheritage/preseeding/cassandra3.txt
/srv/softwareheritage/preseeding/finish_install/cassandra3.sh

[4]

marked the checklist item cassandra2.staging as completed

mentioned in commit ipxe@a567893d

mentioned in commit ipxe@93427c5e

mentioned in commit ipxe@993f6166

connect to serial console to check the installation is running ok

It failed. First with a problem to access deb.debian.org. After that, it stuck on wanting to partition disk with inputs. No diffs between the previous install which went ok and this one beyong the initial failure. I need to find the proper way to drop the boot drive prior to reset the node and let the install run again [1].

[1]

15:25 <+ardumont> (staging) cass3 is less than happy, it does not pass beyond the partitioning step and seems to wait for some input
15:25  * ardumont opens his editor to check the cassandra3.txt
15:26 <+olasd> ardumont: try going to a console and wiping the boot drive (dd if=/dev/zero of=/dev/<whatever the boot drive is> bs=1M count=1024) before restarting the install
15:27 <+ardumont> nothing shockingly different between cass2.txt and cass3.txt
15:27 <+ardumont> thx for the hint!
15:29 <+olasd> the current preseeding config doesn't do a good job at wiping existing partitions, so when the install fails it doesn't really recover; We could probably add that to the partman early command

From the serial console, wipe the boot drive (to identify first) [1] It's not cleaned up in between installation step (because it's assumed no issue arise ;) As exchanged on irc, it's probably an improvment to add in the preseeding file.

[1]

# <C-a> 2 to get in the shell

~ # ls -l /dev/disk/by-id
lrwxrwxrwx    1 root     root            13 Mar 14 14:40 nvme-HPE_NS204i-p_Gen10
                                                                               0+_Boot_Controller_00000000000000000000 -> ../../nvme0n1
lrwxrwxrwx    1 root     root            15 Mar 14 14:40 nvme-HPE_NS204i-p_Gen10
                                                                               0+_Boot_Controller_00000000000000000000-part1 -> ../../nvme0n1p1
lrwxrwxrwx    1 root     root            15 Mar 14 14:40 nvme-HPE_NS204i-p_Gen10
                                                                               0+_Boot_Controller_00000000000000000000-part2 -> ../../nvme0n1p2
lrwxrwxrwx    1 root     root            15 Mar 14 14:40 nvme-HPE_NS204i-p_Gen10
                                                                               0+_Boot_Controller_00000000000000000000-part3 -> ../../nvme0n1p3
~ # dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=1024
~ # sync
~ # reboot

And from further help from olasd on irc (thx), the p1/p2/p3 partitions ^ were most likely the source of the confusion that made the debian installer stop.

changed the description

changed milestone to %MRO 2023

Install the new bare metal server(s) for cassandra staging cluster

Designs

Child items ...

Activity