Install the new bare metal server(s) for Gateway
Orders: https://mybox.inria.fr/smart-link/e1a59e7d-96aa-4c35-ab24-13a7f2ea33a6/
Inventory:
- gateway1: https://inventory.internal.admin.swh.network/dcim/devices/141/
- gateway2: https://inventory.internal.admin.swh.network/dcim/devices/142/
Summary:
- gateway1
- Management address (DNS): 128.93.134.43 (swh-gw1-adm.inria.fr)
- VLAN configuration: VLAN440/VLAN442/VLAN443/VLAN1300 tagged
- Management port: to be defined
- Access ports: to be defined
- Internal IP(s): 192.168.50.4/192.168.100.4/192.168.130.4
- Internal DNS name(s): to be defined xxx.internal.(softwareheritage.org|.swh.network
- gateway2
- Management address (DNS): 128.93.134.44 (swh-gw2-adm.inria.fr)
- VLAN configuration: VLAN440/VLAN442/VLAN443/VLAN1300 tagged
- Management port: to be defined
- Access ports: to be defined
- Internal IP(s): 192.168.50.5/192.168.100.5/192.168.130.5
- Internal DNS name(s): to be defined xxx.internal.(softwareheritage.org|.swh.network
Tasks:
- Declare the servers in the inventory
- Add the management info in the credential store
- Install the OS
- Add puppet configuration (if needed)
- Configure firewall rules (if needed)
- <other actions if needed>
Designs
- #4735Install new racks in Rocquencourt
Activity
changed milestone to %MRO 2023
marked this issue as related to #4735 (closed)
marked the checklist item Declare the servers in the inventory as completed
mentioned in issue #4755 (closed)
I've been struggling with this for a while now.
For now I've focused on gateway1 (still to be named).
OPNSense doesn't have an obvious way to pxe boot (that I could find), so I've tried installing with virtual media. Let's say the experience has not been great.
I've tried:
- (KO) booting the latest OPNSense installer DVD iso, using the HTML5 virtual console
- https://mirror.vraphim.com/opnsense/releases/23.1/OPNsense-23.1-OpenSSL-dvd-amd64.iso.bz2
- this takes ages to load the kernel and initial modules, to eventually fail to mount the root filesystem with a bunch of USB timeouts on the hub of the virtual media.
- (KO) booting the latest OPNSense installer DVD iso, using the Java virtual console
- to use the java virtual console:
- change the ports for the virtual console and the virtual media to ports that aren't firewalled by the DSI (I picked 80 and 22, disabling HTTP and SSH ...)
- use an ancient JRE, using the scripts on https://github.com/markusdd/ilo-utils
- use sshuttle as normal
- this failed with ISO-related seek timeouts
- to use the java virtual console:
- (KO) booting the latest OPNSense USB installer disk image using the Java virtual console
- https://mirror.vraphim.com/opnsense/releases/23.1/OPNsense-23.1-OpenSSL-vga-amd64.iso.bz2
- also failed with different USB timeouts
- (??) booting a bare FreeBSD 13.1 mini-memstick image
- I don't remember if I tried this with java or with html5; but either way it didn't work, with the same usb timeouts as OPNSense.
- (OK) booting grml using the java virtual console
- this worked fine; I then tried:
- (OK) using grml to dump the OPNSense installer on disk
- set up vlan 440 on grml
- downloaded OPNSense usb installer image
- dumped the USB installer image directly on the NVMe drive. This booted fine, but I could not install from that (because the GPT partition table on the usb image doesn't have any free partition slots (!))
- manually replicated the partition scheme on the nvme boot disk, leaving space for the installed system
- EFI system partition
- nonsensical 1MB freebsd-boot partition
- nonsensical 1MB freebsd-swap partition
- UFS partition for installer (2 GB)
- free space
- manually dd'ed the partitions from the USB images into the nvme drive
- (??) installed OPNSense using the OPNSense installer dumped onto the NVMe drive
- the layout of the OPNSense installer's EFI system partition allows the HP UEFI to find the bootloader
- I didn't know there needed to be a label on the UFS partition for the installer, so the boot asks for a root partition; used
ufs:nvd0p4(4th partition of first nvme disk) - the installer boots!
- use the manual partitioning mode, and tell it to automatically partition the free space on the disk
- the installer completes
- rebooting doesn't find the root partition. Telling it to use
ufs:nvd0p5works, somehow.
On the booted system, the virtual keyboard of the HTML5 console breaks the USB driver, which makes the system uncontrollable.
This is all quite broken still...
TODO:
- setup serial console to come up by default to work around USB hub fuckery
- tell the installed system how to find its root device (it's looking for the label of the installer root partition)
- try setting up OPNSense by using its nano image (which allows installing by dumping an image to disk directly), from a successful boot of grml
- try a full freebsd memstick installer
- ask DSI to unfirewall the default remote console and virtual media ports
Edited by Nicolas Dandrimont- (KO) booting the latest OPNSense installer DVD iso, using the HTML5 virtual console
Thanks for this detailed report.
On the booted system, the virtual keyboard of the HTML5 console breaks the USB driver, which makes the system uncontrollable.
lol (or not)
And I didn't know grml, interesting
Edited by Vincent Sellier
-
- Ouverture des flux TCP depuis le bastion sesi-ssh.inria.fr vers les ports 17988 (virtual media) et 17990 (virtual console) sur toutes les adresses dans 128.93.134.0/25.
- Ouverture des flux TCP depuis le réseau 128.93.134.0/25 vers les ports 80 et 443 des reverse-proxies Software Heritage (128.93.166.8, 128.93.166.9, 128.93.166.10 et 128.93.166.14), afin de pouvoir (tenter de) monter des médias virtuels sans devoir rester attaché en console.
-
The adjustments have been applied.
Virtual media and virtual console from the Java app now work without changing the default ports.
Mounting virtual media over HTTP, in a IP-only fashion, now works. To work with HTTPS, we need a DNS recursive server unblocked and to configure the iLO to use it.