Install the new bare metal server esnode + configure it so it's part of the production ELK cluster
Orders: https://mybox.inria.fr/smart-link/731dc7ca-4685-4517-98f8-03b5064710d8/
Inventory: https://inventory.internal.admin.swh.network/dcim/devices/212/
Summary:
- Management address (DNS): (swh-es7-adm.inria.fr)
- VLAN configuration: VLAN440
- Management Port: to be defined
- Access Ports: to be defined
- Internal IP(s): 192.168.100.64
- Internal DNS name(s): esnode7.internal.softwareheritage.org
- Location: C11 - 25
Tasks:
- Declare the servers in the inventory
- Add the management info in the credential store
- Install the OS
- install root password in credentials store
- Unstuck network issue
- Configure zfs partition for elasticsearch
- Add puppet configuration (if needed)
- Configure firewall rules (if needed)
Designs
- #4735Install new racks in Rocquencourt
Activity
changed milestone to %MRO 2023
added activity::Deployment hardware labels
marked this issue as related to #4735 (closed)
mentioned in issue #4755 (closed)
marked the checklist item Declare the servers in the inventory as completed
- Prepare the os iso for the machine
$ HOSTNAME=esnode7 $ cat ./configs/$HOSTNAME.yaml --- VLAN_ID: 440 IPADDRESS: 192.168.100.64 NETMASK: 255.255.255.0 GATEWAY: 192.168.100.1 NAMESERVER: 192.168.100.29 HOSTNAME: esnode7 DOMAINNAME: internal.softwareheritage.org DEPLOYMENT: production SUBNET: sesi_rocquencourt BOOT_DISK_ID_PATTERN: "*_Boot_Controller_*" $ ./configs/build_iso.sh $HOSTNAME make: Entering directory '/home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/src' [DEPS] image/embedded.c [BUILD] bin-x86_64-efi/embedded.o [AR] bin-x86_64-efi/blib.a ar: creating bin-x86_64-efi/blib.a [VERSION] bin-x86_64-efi/version.ipxe.efi.o [LD] bin-x86_64-efi/ipxe.efi.tmp [FINISH] bin-x86_64-efi/ipxe.efi [GENFSIMG] bin-x86_64-efi/ipxe.iso Makefile.efi:54: warning: pattern recipe did not update peer target 'bin-x86_64-efi/ipxe.usb'. rm bin-x86_64-efi/version.ipxe.efi.o bin-x86_64-efi/ipxe.efi make: Leaving directory '/home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/src' built iso image /home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/configs/esnode7.iso Generated preseeding config in /home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/configs/preseeding/esnode7.txt and /home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/configs/preseeding/finish_install/esnode7.sh. $ rsync -av --include="*/" --include="${HOSTNAME}.sh" --include="${HOSTNAME}.txt" \ --exclude="*" configs/preseeding/ \ pergamon.internal.softwareheritage.org:/srv/softwareheritage/preseeding/ $ ssh pergamon.internal.softwareheritage.org find /srv/softwareheritage/preseeding/ -type f -iname "\*${HOSTNAME}\*" /srv/softwareheritage/preseeding/finish_install/esnode7.sh /srv/softwareheritage/preseeding/esnode7.txtEdited by Antoine R. Dumont-
-
Start installation process
- activate serial console
- reset device
- mount iso generated
- connect to ipmi console [1]
- let the installation proceed
[1]
$ IPADDRESS=$(swhpass show infra/$HOSTNAME/idrac | awk -F/ '/^Url/{print $NF}') LOGIN=$(swhpass show infra/$HOSTNAME/idrac | awk '/^User/{print $2}') PASSWORD=$(swhpass show infra/$HOSTNAME/idrac | head -1) ipmitool -I lanplus -H "$IPADDRESS" -U "$LOGIN" -P "$PASSWORD" sol activate gpg: WARNING: server 'gpg-agent' is older than us (2.3.7 < 2.4.0) gpg: WARNING: server 'gpg-agent' is older than us (2.3.7 < 2.4.0) gpg: WARNING: server 'gpg-agent' is older than us (2.3.7 < 2.4.0) [SOL Session operational. Use ~? for help]Note: swhpass is a wrapper on my machine to use the swh creds store (as i have my own already configured with "pass" which predates the swh one, others can simply use "pass").
Edited by Antoine R. Dumont -
Start installation process
-
so far, it complains on the network interface setup [1] [2]:
│ │ │ Failed to run preseeded command │ │ Execution of preseeded command "anna-install net-modules-`uname -r` │ │ && modprobe bonding mode=802.3ad miimon=100 lacp_rate=slow │ │ xmit_hash_policy=layer3+4 && ip l set ens10f0np0 master bond0 && ip l │ │ set ens10f1np1 master bond0 && ip l add link bond0 name vlan440 type │ │ vlan id 440 && for iface in ens10f0np0 ens10f1np1 bond0 vlan440; do │ │ ip l set $iface up; done && ip a add dev vlan440 │ │ 192.168.100.64/255.255.255.0 && ip r add default via 192.168.100.1 && │ │ echo esnode7 > /etc/hostname && echo nameserver 192.168.100.29 > │ │ /etc/resolv.conf && sed -i -e '/ip link set/d' │ │ /bin/check-missing-firmware && (echo '#!/bin/sh'; echo 'exit 0') > │ │ /bin/netcfg && chmod +x /bin/netcfg && sleep 10" failed with exit │ │ code 2. │ │ │[2]
┌───────────────────┤ [!] Detect network hardware ├────────────────────┐ │ │ │ Some of your hardware needs non-free firmware files to operate. The │ │ firmware can be loaded from removable media, such as a USB stick or │ │ floppy. │ │ │ │ The missing firmware files are: │ │ qed/qed_init_values_zipped-8.42.2.0.bin │ │ qed/qed_init_values_zipped-8.42.2.0.bin │ │ │ │ If you have such media available now, insert it, and continue. │ │ │ │ Load missing firmware from removable media? │ │ │ │ <Yes> <No> │ │ │Edited by Antoine R. Dumont -
Fallback to a shell command within the installer (from serial console) to check the network interfaces. They seem to mismatch what we have in the preseed files:
~ # ip a 1: lo: <LOOPBACK> mtu 65536 qdisc noop qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: ens2f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000 link/ether 88:e9:a4:67:81:c0 brd ff:ff:ff:ff:ff:ff 3: ens2f1np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000 link/ether 88:e9:a4:67:81:c1 brd ff:ff:ff:ff:ff:ff 4: bond0: <BROADCAST,MULTICAST400> mtu 1500 qdisc noop qlen 1000 link/ether 56:5c:1c:ce:91:44 brd ff:ff:ff:ff:ff:ff -
I've adapted the preseeding to match those interfaces name and trigger back the install to check whether that unstuck the install.
root@pergamon:~# grep ens2f0np0 /srv/softwareheritage/preseeding/esnode7.txt ip l set ens2f0np0 master bond0 && \ for iface in ens2f0np0 ens2f1np1 bond0 vlan440; do ip l set $iface up; done && \ root@pergamon:~# grep ens2f1np1 /srv/softwareheritage/preseeding/esnode7.txt ip l set ens2f1np1 master bond0 && \ for iface in ens2f0np0 ens2f1np1 bond0 vlan440; do ip l set $iface up; done && \ -
Well, that did not raise the first issue on network interface setup so it helped. That still fails on the hardware detection though [2].
Those files would be part of a debian package (non-free) firmware-qlogic apparently.
[2]
┌───────────────────┤ [!] Detect network hardware ├────────────────────┐ │ │ │ Some of your hardware needs non-free firmware files to operate. The │ │ firmware can be loaded from removable media, such as a USB stick or │ │ floppy. │ │ │ │ The missing firmware files are: │ │ qed/qed_init_values_zipped-8.42.2.0.bin │ │ qed/qed_init_values_zipped-8.42.2.0.bin │ │ │ │ If you have such media available now, insert it, and continue. │ │ │ │ Load missing firmware from removable media? │ │ │ │ <Yes> <No> │ │ │Edited by Antoine R. Dumont -
It rebooted and login is fine.
But the network is down for now.
More on this later.
Edited by Antoine R. Dumont -
I've applied the changes ^.
That does not seem enough as there is no ip associated to those interfaces [1] (I have tried to use
networkctl reconfigureand a reboot too)[1]
root@esnode7:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens2f0np0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether e2:12:55:04:50:b5 brd ff:ff:ff:ff:ff:ff permaddr 88:e9:a4:67:81:c0 altname enp3s0f0np0 3: ens2f1np1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether e2:12:55:04:50:b5 brd ff:ff:ff:ff:ff:ff permaddr 88:e9:a4:67:81:c1 altname enp3s0f1np1 4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether e2:12:55:04:50:b5 brd ff:ff:ff:ff:ff:ff 5: vlan440@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether e2:12:55:04:50:b5 brd ff:ff:ff:ff:ff:ff inet6 fe80::e012:55ff:fe04:50b5/64 scope link valid_lft forever preferred_lft forever root@esnode7:~# cat /etc/systemd/network/bond0-interfaces.network [Match] Name=ens2f0np0 Name=ens2f1np1 [Network] Bond=bond0 root@esnode7:~# dmesg ... [ 5.299224] 8021q: 802.1Q VLAN Support v1.8 [ 5.360830] 8021q: adding VLAN 0 to HW filter on device ens2f0np0 [ 5.372608] 8021q: adding VLAN 0 to HW filter on device ens2f1np1 [ 5.405553] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond [ 5.405598] 8021q: adding VLAN 0 to HW filter on device bond0 [ 5.405684] IPv6: ADDRCONF(NETDEV_CHANGE): vlan440: link becomes ready [ 5.405765] IPv6: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready [ 5.428853] bond0: (slave ens2f1np1): link status definitely up, 25000 Mbps full duplex [ 5.428860] bond0: active interface up! ...Edited by Antoine R. Dumont -
root@esnode7:~# networkctl IDX LINK TYPE OPERATIONAL SETUP 1 lo loopback carrier unmanaged 2 ens2f0np0 ether enslaved configured 3 ens2f1np1 ether enslaved configured 4 bond0 bond carrier configured 5 vlan440 vlan degraded configuring 5 links listed. root@esnode7:~# networkctl status vlan440 ● 5: vlan440 > Link File: /usr/lib/systemd/network/99-default.link Network File: /etc/systemd/network/vlan440.network Type: vlan State: degraded (configuring) Driver: 802.1Q VLAN Support HW Address: e2:12:55:04:50:b5 MTU: 1500 (max: 65535) QDisc: noqueue IPv6 Address Generation Mode: eui64 VLan Id: 440 Queue Length (Tx/Rx): 1/1 Auto negotiation: no Speed: 50Gbps Duplex: full Address: fe80::e012:55ff:fe04:50b5 Gateway: 192.168.100.1 (ICANN, IANA Department) DHCP6 Client DUID: DUID-EN/Vendor:0000ab11ee9854ae655880550000 Mar 23 09:36:12 esnode7 systemd-networkd[587]: vlan440: netdev ready Mar 23 09:36:12 esnode7 systemd-networkd[587]: vlan440: Link UP Mar 23 09:36:12 esnode7 systemd-networkd[587]: vlan440: Gained carrier Mar 23 09:36:13 esnode7 systemd-networkd[587]: vlan440: Gained IPv6LLEdited by Antoine R. Dumont -
Nothing strikingly wrong...
root@esnode7:~# networkctl down vlan440 root@esnode7:~# networkctl up vlan440 root@esnode7:~# networkctl status vlan440 ● 5: vlan440 > Link File: /usr/lib/systemd/network/99-default.link Network File: /etc/systemd/network/vlan440.network Type: vlan State: degraded (configured) Driver: 802.1Q VLAN Support HW Address: e2:12:55:04:50:b5 MTU: 1500 (max: 65535) QDisc: noqueue IPv6 Address Generation Mode: eui64 VLan Id: 440 Queue Length (Tx/Rx): 1/1 Auto negotiation: no Speed: 50Gbps Duplex: full Address: fe80::e012:55ff:fe04:50b5 Gateway: 192.168.100.1 (ICANN, IANA Department) DHCP6 Client DUID: DUID-EN/Vendor:0000ab11ee9854ae655880550000 Mar 23 09:55:37 esnode7 systemd-networkd[587]: vlan440: DHCPv6 lease lost Mar 23 09:55:43 esnode7 systemd-networkd[587]: vlan440: Link UP Mar 23 09:55:43 esnode7 systemd-networkd[587]: vlan440: Gained carrier Mar 23 09:55:45 esnode7 systemd-networkd[587]: vlan440: Gained IPv6LL root@esnode7:~# journalctl -u systemd-networkd | tail -30 Mar 23 09:36:11 esnode7 systemd-networkd[587]: bond0: netdev ready Mar 23 09:36:11 esnode7 systemd-networkd[587]: Enumeration completed Mar 23 09:36:12 esnode7 systemd-networkd[587]: ens2f1np1: Link UP Mar 23 09:36:12 esnode7 systemd-networkd[587]: ens2f0np0: Link UP Mar 23 09:36:12 esnode7 systemd-networkd[587]: ens2f1np1: Gained carrier Mar 23 09:36:12 esnode7 systemd-networkd[587]: ens2f0np0: Gained carrier Mar 23 09:36:12 esnode7 systemd-networkd[587]: vlan440: netdev ready Mar 23 09:36:12 esnode7 systemd-networkd[587]: bond0: Link UP Mar 23 09:36:12 esnode7 systemd-networkd[587]: vlan440: Link UP Mar 23 09:36:12 esnode7 systemd-networkd[587]: vlan440: Gained carrier Mar 23 09:36:12 esnode7 systemd-networkd[587]: bond0: Gained carrier Mar 23 09:36:13 esnode7 systemd-networkd[587]: vlan440: Gained IPv6LL Mar 23 09:55:20 esnode7 systemd-networkd[587]: vlan440: Link DOWN Mar 23 09:55:20 esnode7 systemd-networkd[587]: vlan440: Lost carrier Mar 23 09:55:20 esnode7 systemd-networkd[587]: vlan440: DHCPv6 lease lost Mar 23 09:55:20 esnode7 systemd-networkd[587]: vlan440: Link UP Mar 23 09:55:20 esnode7 systemd-networkd[587]: vlan440: Gained carrier Mar 23 09:55:22 esnode7 systemd-networkd[587]: vlan440: Gained IPv6LL Mar 23 09:55:37 esnode7 systemd-networkd[587]: vlan440: Link DOWN Mar 23 09:55:37 esnode7 systemd-networkd[587]: vlan440: Lost carrier Mar 23 09:55:37 esnode7 systemd-networkd[587]: vlan440: DHCPv6 lease lost Mar 23 09:55:43 esnode7 systemd-networkd[587]: vlan440: Link UP Mar 23 09:55:43 esnode7 systemd-networkd[587]: vlan440: Gained carrier Mar 23 09:55:45 esnode7 systemd-networkd[587]: vlan440: Gained IPv6LL Mar 23 09:59:41 esnode7 systemd-networkd[587]: vlan440: Link DOWN Mar 23 09:59:41 esnode7 systemd-networkd[587]: vlan440: Lost carrier Mar 23 09:59:41 esnode7 systemd-networkd[587]: vlan440: DHCPv6 lease lost Mar 23 09:59:45 esnode7 systemd-networkd[587]: vlan440: Link UP Mar 23 09:59:45 esnode7 systemd-networkd[587]: vlan440: Gained carrier Mar 23 09:59:46 esnode7 systemd-networkd[587]: vlan440: Gained IPv6LL
mentioned in commit ardumont/swh-docs@7479f357
mentioned in merge request swh/devel/swh-docs!344 (merged)
mentioned in commit swh/devel/swh-docs@d3121435
mentioned in commit swh/devel/swh-docs@a459c781
mentioned in commit swh/infra/puppet/puppet-swh-site@f7f08ba4
mentioned in merge request swh/infra/puppet/puppet-swh-site!607 (merged)
mentioned in commit ipxe@10718f31
mentioned in commit swh/devel/swh-docs@ec2e1010
mentioned in commit swh/infra/puppet/puppet-swh-site@58530347
mentioned in commit swh/devel/swh-docs@6cab1c34
mentioned in commit swh/infra/puppet/puppet-swh-site@8235e32d
-
Configure zfs.
As the tools are not installed already (through puppet), i'll do the minimum zfs install needed so i can configure it properly.
root@esnode7:~# puppet agent --disable root@esnode7:~# systemctl stop elasticsearch root@esnode7:~# rm -rf /srv/elasticsearch/nodes # <- it started to write too much stuff there root@esnode7:~# cat /etc/apt/sources.list.d/debian.list # This file is managed by Puppet. DO NOT EDIT. # debian deb http://deb.debian.org/debian/ bullseye main contrib non-free root@esnode7:~# apt update ... root@esnode7:~# apt install zfs-dkms root@esnode7:~# /sbin/modprobe zfs ls -lah /dev/disk/by-id/scsi-350000f0072538* lrwxrwxrwx 1 root root 9 Mar 23 10:06 /dev/disk/by-id/scsi-350000f00725383e0 -> ../../sdb lrwxrwxrwx 1 root root 9 Mar 23 10:06 /dev/disk/by-id/scsi-350000f0072538600 -> ../../sda root@esnode7:~# zpool create \ -o ashift=12 \ -O atime=off \ -O relatime=on \ -O mountpoint=none \ -O compression=off \ elasticsearch-data \ /dev/disk/by-id/scsi-* root@esnode7:~# zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT elasticsearch 14.0T 488K 14.0T - - 0% 0% 1.00x ONLINE - root@esnode7:~# zfs create -o mountpoint=/srv/elasticsearch/nodes elasticsearch/data root@esnode7:~# zfs create -o mountpoint=/srv/elasticsearch/nodes elasticsearch/data root@esnode7:~# zfs list NAME USED AVAIL REFER MOUNTPOINT elasticsearch 612K 13.5T 96K none elasticsearch/data 96K 13.5T 96K /srv/elasticsearch/nodes -
Puppet applied and now the elasticsearch cluster is rebalancing data between nodes.
root@esnode7:~# while true; do date; df -h /srv/elasticsearch/nodes/; sleep 10; done Thu 23 Mar 2023 03:19:59 PM UTC Filesystem Size Used Avail Use% Mounted on elasticsearch/data 14T 25G 14T 1% /srv/elasticsearch/nodes Thu 23 Mar 2023 03:20:09 PM UTC Filesystem Size Used Avail Use% Mounted on elasticsearch/data 14T 26G 14T 1% /srv/elasticsearch/nodes Thu 23 Mar 2023 03:20:19 PM UTC Filesystem Size Used Avail Use% Mounted on elasticsearch/data 14T 28G 14T 1% /srv/elasticsearch/nodes assigned to @ardumont
-
Adding the extra node could have gone smoothlier.
Next time, deactivate puttet and apply the changes (add new node) gradually, waiting in between for the cluster to get back from orange to green prior to execute yet another puppet run (in another cluster node).
Here, what happens is that the puppet change got applied to the new node which went fine (it started receiving shards). And then puppet executed on the other nodes at around the same window period span (automatically), which made the cluster go red (as es restarted on all nodes but 1). It took some time to get back to green after that.
mentioned in commit swh/devel/swh-docs@7956a397
Please register or sign in to reply