staging: Recycle beaubourg hardware into a staging rancher cluster worker node

changed milestone to %MRO 2023

changed the description

mentioned in commit swh/infra/puppet/puppet-swh-site@68692ad8

mentioned in merge request swh/infra/puppet/puppet-swh-site!694 (merged)

changed the description

Shutdown beaubourg (ensure idrac or serial console access first)

Currently facing issue to connect to the idrac or the serial console though. We need it because we need to stop the node prior to disconnect it to the proxmox cluster.

The sshuttle connection is ok [1] The dns resolution of the idrac too [2]

But the serial console access is not [3] nor the ui console returning anything [4]

[4]

This page isn’t working
128.93.134.8 didn’t send any data.
ERR_EMPTY_RESPONSE

[3]

$ ipmitool -I lanplus -H swh5-adm.inria.fr -U root -P  sol activate
IANA PEN registry open failed: No such file or directory
Error: Unable to establish IPMI v2 / RMCP+ session

[2]

$ host swh5-adm.inria.fr
swh5-adm.inria.fr has address 128.93.134.8

[1]

$ username=andumont; sudo SSH_AUTH_SOCK="$SSH_AUTH_SOCK" sshuttle --python python3 \
    --method tproxy \
    -r ${username}@sesi-ssh.inria.fr 128.93.162.142 128.93.134.0/26

[sudo] password for tony:
c : Connected to server.
...

I've filed a ticket in the helpdesk [1] (cc-ed swh-sysadm@inria.fr ml) to ask for a temporary workaround.

[1] https://support.inria.fr/SelfService/Display.html?id=297633

Analysis ongoing. Configuration is ok on their side.

Possibly an issue on hardware, check is ongoing, switch port is orange with a direct cable plugged in.

idrac access is up again (but i'm unclear whether the dsi is done or not yet).

changed the description

mentioned in commit swh/infra/puppet/puppet-swh-site@ae6c13b3

mentioned in commit swh/infra/puppet/puppet-swh-site@561b0f48

marked the checklist item swh/infra/puppet/puppet-swh-site!694 (merged): Decommission somerset (puppet, icinga alerts...) as completed

marked the checklist item Stop somerset container (e.g. from the proxmox ui) as completed

changed the description

mentioned in commit swh/infra/ci-cd/swh-charts@e0b43dbf

mentioned in commit swh/infra/ci-cd/swh-charts@ccbe27e8

mentioned in commit swh/infra/ci-cd/swh-charts@bdff418f

mentioned in commit swh/infra/ci-cd/swh-charts@4ed5b16e

mentioned in commit swh/infra/ci-cd/swh-charts@4c59d5c0

mentioned in commit swh/infra/ci-cd/swh-charts@29244f8f

marked the checklist item (blocked [1]) ensure idrac or serial console access (to reboot machine from a distance) as completed

changed the description

mentioned in issue #5260

marked this issue as related to #5260

marked the checklist item Shutdown beaubourg as completed

marked the checklist item Connect to another proxmox hypervisor (e.g. mucem) as completed

marked the checklist item mucem: remove beaubourg from proxmox cluster as completed

Shutdown beaubourg

Connect to another proxmox hypervisor (e.g. mucem)

mucem: remove beaubourg from proxmox cluster

# Beaubourg is up, it's referenced in the cluster nodes
root@mucem:~# pvecm nodes

Membership information
----------------------
    Nodeid      Votes Name
         1          1 branly
         2          1 beaubourg
         3          1 uffizi
         4          1 hypervisor3
         5          1 pompidou
         6          1 mucem (local)

# Stop beaubourg. Beaubourg is stopped, it's no longer referenced in the
# following output (beaubourg id 2 is not there)

root@mucem:~# pvecm nodes

Membership information
----------------------
    Nodeid      Votes Name
         1          1 branly
         3          1 uffizi
         4          1 hypervisor3
         5          1 pompidou
         6          1 mucem (local)
root@mucem:~# pvecm nodes

Membership information
----------------------
    Nodeid      Votes Name
         1          1 branly
         3          1 uffizi
         4          1 hypervisor3
         5          1 pompidou
         6          1 mucem (local)

# Dropping the node from the proxmox cluster
root@mucem:~# pvecm delnode beaubourg
Could not kill node (error = CS_ERR_NOT_EXIST)*
Killing node 2

# *At this point, it is possible that you will receive an error message
# stating Could not kill node (error = CS_ERR_NOT_EXIST). This does not
# signify an actual failure in the deletion of the node, but rather a failure
# in corosync trying to kill an offline node. Thus, it can be safely ignored.

root@mucem:~# pvecm status
Cluster information
-------------------
Name:             swh-inria-rocq
Config Version:   18
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Thu Feb 22 10:15:32 2024
Quorum provider:  corosync_votequorum
Nodes:            5
Node ID:          0x00000006
Ring ID:          1.3735
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   5
Highest expected: 5
Total votes:      5
Quorum:           3
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 192.168.100.108
0x00000003          1 192.168.100.101
0x00000004          1 192.168.100.34
0x00000005          1 192.168.100.170
0x00000006          1 192.168.100.33 (local)