Skip to content
Snippets Groups Projects
New look: Off

Recycle belvedere to a new production rancher node

    • Closed Issue created by Vincent Sellier @vsellier

    Belvedere is not used anymore now the databases were moved to albertina.

    We can use the server as a compute node for the production cluster.

    • Perform a fresh bullseye install and name the server rancher-node-metal04 (cannot, no idrac)
    • Configure a zfs pool with a subset of disks(?)
    • Configure a swap dataset partition at least of the size of the memory (754Gi)
    • Add the server to the rancher cluster
    • Declare the labels to launch workload on it
    ✓ 5 of 5 checklist items completed · Edited

    Designs

    Child items ...

    • Activity

    • Vincent Sellier added environment: production hardware kubernetes labels

      added environment: production hardware kubernetes labels

    • Vincent Sellier changed title from Recycle belvedere as a new production ranche node to Recycle belvedere as a new production rancher node

      changed title from Recycle belvedere as a new production ranche node to Recycle belvedere as a new production rancher node

    • Vincent Sellier changed the description

      changed the description

    • Vincent Sellier changed title from Recycle belvedere as a new production rancher node to Recycle belvedere to a new production rancher node

      changed title from Recycle belvedere as a new production rancher node to Recycle belvedere to a new production rancher node

    • Guillaume Samson assigned to @guillaume

      assigned to @guillaume

    • Guillaume Samson
      Guillaume Samson @guillaume ·
      Owner

      The server cannot be reinstalled because it cannot be reached by iDRAC.

      • get and save the ZFS status (mirrors and disk names)
      • remove belvedere puppet TLS certs (/var/lib/puppet/ssl/*)
      • replace belvedere hostname by rancher-node-metal04 (/etc/hostname and hostnamectl)
      • remove ZFS pool data on belvedere
      • remove saam NFS mount
      • decommission belevdere on pergamon
      • clean up belvedere packages
      • create puppet configuration for rancher-node-metal04 (mirror and spare: see belvedere ZFS status backup)
      • run puppet agent on rancher-node-metal04
      • create swap on rancher-node-metal04 (SWAP size = RAM size)
      • check /tmp (ensure type is tmpfs and size = SWAP size)
      • reboot
      • update fw aliases
      • register as worker in archive-production-rke2 cluster
      • update inventory
      • add swh labels on the new node
      Edited by Guillaume Samson
    • Guillaume Samson
      Guillaume Samson @guillaume ·
      Owner

      belvedere ZFS status:

      root@belvedere:~# zpool status -v
  pool: data
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 1 days 04:19:14 with 5 errors on Thu Aug  3 04:43:18 2023
remove: Removal of vdev 5 copied 2.15T in 3h25m, completed on Sun Jan 15 20:30:48 2023
    279M memory used for removed device mappings
config:

        NAME                        STATE     READ WRITE CKSUM
        data                        DEGRADED     0     0     0
          mirror-0                  ONLINE       0     0     0
            wwn-0x50000397bc8969c1  ONLINE       0     0    23
            wwn-0x50000396dc8a2191  ONLINE       0     0    23
          mirror-1                  DEGRADED     0     0     0
            wwn-0x500003965c898904  ONLINE       0     0    17
            wwn-0x500003965c898910  UNAVAIL      9   525    17
          mirror-6                  ONLINE       0     0     0
            wwn-0x50000396dc8a2185  ONLINE       0     0    32
            wwn-0x50000397bc8969b1  ONLINE       0     0    31
          mirror-7                  ONLINE       0     0     0
            wwn-0x5002538c409a8258  ONLINE       0     0    18
            wwn-0x5002538c40ba6713  ONLINE       0     0    18
          mirror-8                  ONLINE       0     0     0
            wwn-0x5002538c409a825a  ONLINE       0     0    24
            wwn-0x5002538c40ba66ab  ONLINE       0     0    23
          mirror-9                  ONLINE       0     0     0
            wwn-0x5002538c409a8252  ONLINE       0     0    19
            wwn-0x5002538c409a8256  ONLINE       0     0    19
          mirror-10                 ONLINE       0     0     0
            wwn-0x5002538c40ba6716  ONLINE       0     0    18
            wwn-0x5002538c409a8253  ONLINE       0     0    18
          mirror-12                 ONLINE       0     0     0
            wwn-0x5002538b00657d80  ONLINE       0     0     0
            wwn-0x500003965c89895c  ONLINE       0     0     0
          mirror-13                 ONLINE       0     0     0
            wwn-0x500003972c8a44d1  ONLINE       0     0     0
            wwn-0x58ce38ee224a3ce1  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        /srv/softwareheritage/postgres/11/indexer/base/16406/488213705.106
        /srv/softwareheritage/postgres/11/indexer/base/16406/488213705.204
    • Guillaume Samson
      Guillaume Samson @guillaume ·
      Owner

      Remove puppet certs:

      root@belvedere:/var/lib/puppet# puppet agent --disable
root@belvedere:/var/lib/puppet# rm -r /var/lib/puppet/ssl
      Edited by Guillaume Samson
    • Guillaume Samson
      Guillaume Samson @guillaume ·
      Owner

      Replace belvedere hostname:

      root@belvedere:~# sed -i 's/belvedere/rancher-node-metal04/' /etc/hostname
root@belvedere:~# hostnamectl set-hostname $(cat /etc/hostname) 
root@belvedere:~# hostnamectl
   Static hostname: rancher-node-metal04.internal.softwareheritage.org
         Icon name: computer-server
           Chassis: server
        Machine ID: 314315373993426088e9fad67a1edbbb
           Boot ID: f715e99110b84137b71a3446cfd7ba75
  Operating System: Debian GNU/Linux 11 (bullseye)
            Kernel: Linux 5.10.0-21-amd64
      Architecture: x86-64
      gsamson@rancher-node-metal04 ~
 % sudo -i
sudo: unable to resolve host rancher-node-metal04.internal.softwareheritage.org: Name or service not known
[sudo] password for gsamson: 
root@rancher-node-metal04:~#

      Get belvedere IP address:

      ᐅ dig belvedere.internal.softwareheritage.org +noall +answer
belvedere.internal.softwareheritage.org. 43200 IN A 192.168.100.210
      Edited by Guillaume Samson
    • Guillaume Samson
      Guillaume Samson @guillaume ·
      Owner

      Remove zpool data on belvedere:

      root@rancher-node-metal04:~# zpool destroy data
    • Guillaume Samson
      Guillaume Samson @guillaume ·
      Owner

      Remove saam mount:

      root@rancher-node-metal04:~# umount /srv/remote-backups
root@rancher-node-metal04:/# sed -i '/^saam:/d' /etc/fstab
root@rancher-node-metal04:/# cat !$ 
cat /etc/fstab
# HEADER: This file was autogenerated at 2023-01-31 08:58:21 +0000
# HEADER: by puppet.  While it can still be managed manually, it
# HEADER: is definitely not recommended.
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
UUID=1a4beb50-a78d-4384-b7f6-801955e50c1e       /       ext4    errors=remount-ro       0       1
UUID=86B6-F861  /boot/efi       vfat    umask=0077      0       1

root@rancher-node-metal04:/# mount -a
    • Guillaume Samson
      Guillaume Samson @guillaume ·
      Owner

      Decommission belvedere:

      root@pergamon:~# /usr/local/sbin/swh-puppet-master-decommission belvedere.internal.softwareheritage.org
+ puppet node deactivate belvedere.internal.softwareheritage.org
Submitted 'deactivate node' for belvedere.internal.softwareheritage.org with UUID 299910e3-9718-400d-b70a-eee2cf5419f6
+ puppet node clean belvedere.internal.softwareheritage.org
Notice: Revoked certificate with serial 155
Notice: Removing file Puppet::SSL::Certificate belvedere.internal.softwareheritage.org at '/var/lib/puppet/ssl/ca/signed/belvedere.internal.softwareheritage.org.pem'
Notice: Removing file Puppet::SSL::Key belvedere.internal.softwareheritage.org at '/var/lib/puppet/ssl/private_keys/belvedere.internal.softwareheritage.org.pem'
belvedere.internal.softwareheritage.org
+ puppet cert clean belvedere.internal.softwareheritage.org
Warning: `puppet cert` is deprecated and will be removed in a future release.
   (location: /usr/lib/ruby/vendor_ruby/puppet/application.rb:370:in `run')
Notice: Revoked certificate with serial 155
+ systemctl restart apache2
root@pergamon:~# puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for pergamon.softwareheritage.org
Info: Applying configuration version '1706198862'
Info: Computing checksum on file /etc/ssh/puppet_known_hosts/belvedere.internal.softwareheritage.org.keys
Info: /Stage[main]/Profile::Ssh::Server/File[/etc/ssh/puppet_known_hosts/belvedere.internal.softwareheritage.org.keys]: Filebucketed /etc/ssh/puppet_known_hosts/belvedere.internal.softwareheritage.org.keys to puppet with sum ff2451f1aaf7ea3fcb206ab5637ecf4d
Notice: /Stage[main]/Profile::Ssh::Server/File[/etc/ssh/puppet_known_hosts/belvedere.internal.softwareheritage.org.keys]/ensure: removed
Info: /etc/ssh/puppet_known_hosts: Scheduling refresh of Exec[update ssh_known_hosts]
Notice: /Stage[main]/Profile::Ssh::Server/Exec[update ssh_known_hosts]: Triggered 'refresh' from 1 event
Info: Computing checksum on file /etc/icinga2/zones.d/master/belvedere.internal.softwareheritage.org.conf
Info: /Stage[main]/Profile::Icinga2::Master/File[/etc/icinga2/zones.d/master/belvedere.internal.softwareheritage.org.conf]: Filebucketed /etc/icinga2/zones.d/master/belvedere.internal.softwareheritage.org.conf to puppet with sum 8ae3b13bf171a486866e95e986400247
Notice: /Stage[main]/Profile::Icinga2::Master/File[/etc/icinga2/zones.d/master/belvedere.internal.softwareheritage.org.conf]/ensure: removed
Info: /etc/icinga2/zones.d/master: Scheduling refresh of Class[Icinga2::Service]
Info: Computing checksum on file /etc/prometheus/exported-configs/belvedere.internal.softwareheritage.org_node.yaml
Info: FileBucket got a duplicate file {md5}6a7c7921b53d9f3f9c5d6bb256fb50ba
Info: /Stage[main]/Profile::Prometheus::Server/File[/etc/prometheus/exported-configs/belvedere.internal.softwareheritage.org_node.yaml]: Filebucketed /etc/prometheus/exported-configs/belvedere.internal.softwareheritage.org_node.yaml to puppet with sum 6a7c7921b53d9f3f9c5d6bb256fb50ba
Notice: /Stage[main]/Profile::Prometheus::Server/File[/etc/prometheus/exported-configs/belvedere.internal.softwareheritage.org_node.yaml]/ensure: removed
Info: Computing checksum on file /etc/prometheus/exported-configs/belvedere.internal.softwareheritage.org_sql.yaml
Info: FileBucket got a duplicate file {md5}d846b867af2fbe744290e7980d994547
Info: /Stage[main]/Profile::Prometheus::Server/File[/etc/prometheus/exported-configs/belvedere.internal.softwareheritage.org_sql.yaml]: Filebucketed /etc/prometheus/exported-configs/belvedere.internal.softwareheritage.org_sql.yaml to puppet with sum d846b867af2fbe744290e7980d994547
Notice: /Stage[main]/Profile::Prometheus::Server/File[/etc/prometheus/exported-configs/belvedere.internal.softwareheritage.org_sql.yaml]/ensure: removed
Info: Computing checksum on file /etc/prometheus/exported-configs/belvedere.internal.softwareheritage.org_statsd.yaml
Info: FileBucket got a duplicate file {md5}4ed24c8e6a49c4072b7b1d1322b1ce94
Info: /Stage[main]/Profile::Prometheus::Server/File[/etc/prometheus/exported-configs/belvedere.internal.softwareheritage.org_statsd.yaml]: Filebucketed /etc/prometheus/exported-configs/belvedere.internal.softwareheritage.org_statsd.yaml to puppet with sum 4ed24c8e6a49c4072b7b1d1322b1ce94
Notice: /Stage[main]/Profile::Prometheus::Server/File[/etc/prometheus/exported-configs/belvedere.internal.softwareheritage.org_statsd.yaml]/ensure: removed
Info: /etc/prometheus/exported-configs: Scheduling refresh of Exec[update-prometheus-config]
Notice: /Stage[main]/Profile::Prometheus::Server/Exec[update-prometheus-config]: Triggered 'refresh' from 1 event
Info: /Stage[main]/Profile::Prometheus::Server/Exec[update-prometheus-config]: Scheduling refresh of Service[prometheus]
Notice: /Stage[main]/Profile::Prometheus::Server/Service[prometheus]: Triggered 'refresh' from 1 event
Info: Class[Icinga2::Service]: Scheduling refresh of Service[icinga2]
Notice: /Stage[main]/Icinga2::Service/Service[icinga2]: Triggered 'refresh' from 1 event
Notice: Applied catalog in 41.49 seconds
      Edited by Guillaume Samson
    • Guillaume Samson mentioned in commit swh/infra/puppet/puppet-swh-site@7455a25d

      mentioned in commit swh/infra/puppet/puppet-swh-site@7455a25d

    • Guillaume Samson mentioned in commit swh/infra/puppet/puppet-swh-site@0aa3b7ee

      mentioned in commit swh/infra/puppet/puppet-swh-site@0aa3b7ee

    • Guillaume Samson mentioned in commit swh/infra/puppet/puppet-swh-site@ce61f880

      mentioned in commit swh/infra/puppet/puppet-swh-site@ce61f880

    • Guillaume Samson mentioned in commit swh/infra/puppet/puppet-swh-site@ebd515e9

      mentioned in commit swh/infra/puppet/puppet-swh-site@ebd515e9

    • Guillaume Samson mentioned in commit swh/infra/puppet/puppet-swh-site@ddb364cc

      mentioned in commit swh/infra/puppet/puppet-swh-site@ddb364cc

    • Guillaume Samson
      Guillaume Samson @guillaume ·
      Owner

      Create ZFS pool data:

      root@rancher-node-metal04:~# zpool create -o ashift=12 -O atime=off -O relatime=on -O mountpoint=none -O compression=off data mirror wwn-0x5002538c409a8258 wwn-0x5002538c409a825a mirror wwn-0x5002538c409a8252 wwn-0x5002538b00657d80 mirror wwn-0x5002538c40ba6716 wwn-0x5002538c409a8253 mirror wwn-0x5002538c40ba6713 wwn-0x5002538c40ba66ab spare wwn-0x5002538c409a8256
root@rancher-node-metal04:~# zpool status -v
  pool: data
 state: ONLINE
config:

        NAME                        STATE     READ WRITE CKSUM
        data                        ONLINE       0     0     0
          mirror-0                  ONLINE       0     0     0
            wwn-0x5002538c409a8258  ONLINE       0     0     0
            wwn-0x5002538c409a825a  ONLINE       0     0     0
          mirror-1                  ONLINE       0     0     0
            wwn-0x5002538c409a8252  ONLINE       0     0     0
            wwn-0x5002538b00657d80  ONLINE       0     0     0
          mirror-2                  ONLINE       0     0     0
            wwn-0x5002538c40ba6716  ONLINE       0     0     0
            wwn-0x5002538c409a8253  ONLINE       0     0     0
          mirror-3                  ONLINE       0     0     0
            wwn-0x5002538c40ba6713  ONLINE       0     0     0
            wwn-0x5002538c40ba66ab  ONLINE       0     0     0
        spares
          wwn-0x5002538c409a8256    AVAIL

errors: No known data errors
    • Guillaume Samson marked the checklist item Configure a zfs pool with a subset of disks(?) as completed

      marked the checklist item Configure a zfs pool with a subset of disks(?) as completed

    • Guillaume Samson marked the checklist item Configure a swap dataset partition at least of the size of the memory (754Gi) as completed

      marked the checklist item Configure a swap dataset partition at least of the size of the memory (754Gi) as completed

    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    Please register or sign in to reply