Skip to content
Snippets Groups Projects
New look: Off

production: Deploy new provenance grpc service

    • Closed Issue created by Antoine R. Dumont @ardumont

    Follow up of the staging deployment which went fine [1] A lot more work will be needed as we need to recycle the mam machine to a new highmem03 machine into the production kubernetes cluster.

    Plan:

    • Backup disk, network information (in case there is some issues during reinstall)
    • Update inventory about this machine
    • Backup any data interesting from mam (previous prototype machine) if any
    • mam: Disable puppet agent
    • pergamon: Decommission mam from puppet & disable alerts running for services on mam
    • ipxe!4 (closed): Reinstall mam machine (bookworm)
    • swh/infra/puppet/puppet-swh-site!733 (merged): puppet: Prepare mam recycling to rancher-node-highmem03
    • Integrate into production cluster (install step should do it ^, unstuck it if needed)
    • swh/infra/ci-cd/swh-charts!565 (merged): swh-charts: Adapt values to deploy new instance
    • (stopped) Let the datasets install itself (will take a while) ~> too slow from s3 (eta 16 days)
    • Install datasets from closer machine (maxxi)
    • Trigger smoke tests and monitor current grpc service behavior
    • #5624 (closed): Update to latest provenance and adapt deployment resource according to use
    • swh/infra/ci-cd/swh-charts!565 (merged) (2nd commit): swh-charts: Add webapp to use the new provenance api
    • swh/devel/swh-provenance!203 (merged): Fix provenance api client issue
    • swh/devel/swh-docs!478 (merged): docs: Update information on provenance

    Impacts:

    • #5209 (closed): Old Provenance indexes served by the server on mam will go down

    [1] #5608 (closed)

    [2] #4743 (closed): Ticket to install mam

    [3] https://inventory.internal.admin.swh.network/dcim/devices/225/

    ✓ 16 of 16 checklist items completed · Edited

    Designs

      Child items ...

      2. Activity

      • Antoine R. Dumont mentioned in issue #5608 (closed)

        mentioned in issue #5608 (closed)

      • Antoine R. Dumont changed the description

        changed the description

      • Antoine R. Dumont changed the description

        changed the description

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        Disk information on current mam [1] [2]

        [2] List of disks in the same order as the current zpool

        # Not in the zpool
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ21 -> ../../nvme0n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ26 -> ../../nvme1n1

# List of disk in the same zpool order [1]
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ23 -> ../../nvme2n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ24 -> ../../nvme3n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ35 -> ../../nvme4n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ30 -> ../../nvme5n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ33 -> ../../nvme6n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ34 -> ../../nvme7n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ22 -> ../../nvme8n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ25 -> ../../nvme9n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2W -> ../../nvme10n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2V -> ../../nvme11n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2E -> ../../nvme12n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2D -> ../../nvme13n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2C -> ../../nvme14n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2B -> ../../nvme15n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2Y -> ../../nvme16n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ32 -> ../../nvme17n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2X -> ../../nvme18n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ31 -> ../../nvme19n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ29 -> ../../nvme20n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ28 -> ../../nvme21n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2A -> ../../nvme22n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ27 -> ../../nvme23n1

        [1]

        root@mam:~# zfs get all data | grep "time\\|xattr\\|compression"
data  compression           zstd                   local
data  atime                 on                     default
data  xattr                 on                     default
data  relatime              off                    default
root@mam:~# zfs list | grep -v docker | grep -v rancher
NAME                                                                                    USED  AVAIL     REFER  MOUNTPOINT
data                                                                                   46.2T  26.5T     28.4T  /data
root@mam:~# zpool status
  pool: data
 state: ONLINE
  scan: scrub repaired 0B in 1 days 11:10:10 with 0 errors on Mon Mar 10 11:34:11 2025
config:

        NAME          STATE     READ WRITE CKSUM
        data          ONLINE       0     0     0
          raidz1-0    ONLINE       0     0     0
            nvme2n1   ONLINE       0     0     0
            nvme3n1   ONLINE       0     0     0
            nvme4n1   ONLINE       0     0     0
            nvme5n1   ONLINE       0     0     0
            nvme6n1   ONLINE       0     0     0
            nvme7n1   ONLINE       0     0     0
            nvme8n1   ONLINE       0     0     0
            nvme9n1   ONLINE       0     0     0
            nvme10n1  ONLINE       0     0     0
            nvme11n1  ONLINE       0     0     0
            nvme12n1  ONLINE       0     0     0
            nvme13n1  ONLINE       0     0     0
            nvme14n1  ONLINE       0     0     0
            nvme15n1  ONLINE       0     0     0
            nvme16n1  ONLINE       0     0     0
            nvme17n1  ONLINE       0     0     0
            nvme18n1  ONLINE       0     0     0
            nvme19n1  ONLINE       0     0     0
            nvme20n1  ONLINE       0     0     0
            nvme21n1  ONLINE       0     0     0
            nvme22n1  ONLINE       0     0     0
            nvme23n1  ONLINE       0     0     0

errors: No known data errors
root@mam:~# ls -lah /dev/disk/by-uuid/
total 0
drwxr-xr-x 2 root root 160 Feb  4 12:17 .
drwxr-xr-x 9 root root 180 Feb  4 12:17 ..
lrwxrwxrwx 1 root root  16 Feb  4 12:17 3057112486231962622 -> ../../nvme15n1p1
lrwxrwxrwx 1 root root  10 Feb  4 12:17 3465651f-4298-47ec-8b26-5f59e0530d57 -> ../../dm-1
lrwxrwxrwx 1 root root  15 Feb  4 12:17 49F2-30D2 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root  15 Feb  4 12:17 4A07-6A82 -> ../../nvme1n1p1
lrwxrwxrwx 1 root root  10 Feb  4 12:17 597bdd89-4d21-4691-938a-84d9aaee1e97 -> ../../dm-2
lrwxrwxrwx 1 root root  10 Feb  4 12:17 de7cf0e2-4fc9-4218-b42c-854bf1c4c924 -> ../../dm-0

root@mam:~# ls -lah /dev/disk/by-id/*nvme*V* | grep -v -- "-part"
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2V -> ../../nvme11n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2V_1 -> ../../nvme11n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2W -> ../../nvme10n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2W_1 -> ../../nvme10n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2X -> ../../nvme18n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2X_1 -> ../../nvme18n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2Y -> ../../nvme16n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2Y_1 -> ../../nvme16n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ30 -> ../../nvme5n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ30_1 -> ../../nvme5n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ31 -> ../../nvme19n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ31_1 -> ../../nvme19n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ32 -> ../../nvme17n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ32_1 -> ../../nvme17n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ33 -> ../../nvme6n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ33_1 -> ../../nvme6n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ34 -> ../../nvme7n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ34_1 -> ../../nvme7n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ35 -> ../../nvme4n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ35_1 -> ../../nvme4n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ21 -> ../../nvme0n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ21_1 -> ../../nvme0n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ22 -> ../../nvme8n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ22_1 -> ../../nvme8n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ23 -> ../../nvme2n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ23_1 -> ../../nvme2n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ24 -> ../../nvme3n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ24_1 -> ../../nvme3n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ25 -> ../../nvme9n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ25_1 -> ../../nvme9n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ26 -> ../../nvme1n1
lrwxrwxrwx 1 root root 13 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ26_1 -> ../../nvme1n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ27 -> ../../nvme23n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ27_1 -> ../../nvme23n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ28 -> ../../nvme21n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ28_1 -> ../../nvme21n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ29 -> ../../nvme20n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ29_1 -> ../../nvme20n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2A -> ../../nvme22n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2A_1 -> ../../nvme22n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2B -> ../../nvme15n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2B_1 -> ../../nvme15n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2C -> ../../nvme14n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2C_1 -> ../../nvme14n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2D -> ../../nvme13n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2D_1 -> ../../nvme13n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2E -> ../../nvme12n1
lrwxrwxrwx 1 root root 14 Feb  4 12:17 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2E_1 -> ../../nvme12n1
        Edited by Antoine R. Dumont
      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        Interesting current network setup:

        root@mam:~# ls /etc/systemd/network
bond0-interfaces.network  bond0.netdev  bond0.network  vlan440.netdev  vlan440.network
root@mam:~# cat /etc/systemd/network/bond0-interfaces.network
[Match]
Name=ens10f0np0
Name=ens10f1np1

[Network]
Bond=bond0
root@mam:~# cat /etc/systemd/network/bond0-interfaces.network
[Match]
Name=ens10f0np0
Name=ens10f1np1

[Network]
Bond=bond0
root@mam:~# cat /etc/systemd/network/bond0.netdev
[NetDev]
Name=bond0
Description=Public bonded interface
Kind=bond

[Bond]
Mode=802.3ad
TransmitHashPolicy=layer2+3
LACPTransmitRate=fast
MIIMonitorSec=100ms
DownDelaySec=200ms
root@mam:~# cat /etc/systemd/network/bond0.network
[Match]
Name=bond0
Type=bond

[Network]
VLAN=vlan440

LinkLocalAddressing=no
LLDP=no
EmitLLDP=no
IPv6AcceptRA=no
IPv6SendRA=no
root@mam:~# cat /etc/systemd/network/bond0.network
[Match]
Name=bond0
Type=bond

[Network]
VLAN=vlan440

LinkLocalAddressing=no
LLDP=no
EmitLLDP=no
IPv6AcceptRA=no
IPv6SendRA=no
root@mam:~# cat /etc/systemd/network/vlan440.netdev
[NetDev]
Name=vlan440
Description=Internal network vlan
Kind=vlan

[VLAN]
Id=440
root@mam:~# cat /etc/systemd/network/bond0.network
[Match]
Name=bond0
Type=bond

[Network]
VLAN=vlan440

LinkLocalAddressing=no
LLDP=no
EmitLLDP=no
IPv6AcceptRA=no
IPv6SendRA=no
root@mam:~# cat /etc/systemd/network/vlan440.netdev
[NetDev]
Name=vlan440
Description=Internal network vlan
Kind=vlan

[VLAN]
Id=440
root@mam:~# cat /etc/systemd/network/vlan440.network
[Match]
Name=vlan440
Type=vlan

[Network]
Description=Internal network
Address=192.168.100.111/24
Gateway=192.168.100.1
root@mam:~# ip route
default via 192.168.100.1 dev vlan440 proto static
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.20.0.0/16 dev br-3cfedb34f2fe proto kernel scope link src 172.20.0.1 linkdown
172.25.0.0/16 dev br-05e320ea0191 proto kernel scope link src 172.25.0.1 linkdown
192.168.100.0/24 dev vlan440 proto kernel scope link src 192.168.100.111
root@mam:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: ens10f0np0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 8e:57:2e:8c:0c:74 brd ff:ff:ff:ff:ff:ff permaddr 88:e9:a4:5a:d4:7c
    altname enp69s0f0np0
3: ens10f1np1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 8e:57:2e:8c:0c:74 brd ff:ff:ff:ff:ff:ff permaddr 88:e9:a4:5a:d4:7d
    altname enp69s0f1np1
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 8e:57:2e:8c:0c:74 brd ff:ff:ff:ff:ff:ff
5: vlan440@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 8e:57:2e:8c:0c:74 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.111/24 brd 192.168.100.255 scope global vlan440
       valid_lft forever preferred_lft forever
    inet6 fe80::8c57:2eff:fe8c:c74/64 scope link
       valid_lft forever preferred_lft forever
6: br-05e320ea0191: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:29:44:32:b2 brd ff:ff:ff:ff:ff:ff
    inet 172.25.0.1/16 brd 172.25.255.255 scope global br-05e320ea0191
       valid_lft forever preferred_lft forever
7: br-3cfedb34f2fe: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:40:68:8b:23 brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.1/16 brd 172.20.255.255 scope global br-3cfedb34f2fe
       valid_lft forever preferred_lft forever
8: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:d9:d7:35:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
      • Antoine R. Dumont changed the description

        changed the description

      • Antoine R. Dumont mentioned in commit swh/infra/puppet/puppet-swh-site@31332a2e

        mentioned in commit swh/infra/puppet/puppet-swh-site@31332a2e

      • Antoine R. Dumont mentioned in commit swh/infra/puppet/puppet-swh-site@48600d06

        mentioned in commit swh/infra/puppet/puppet-swh-site@48600d06

      • Antoine R. Dumont mentioned in commit swh/infra/puppet/puppet-swh-site@b779db35

        mentioned in commit swh/infra/puppet/puppet-swh-site@b779db35

      • Antoine R. Dumont mentioned in merge request swh/infra/puppet/puppet-swh-site!733 (merged)

        mentioned in merge request swh/infra/puppet/puppet-swh-site!733 (merged)

      • Antoine R. Dumont changed the description

        changed the description

      • Antoine R. Dumont mentioned in commit ipxe@66e452e2

        mentioned in commit ipxe@66e452e2

      • Antoine R. Dumont mentioned in merge request ipxe!4 (closed)

        mentioned in merge request ipxe!4 (closed)

      • Antoine R. Dumont changed the description

        changed the description

      • Antoine R. Dumont changed the description

        changed the description

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        [x] Backup any data interesting from mam (previous prototype machine)

        I've asked to relevant people (the ones using it) and they have nothing left there.

        So nothing to do.

        00:00:00 <*> Date changed to March 27, 2025
10:08:55 <ardumont> david: vlorentz: mam will get integrated into the kubernetes production cluster (as in it will get reinstalled), any data you want to backup? please let me know soon or write in https://gitlab.softwareheritage.org/swh/infra/sysadm-environment/-/issues/5620, tia
10:09:15 <ardumont> because it will then run the actual provenance grpc server ^
10:10:01 <ardumont> (i'm planning on doing this after lunch because before we've got the tech meeting)
15:36:28 <ardumont> david: vlorentz: I gather since there was no response back that all is fine and there is nothing to backup
15:40:49 <vlorentz> I got nothing left
15:42:49 <david> I did some retirieval yesterday (nothing critical, just in case), I'm ok now I believe (was about to add a comment in the issue but got caught by things)
        Edited by Antoine R. Dumont
      • Antoine R. Dumont changed the description

        changed the description

      • Antoine R. Dumont mentioned in commit swh/infra/ci-cd/swh-charts@fb86146b

        mentioned in commit swh/infra/ci-cd/swh-charts@fb86146b

      • Antoine R. Dumont mentioned in merge request swh/infra/ci-cd/swh-charts!565 (merged)

        mentioned in merge request swh/infra/ci-cd/swh-charts!565 (merged)

      • Antoine R. Dumont changed the description

        changed the description

      • Antoine R. Dumont mentioned in commit swh/infra/ci-cd/swh-charts@252e5e96

        mentioned in commit swh/infra/ci-cd/swh-charts@252e5e96

      • Antoine R. Dumont mentioned in commit swh/infra/puppet/puppet-swh-site@7dcb01d8

        mentioned in commit swh/infra/puppet/puppet-swh-site@7dcb01d8

      • Antoine R. Dumont mentioned in commit ipxe@9eaeaf24

        mentioned in commit ipxe@9eaeaf24

      • Antoine R. Dumont changed the description

        changed the description

      • Antoine R. Dumont mentioned in commit ipxe@334b341b

        mentioned in commit ipxe@334b341b

      • Antoine R. Dumont mentioned in commit ipxe@792473cb

        mentioned in commit ipxe@792473cb

      • Antoine R. Dumont changed the description

        changed the description

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        Disk installation got done manually to avoid losing too much data disk. Only 4Tb disk and it used to be a (raid1) disk for the system (which takes less than 100g of space). Now that space got sized for 100g.

        And the remaining space will get injected in the future zfs pool of data [1] \o/

        [1] Those partitions

        root@rancher-node-highmem03:~# fdisk -l /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ21 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ26 | grep "Linux filesystem"
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ21-part3 196362240 7501475839 7305113600  3.4T Linux filesystem
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ26-part3 196362240 7501475839 7305113600  3.4T Linux filesystem

        2025-03-28_11-37-33

        Edited by Antoine R. Dumont
      • Antoine R. Dumont mentioned in commit swh/infra/puppet/puppet-swh-site@0151126c

        mentioned in commit swh/infra/puppet/puppet-swh-site@0151126c

      • Antoine R. Dumont marked the checklist item swh/infra/puppet/puppet-swh-site!733 (merged): puppet: Prepare mam recycling to rancher-node-highmem03 as completed

        marked the checklist item swh/infra/puppet/puppet-swh-site!733 (merged): puppet: Prepare mam recycling to rancher-node-highmem03 as completed

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        [x] swh/infra/puppet/puppet-swh-site!733 (merged) (merged): puppet: Prepare mam recycling to rancher-node-highmem03 Task actions  Preparatory cleanup:

        root@rancher-node-highmem03:~# for disk in nvme-VO003840KXAVQ_SNABN5890I130BQ23 \
            nvme-VO003840KXAVQ_SNABN5890I130BQ24 \
            nvme-VO003840KXAVQ_SNABN5890I070BQ35 \
            nvme-VO003840KXAVQ_SNABN5890I070BQ30 \
            nvme-VO003840KXAVQ_SNABN5890I070BQ33 \
            nvme-VO003840KXAVQ_SNABN5890I070BQ34 \
            nvme-VO003840KXAVQ_SNABN5890I130BQ22 \
            nvme-VO003840KXAVQ_SNABN5890I130BQ25 \
            nvme-VO003840KXAVQ_SNABN5890I070BQ2W \
            nvme-VO003840KXAVQ_SNABN5890I070BQ2V \
            nvme-VO003840KXAVQ_SNABN5890I130BQ2E \
            nvme-VO003840KXAVQ_SNABN5890I130BQ2D \
            nvme-VO003840KXAVQ_SNABN5890I130BQ2C \
            nvme-VO003840KXAVQ_SNABN5890I130BQ2B \
            nvme-VO003840KXAVQ_SNABN5890I070BQ2Y \
            nvme-VO003840KXAVQ_SNABN5890I070BQ32 \
            nvme-VO003840KXAVQ_SNABN5890I070BQ2X \
            nvme-VO003840KXAVQ_SNABN5890I070BQ31 \
            nvme-VO003840KXAVQ_SNABN5890I130BQ29 \
            nvme-VO003840KXAVQ_SNABN5890I130BQ28 \
            nvme-VO003840KXAVQ_SNABN5890I130BQ2A \
            nvme-VO003840KXAVQ_SNABN5890I130BQ27 ; do \
  wipefs "/dev/disk/by-id/${disk}"
done
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I130BQ23 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ23 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ23 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I130BQ24 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ24 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ24 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I070BQ35 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ35 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ35 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I070BQ30 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ30 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ30 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I070BQ33 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ33 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ33 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I070BQ34 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ34 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ34 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I130BQ22 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ22 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ22 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I130BQ25 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ25 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ25 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I070BQ2W 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ2W 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ2W 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I070BQ2V 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ2V 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ2V 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I130BQ2E 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ2E 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ2E 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I130BQ2D 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ2D 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ2D 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I130BQ2C 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ2C 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ2C 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I130BQ2B 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ2B 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ2B 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I070BQ2Y 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ2Y 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ2Y 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I070BQ32 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ32 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ32 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I070BQ2X 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ2X 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ2X 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I070BQ31 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ31 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I070BQ31 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I130BQ29 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ29 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ29 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I130BQ28 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ28 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ28 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I130BQ2A 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ2A 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ2A 0x1fe         PMBR
DEVICE                               OFFSET        TYPE UUID LABEL
nvme-VO003840KXAVQ_SNABN5890I130BQ27 0x200         gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ27 0x37e3ee55e00 gpt
nvme-VO003840KXAVQ_SNABN5890I130BQ27 0x1fe         PMBR

        puppet is still unhappy

        root@rancher-node-highmem03:~# puppet agent --test
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Warning: The current total number of facts: 2417 exceeds the number of facts limit: 2048
Info: Caching catalog for rancher-node-highmem03.internal.softwareheritage.org
Info: Applying configuration version '1743165079'
Error: Execution of '/usr/sbin/zpool create data mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ23 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ28 mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ24 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ30 mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ22 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ33 mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ25 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ34 mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2V /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2D mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2E /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2X mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2W /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2C mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2Y /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2B mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ32 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2A mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ31 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ27 mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ29 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ35 mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ21-part3 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ26-part3' returned 1: invalid vdev specification
use '-f' to override the following errors:
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ23-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ28-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ24-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ30-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ22-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ33-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ25-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ34-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2V-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2D-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2E-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2X-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2W-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2C-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2Y-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2B-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ32-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2A-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ31-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ27-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ29-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ35-part1 is part of potentially active pool 'data'
Error: /Stage[main]/Profile::Zfs::Common/Zpool[data]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/sbin/zpool create data mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ23 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ28 mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ24 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ30 mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ22 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ33 mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ25 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ34 mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2V /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2D mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2E /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2X mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2W /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2C mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2Y /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2B mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ32 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2A mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ31 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ27 mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ29 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ35 mirror /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ21-part3 /dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ26-part3' returned 1: invalid vdev specification
use '-f' to override the following errors:
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ23-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ28-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ24-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ30-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ22-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ33-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ25-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ34-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2V-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2D-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2E-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2X-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2W-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2C-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ2Y-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2B-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ32-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ2A-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ31-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ27-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I130BQ29-part1 is part of potentially active pool 'data'
/dev/disk/by-id/nvme-VO003840KXAVQ_SNABN5890I070BQ35-part1 is part of potentially active pool 'data'
Notice: /Stage[main]/Profile::Zfs::Kubelet/Zfs[data/kubelet]: Dependency Zpool[data] has failures: true
Warning: /Stage[main]/Profile::Zfs::Kubelet/Zfs[data/kubelet]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Zfs::Rancher/Zfs[data/rancher]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Zfs::Rancher/Zfs[data/volumes]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Rancher/File[/var/lib/rancher]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Rancher/File[/var/lib/rancher/rke2]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Rancher/File[/var/lib/rancher/rke2/agent]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Rancher/File[/var/lib/rancher/rke2/agent/containerd]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Rancher/File[/var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.zfs]: Skipping because of failed dependencies
Notice: Applied catalog in 7.77 seconds

        so import the potentially activate data

        root@rancher-node-highmem03:~# zpool import
   pool: data
     id: 3057112486231962622
  state: ONLINE
status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
 config:

        data          ONLINE
          raidz1-0    ONLINE
            nvme2n1   ONLINE
            nvme3n1   ONLINE
            nvme4n1   ONLINE
            nvme5n1   ONLINE
            nvme6n1   ONLINE
            nvme7n1   ONLINE
            nvme8n1   ONLINE
            nvme9n1   ONLINE
            nvme10n1  ONLINE
            nvme11n1  ONLINE
            nvme12n1  ONLINE
            nvme13n1  ONLINE
            nvme14n1  ONLINE
            nvme15n1  ONLINE
            nvme16n1  ONLINE
            nvme17n1  ONLINE
            nvme18n1  ONLINE
            nvme19n1  ONLINE
            nvme20n1  ONLINE
            nvme21n1  ONLINE
            nvme22n1  ONLINE
            nvme23n1  ONLINE
root@rancher-node-highmem03:~# zpool status
no pools available
root@rancher-node-highmem03:~# zpool import data
cannot import 'data': pool was previously in use from another system.
Last accessed by mam (hostid=11e9a99e) at Fri Mar 28 09:26:33 2025
The pool can be imported, use 'zpool import -f' to import the pool.
root@rancher-node-highmem03:~# zpool import -f data
root@rancher-node-highmem03:~# zpool status
  pool: data
 state: ONLINE
  scan: scrub repaired 0B in 1 days 11:10:10 with 0 errors on Mon Mar 10 11:34:11 2025
config:

        NAME          STATE     READ WRITE CKSUM
        data          ONLINE       0     0     0
          raidz1-0    ONLINE       0     0     0
            nvme2n1   ONLINE       0     0     0
            nvme3n1   ONLINE       0     0     0
            nvme4n1   ONLINE       0     0     0
            nvme5n1   ONLINE       0     0     0
            nvme6n1   ONLINE       0     0     0
            nvme7n1   ONLINE       0     0     0
            nvme8n1   ONLINE       0     0     0
            nvme9n1   ONLINE       0     0     0
            nvme10n1  ONLINE       0     0     0
            nvme11n1  ONLINE       0     0     0
            nvme12n1  ONLINE       0     0     0
            nvme13n1  ONLINE       0     0     0
            nvme14n1  ONLINE       0     0     0
            nvme15n1  ONLINE       0     0     0
            nvme16n1  ONLINE       0     0     0
            nvme17n1  ONLINE       0     0     0
            nvme18n1  ONLINE       0     0     0
            nvme19n1  ONLINE       0     0     0
            nvme20n1  ONLINE       0     0     0
            nvme21n1  ONLINE       0     0     0
            nvme22n1  ONLINE       0     0     0
            nvme23n1  ONLINE       0     0     0

errors: No known data errors

        And destroy it.

        root@rancher-node-highmem03:~# zpool destroy data
root@rancher-node-highmem03:~# zpool status
no pools available

        It accepted to run now:

        root@rancher-node-highmem03:~# puppet agent --test
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Warning: The current total number of facts: 2417 exceeds the number of facts limit: 2048
Info: Caching catalog for rancher-node-highmem03.internal.softwareheritage.org
Info: Applying configuration version '1743165359'
Notice: /Stage[main]/Profile::Zfs::Common/Zpool[data]/ensure: created
Notice: /Stage[main]/Profile::Zfs::Kubelet/Zfs[data/kubelet]/ensure: created
Notice: /Stage[main]/Profile::Zfs::Rancher/Zfs[data/rancher]/ensure: created
Notice: /Stage[main]/Profile::Zfs::Rancher/Zfs[data/volumes]/ensure: created
Notice: /Stage[main]/Profile::Rancher/File[/var/lib/rancher/rke2]/ensure: created
Notice: /Stage[main]/Profile::Rancher/File[/var/lib/rancher/rke2/agent]/ensure: created
Notice: /Stage[main]/Profile::Rancher/File[/var/lib/rancher/rke2/agent/containerd]/ensure: created
Notice: /Stage[main]/Profile::Rancher/File[/var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.zfs]/ensure: created
Notice: Applied catalog in 12.39 seconds
        root@rancher-node-highmem03:~# zpool status
  pool: data
 state: ONLINE
config:

        NAME                                            STATE     READ WRITE CKSUM
        data                                            ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I130BQ23        ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I130BQ28        ONLINE       0     0     0
          mirror-1                                      ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I130BQ24        ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I070BQ30        ONLINE       0     0     0
          mirror-2                                      ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I130BQ22        ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I070BQ33        ONLINE       0     0     0
          mirror-3                                      ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I130BQ25        ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I070BQ34        ONLINE       0     0     0
          mirror-4                                      ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I070BQ2V        ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I130BQ2D        ONLINE       0     0     0
          mirror-5                                      ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I130BQ2E        ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I070BQ2X        ONLINE       0     0     0
          mirror-6                                      ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I070BQ2W        ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I130BQ2C        ONLINE       0     0     0
          mirror-7                                      ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I070BQ2Y        ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I130BQ2B        ONLINE       0     0     0
          mirror-8                                      ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I070BQ32        ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I130BQ2A        ONLINE       0     0     0
          mirror-9                                      ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I070BQ31        ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I130BQ27        ONLINE       0     0     0
          mirror-10                                     ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I130BQ29        ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I070BQ35        ONLINE       0     0     0
          mirror-11                                     ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I130BQ21-part3  ONLINE       0     0     0
            nvme-VO003840KXAVQ_SNABN5890I130BQ26-part3  ONLINE       0     0     0

errors: No known data errors

        Let's install some default options on dataset data:

        root@rancher-node-highmem03:~# zfs set compression=zstd data; zfs set atime=off data
root@rancher-node-highmem03:~# zfs get all data | grep "time\\|xattr\\|compression"
data  compression           off                    default
data  atime                 on                     default
data  xattr                 on                     default
data  relatime              off                    default
root@rancher-node-highmem03:~# zfs get all data | grep "time\\|xattr\\|compression"
data  compression           zstd                   local
data  atime                 off                    local
data  xattr                 on                     default
data  relatime              off                    default
      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        And we got the ~40T we wanted.

        root@rancher-node-highmem03:~# df -h | grep /data
data                                            42T  128K   42T   1% /data
      • Antoine R. Dumont mentioned in commit swh/infra/ci-cd/swh-charts@36c45a9a

        mentioned in commit swh/infra/ci-cd/swh-charts@36c45a9a

      • Antoine R. Dumont mentioned in commit swh/infra/ci-cd/swh-charts@885cf29d

        mentioned in commit swh/infra/ci-cd/swh-charts@885cf29d

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        [x] Integrate into production cluster (install step should do it ^, unstuck it if needed)

        Register node in rancher cluster with the right label directly [1] [2]

        [2]

        tony@kessel:~ $ swh-kubectl prd get nodes | grep highmem03
+ case "$1" in
+ context=archive-production-rke2
+ shift
+ kubectl --context archive-production-rke2 get nodes
rancher-node-highmem03               Ready    worker                      110s    v1.28.15+rke2r1

        [1]

        root@rancher-node-highmem03:~# url=https://rancher.euwest.azure.internal.softwareheritage.org
token=****
curl -fL "${url}/system-agent-install.sh" | \
  sudo sh -s - --server "${url}" --label 'cattle.io/os=linux' \
               --token "${token}" --worker \
               --label 'swh/provenance=true'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 32274    0 32274    0     0   316k      0 --:--:-- --:--:-- --:--:--  318k
[INFO]  Label: cattle.io/os=linux
[INFO]  Role requested: worker
[INFO]  Label: swh/provenance=true
[INFO]  Using default agent configuration directory /etc/rancher/agent
[INFO]  Using default agent var directory /var/lib/rancher/agent
[INFO]  Successfully tested Rancher connection
[INFO]  Downloading rancher-system-agent binary from https://rancher.euwest.azure.internal.softwareheritage.org/assets/rancher-system-agent-amd64
[INFO]  Successfully downloaded the rancher-system-agent binary.
[INFO]  Downloading rancher-system-agent-uninstall.sh script from https://rancher.euwest.azure.internal.softwareheritage.org/assets/system-agent-uninstall.sh
[INFO]  Successfully downloaded the rancher-system-agent-uninstall.sh script.
[INFO]  Generating Cattle ID
[INFO]  Successfully downloaded Rancher connection information
[INFO]  systemd: Creating service file
[INFO]  Creating environment file /etc/systemd/system/rancher-system-agent.env
[INFO]  Enabling rancher-system-agent.service
Created symlink /etc/systemd/system/multi-user.target.wants/rancher-system-agent.service → /etc/systemd/system/rancher-system-agent.service.
[INFO]  Starting/restarting rancher-system-agent.service
        Edited by Antoine R. Dumont
      • Antoine R. Dumont marked the checklist item Integrate into production cluster (install step should do it ^, unstuck it if needed) as completed

        marked the checklist item Integrate into production cluster (install step should do it ^, unstuck it if needed) as completed

      • Antoine R. Dumont mentioned in commit swh/infra/ci-cd/swh-charts@ed092e82

        mentioned in commit swh/infra/ci-cd/swh-charts@ed092e82

      • Antoine R. Dumont mentioned in commit swh/infra/ci-cd/swh-charts@515da347

        mentioned in commit swh/infra/ci-cd/swh-charts@515da347

      • Antoine R. Dumont mentioned in commit swh/infra/ci-cd/swh-charts@50286e1a

        mentioned in commit swh/infra/ci-cd/swh-charts@50286e1a

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        To match what's been configured on other machines, a new zfs dataset has been created and mounted on /srv/softwareheritage/ssd [1]

        It will received the persistent volume where the provenance dataset will be installed.

        I adapted accordingly the template (in the already opened mr) to unify with this [2] [3]

        [1]

        root@rancher-node-highmem03:~# mkdir -p /srv/softwareheritage/ssd
root@rancher-node-highmem03:~# zfs create -o mountpoint=/srv/softwareheritage/ssd data/data
root@rancher-node-highmem03:~# df -h /srv/softwareheritage/ssd
Filesystem      Size  Used Avail Use% Mounted on
data/data        42T  128K   42T   1% /srv/softwareheritage/ssd
root@rancher-node-highmem03:~# mkdir /srv/softwareheritage/ssd/provenance-20241206
root@rancher-node-highmem03:~# chown zack:zack /srv/softwareheritage/ssd/provenance-20241206  # zack is uid 1000 which is expected by the container

        [2] swh/infra/ci-cd/swh-charts!565 (merged)

        [3] swh/infra/ci-cd/swh-charts@551de5e6

        Edited by Antoine R. Dumont
      • Antoine R. Dumont mentioned in commit swh/infra/ci-cd/swh-charts@551de5e6

        mentioned in commit swh/infra/ci-cd/swh-charts@551de5e6

      • Antoine R. Dumont mentioned in commit swh/infra/ci-cd/swh-charts@86b4688f

        mentioned in commit swh/infra/ci-cd/swh-charts@86b4688f

      • Antoine R. Dumont mentioned in commit swh/infra/ci-cd/swh-charts@29e76d0c

        mentioned in commit swh/infra/ci-cd/swh-charts@29e76d0c

      • Antoine R. Dumont marked the checklist item swh/infra/ci-cd/swh-charts!565 (merged): swh-charts: Adapt values to deploy new instance as completed

        marked the checklist item swh/infra/ci-cd/swh-charts!565 (merged): swh-charts: Adapt values to deploy new instance as completed

      • Antoine R. Dumont changed the description

        changed the description

      • Antoine R. Dumont mentioned in commit ardumont/swh-docs@76e9d0f1

        mentioned in commit ardumont/swh-docs@76e9d0f1

      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      Please register or sign in to reply