Skip to content
Snippets Groups Projects
New look: Off

Install the new bare metal server(s) for cassandra production cluster

    • Closed Issue created by Vincent Sellier @vsellier

    Orders: https://mybox.inria.fr/smart-link/e5210831-06c7-4bcc-9a20-c448a6f908b2

    Inventory:

    Summary:

    • cassandra07:

      • Management address (DNS): 128.93.134.38 (swh-cas7-adm.inria.fr)
      • Management ports:
      • Access ports:
      • VLAN configuration: VLAN440 tagged
      • Internal IP(s): 192.168.100.187
      • Internal DNS name(s): cassandra07.internal.softwareheritage.org
      • Location: C11 U19

    • cassandra08:

      • Management address (DNS): 128.93.134.39 (swh-cas7-adm.inria.fr)
      • VLAN configuration: VLAN440 tagged
      • Internal IP(s): 192.168.100.188
      • Internal DNS name(s): cassandra08.internal.softwareheritage.org
      • Location: C11 U20

    Tasks:

    • Declare the servers in the inventory
    • Add the management info in the credential store
    • Install the OS
    • Add puppet configuration (if needed)
    • configure zfs [1]
    • Configure firewall rules (if needed)
    • <other actions if needed>

    zfs configuration for the cassandra0[1-6] servers: #4446 (closed)

    5 of 7 checklist items completed · Edited

    Designs

      Child items ...

      2. Activity

      • Vincent Sellier changed milestone to %Cassandra in production as primary storage [Roadmap - Tooling and infrastructure]

        changed milestone to %Cassandra in production as primary storage [Roadmap - Tooling and infrastructure]

      • Vincent Sellier added activity::Implementation cassandra labels

        added activity::Implementation cassandra labels

      • Vincent Sellier marked this issue as related to #4735 (closed)

        marked this issue as related to #4735 (closed)

      • Vincent Sellier changed the description

        changed the description

      • Vincent Sellier marked the checklist item Declare the servers in the inventory as completed

        marked the checklist item Declare the servers in the inventory as completed

      • Vincent Sellier changed the description

        changed the description

      • Vincent Sellier changed the description

        changed the description

      • Vincent Sellier mentioned in issue #4755 (closed)

        mentioned in issue #4755 (closed)

      • Vincent Sellier changed the description

        changed the description

      • Vincent Sellier changed the description

        changed the description

      • Vincent Sellier marked the checklist item Add the management info in the credential store as completed

        marked the checklist item Add the management info in the credential store as completed

      • Nicolas Dandrimont mentioned in commit swh/infra/puppet/puppet-swh-site@7349cb97

        mentioned in commit swh/infra/puppet/puppet-swh-site@7349cb97

      • Nicolas Dandrimont marked the checklist item Install the OS as completed

        marked the checklist item Install the OS as completed

      • Nicolas Dandrimont marked the checklist item Add puppet configuration (if needed) as completed

        marked the checklist item Add puppet configuration (if needed) as completed

      • Nicolas Dandrimont
        Nicolas Dandrimont @olasd ·
        Owner

        We've generalized the ipxe setup in https://gitlab.softwareheritage.org/swh/infra/ipxe.

        Using this setup, we've bootstrapped cassandra07 and cassandra08 up to the point puppet is running on them.

        However, the (minor) version of cassandra we're installing has been archived, so the puppet run fails to install it. The manifests need to be updated.

        zfs needs to be configured on these machines as well.

      • Vincent Sellier assigned to @vsellier

        assigned to @vsellier

      • Vincent Sellier added state:wip label

        added state:wip label

      • Vincent Sellier marked this issue as related to #4801 (closed)

        marked this issue as related to #4801 (closed)

      • Vincent Sellier marked this issue as related to #4800 (closed)

        marked this issue as related to #4800 (closed)

      • Vincent Sellier changed the description

        changed the description

      • Guillaume Samson
        Guillaume Samson @guillaume ·
        Owner

        ZFS was configured on cassandra07.

        Get the current disk's size block.

        root@cassandra07:~# lsblk -o NAME,PHY-SeC | grep nvme[1-3]
nvme3n1                        512                                                             
nvme1n1                        512                                                             
nvme2n1                        512

        Get the available formats.

        root@cassandra07:~# ls /dev/nvme[1-3]n1 | xargs -n1 -t nvme id-ns -H | grep LBA
nvme id-ns -H /dev/nvme1n1                                                                                                                                                                    
  [4:4] : 0x1   Metadata Transferred at End of Data LBA
  [3:0] : 0     Current LBA Format Selected
  [0:0] : 0x1   Metadata as Part of Extended Data LBA Supported
LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x1 Better (in use)
LBA Format  1 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0 Best 
LBA Format  2 : Metadata Size: 8   bytes - Data Size: 512 bytes - Relative Performance: 0x3 Degraded
LBA Format  3 : Metadata Size: 8   bytes - Data Size: 4096 bytes - Relative Performance: 0x2 Good
LBA Format  4 : Metadata Size: 64  bytes - Data Size: 4096 bytes - Relative Performance: 0x3 Degraded
nvme id-ns -H /dev/nvme2n1
  [4:4] : 0x1   Metadata Transferred at End of Data LBA
  [3:0] : 0     Current LBA Format Selected
  [0:0] : 0x1   Metadata as Part of Extended Data LBA Supported
LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x1 Better (in use)
LBA Format  1 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0 Best 
LBA Format  2 : Metadata Size: 8   bytes - Data Size: 512 bytes - Relative Performance: 0x3 Degraded
LBA Format  3 : Metadata Size: 8   bytes - Data Size: 4096 bytes - Relative Performance: 0x2 Good
LBA Format  4 : Metadata Size: 64  bytes - Data Size: 4096 bytes - Relative Performance: 0x3 Degraded
nvme id-ns -H /dev/nvme3n1
  [3:0] : 0     Current LBA Format Selected
  [0:0] : 0x1   Metadata as Part of Extended Data LBA Supported
LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use)
LBA Format  1 : Metadata Size: 8   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good
LBA Format  2 : Metadata Size: 16  bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good
LBA Format  3 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0 Best 
LBA Format  4 : Metadata Size: 8   bytes - Data Size: 4096 bytes - Relative Performance: 0 Best 
LBA Format  5 : Metadata Size: 64  bytes - Data Size: 4096 bytes - Relative Performance: 0 Best 
LBA Format  6 : Metadata Size: 128 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best

        Format the disks which don't generate errors.

        root@cassandra07:~# nvme format -f --lbaf=1 /dev/nvme1n1
Success formatting namespace:1
root@cassandra07:~# nvme format -f --lbaf=1 /dev/nvme2n1
Success formatting namespace:1

        Try to format the last disk (writeintensive) : command nerver return, then kill have no effect, then cold reboot from iLO.

        root@cassandra07:~# nvme format -f --lbaf=5 /dev/nvme3n1
NVMe status: INVALID_FORMAT: The LBA Format specified is not supported. This may be due to various conditions(0x410a)
root@cassandra07:~# nvme format -f --lbaf=4 /dev/nvme3n1
NVMe status: INVALID_FORMAT: The LBA Format specified is not supported. This may be due to various conditions(0x410a)
root@cassandra07:~# nvme format -f --lbaf=3 /dev/nvme3n1

        Create the pool mixeduse.

        root@cassandra07:~# zpool create -o ashift=12 -O atime=off -O relatime=on -O mountpoint=none -O compression=off mixeduse /dev/disk/by-id/nvme-MZXLR6T4HALA-000H3_S6C1NA0T700736 /dev/disk/by-i
d/nvme-MZXLR6T4HALA-000H3_S6C1NA0T803294

        Check the mixeduse pool options.

        root@cassandra07:~# zfs get all mixeduse | grep "atime\|relatime\|compression"
mixeduse  compression           off                    local
mixeduse  atime                 off                    local
mixeduse  relatime              on                     local

        Create the writeintensive pool.

        root@cassandra07:~# zpool create -o ashift=12 -O atime=off -O relatime=on -O mountpoint=none -O compression=off writeintensive nvme-EO000750KWTXC_PHKE1254000B750BGN

        Check the pool list.

        root@cassandra07:~# zpool list
NAME             SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
mixeduse        11.6T   408K  11.6T        -         -     0%     0%  1.00x    ONLINE  -
writeintensive   696G   464K   696G        -         -     0%     0%  1.00x    ONLINE  -

        Create the ZFS filesystem and the mountpoint.

        root@cassandra07:~# zfs create -o mountpoint=/srv/cassandra/instance1/data mixeduse/cassandra-data
root@cassandra07:~# zfs create -o mountpoint=/srv/cassandra/instance1/commitlog writeintensive/commitlog
root@cassandra07:~# zfs list
NAME                       USED  AVAIL     REFER  MOUNTPOINT
mixeduse                   612K  11.3T       96K  none
mixeduse/cassandra-data     96K  11.3T       96K  /srv/cassandra/instance1/data
writeintensive             576K   674G       96K  none
writeintensive/commitlog    96K   674G       96K  /srv/cassandra/instance1
        Edited by Guillaume Samson
      • Vincent Sellier mentioned in commit swh/infra/ci-cd/swh-charts@269233e3

        mentioned in commit swh/infra/ci-cd/swh-charts@269233e3

      • Vincent Sellier mentioned in commit swh/infra/ci-cd/swh-charts@2a6f4e47

        mentioned in commit swh/infra/ci-cd/swh-charts@2a6f4e47

      • Vincent Sellier marked the checklist item configure zfs [1] as completed

        marked the checklist item configure zfs [1] as completed

      • Vincent Sellier
        Vincent Sellier @vsellier ·
        Author Owner

        The installation of the nodes is done.

        The remaining task are related to cassandra (#4801 (closed)) not to the server installation

      • Vincent Sellier closed

        closed

      Please register or sign in to reply