Skip to content

Prepare the disks and configure zfs

The disk are configured with a 512k block size by default

root@cassandra03:/sys/block/nvme0n1/device# lsblk -o NAME,PHY-SeC | grep nvme
nvme0n1                          512
├─nvme0n1p1                      512
└─nvme0n1p9                      512
nvme2n1                          512
├─nvme2n1p1                      512
└─nvme2n1p9                      512
nvme1n1                          512
├─nvme1n1p1                      512
└─nvme1n1p9                      512
nvme4n1                          512
├─nvme4n1p1                      512
└─nvme4n1p9                      512
nvme3n1                          512
├─nvme3n1p1                      512
└─nvme3n1p9                      512

They will have better performance with a 4k block size:

root@cassandra03:/sys/block/nvme0n1/device# ls /dev/nvme?n1 | xargs -n1 -t nvme id-ns -H | grep LBA
nvme id-ns -H /dev/nvme0n1
  [3:0] : 0	Current LBA Format Selected
  [0:0] : 0x1	Metadata as Part of Extended Data LBA Supported
LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use)
LBA Format  1 : Metadata Size: 8   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good 
LBA Format  2 : Metadata Size: 16  bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good 
LBA Format  3 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0 Best 
LBA Format  4 : Metadata Size: 8   bytes - Data Size: 4096 bytes - Relative Performance: 0 Best 
LBA Format  5 : Metadata Size: 64  bytes - Data Size: 4096 bytes - Relative Performance: 0 Best 
LBA Format  6 : Metadata Size: 128 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best 
nvme id-ns -H /dev/nvme1n1
  [3:0] : 0	Current LBA Format Selected
  [0:0] : 0	Metadata as Part of Extended Data LBA Not Supported
LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use)
LBA Format  1 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0 Best 
nvme id-ns -H /dev/nvme2n1
  [3:0] : 0	Current LBA Format Selected
  [0:0] : 0	Metadata as Part of Extended Data LBA Not Supported
LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use)
LBA Format  1 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0 Best 
nvme id-ns -H /dev/nvme3n1
  [3:0] : 0	Current LBA Format Selected
  [0:0] : 0	Metadata as Part of Extended Data LBA Not Supported
LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use)
LBA Format  1 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0 Best 
nvme id-ns -H /dev/nvme4n1
  [3:0] : 0	Current LBA Format Selected
  [0:0] : 0	Metadata as Part of Extended Data LBA Not Supported
LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use)
LBA Format  1 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0 Best

They can be reformated with this command [1] :

nvme format --lbaf=<lba format id> /dev/<device>

Migrated from T4446 (view on Phabricator)

Edited by Vincent Sellier