Check impact of special and cache VDEVs in production kafka cluster

When we set up kafka[1-4], we set up half of the machines with a special vdev (storing metadata for the pool), and half of the machines with a cache vdev (storing the L2 ARC).

It's not obvious whether these SSDs are useful, and we're looking to order new machines, so we should compare system load on the machines with the following configurations:

• special vdev (kafka1)
• no SSD vdev (kafka2)
• cache vdev (kafka3)
• no SSD vdev (kafka4)

added activity::MRO environment: production labels

assigned to @olasd

changed the description

On kafka2: zpool remove data mirror-4 removed the special VDEV.

Trying to remove the cache VDEV on kafka4 just completely froze i/o on the server; this is apparently a bug identified and fixed upstream (https://github.com/openzfs/zfs/pull/12054), so we won't try to do that anymore

indeed

lack of alerting for this situation is #5443 (closed)

Seems that we have no notable impact of these VDEVs, so we've ordered the new servers with no NVMe disks.

closed