Skip to content

Dead on arrival disk in the kafka3 server

One of the kafka3 disks has reported as failed by zfs.

The number of I/O errors associated with a ZFS device exceeded
acceptable levels. ZFS has marked the device as faulted.

 impact: Fault tolerance of the pool may be compromised.
    eid: 62
  class: statechange
  state: FAULTED
   host: kafka3
   time: 2020-08-26 13:32:07+0000
  vpath: /dev/disk/by-id/wwn-0x50000399f8982a7d-part1
  vphys: pci-0000:18:00.0-scsi-0:0:0:0
  vguid: 0xBD45E7F1FCC8E8CE
  devid: scsi-350000399f8982a7d-part1
   pool: 0x2030E25D0D8D754E

dmesg output:

Aug 26 13:30:03 kafka3 kernel: sd 0:0:0:0: [sda] tag#599 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=3s
Aug 26 13:30:03 kafka3 kernel: sd 0:0:0:0: [sda] tag#599 Sense Key : Hardware Error [deferred] [descriptor] 
Aug 26 13:30:03 kafka3 kernel: sd 0:0:0:0: [sda] tag#599 ASC=0x44 <<vendor>>ASCQ=0xa3 
Aug 26 13:30:03 kafka3 kernel: sd 0:0:0:0: [sda] tag#599 CDB: Write(16) 8a 00 00 00 00 00 0c 64 81 88 00 00 00 28 00 00
Aug 26 13:30:03 kafka3 kernel: blk_update_request: I/O error, dev sda, sector 207913352 op 0x1:(WRITE) flags 0x700 phys_seg 2 prio class 0
Aug 26 13:30:03 kafka3 kernel: sd 0:0:0:0: [sda] tag#598 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=3s
Aug 26 13:30:03 kafka3 kernel: zio pool=data vdev=/dev/disk/by-id/wwn-0x50000399f8982a7d-part1 error=5 type=2 offset=106450587648 size=20480 flags=40080c80
Aug 26 13:30:03 kafka3 kernel: sd 0:0:0:0: [sda] tag#598 Sense Key : Hardware Error [deferred] [descriptor] 
Aug 26 13:30:03 kafka3 kernel: sd 0:0:0:0: [sda] tag#598 ASC=0x44 <<vendor>>ASCQ=0xa3 
Aug 26 13:30:03 kafka3 kernel: sd 0:0:0:0: [sda] tag#598 CDB: Write(16) 8a 00 00 00 00 00 0c 64 81 b0 00 00 00 08 00 00
Aug 26 13:30:03 kafka3 kernel: blk_update_request: I/O error, dev sda, sector 207913392 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
Aug 26 13:30:03 kafka3 kernel: zio pool=data vdev=/dev/disk/by-id/wwn-0x50000399f8982a7d-part1 error=5 type=2 offset=106450608128 size=4096 flags=180880
Aug 26 13:31:36 kafka3 kernel: sd 0:0:0:0: Power-on or device reset occurred
Aug 26 13:31:51 kafka3 kernel: sd 0:0:0:0: [sda] tag#630 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s
Aug 26 13:31:51 kafka3 kernel: sd 0:0:0:0: [sda] tag#630 Sense Key : Hardware Error [deferred] 
Aug 26 13:31:51 kafka3 kernel: sd 0:0:0:0: [sda] tag#630 ASC=0x44 <<vendor>>ASCQ=0xa3 
Aug 26 13:31:51 kafka3 kernel: sd 0:0:0:0: [sda] tag#630 CDB: Write(16) 8a 00 00 00 00 00 04 cc b1 30 00 00 00 10 00 00
Aug 26 13:31:51 kafka3 kernel: blk_update_request: I/O error, dev sda, sector 80523568 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
Aug 26 13:31:51 kafka3 kernel: zio pool=data vdev=/dev/disk/by-id/wwn-0x50000399f8982a7d-part1 error=5 type=2 offset=41227018240 size=8192 flags=40080c80
Aug 26 13:32:03 kafka3 kernel: sd 0:0:0:0: [sda] tag#633 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=1s
Aug 26 13:32:03 kafka3 kernel: sd 0:0:0:0: [sda] tag#633 Sense Key : Hardware Error [deferred] 
Aug 26 13:32:03 kafka3 kernel: sd 0:0:0:0: [sda] tag#633 ASC=0x44 <<vendor>>ASCQ=0xa3 
Aug 26 13:32:03 kafka3 kernel: sd 0:0:0:0: [sda] tag#633 CDB: Write(16) 8a 00 00 00 00 00 08 8a 88 90 00 00 00 40 00 00
Aug 26 13:32:03 kafka3 kernel: blk_update_request: I/O error, dev sda, sector 143296656 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
Aug 26 13:32:03 kafka3 kernel: zio pool=data vdev=/dev/disk/by-id/wwn-0x50000399f8982a7d-part1 error=5 type=2 offset=73366839296 size=32768 flags=40080c80

Looks like the disk in the first SAS slot.

Migrated from T2545 (view on Phabricator)