investigate libvirt I/O slowdown
There seems to be a 50% read speedup (1.5x) when accessing uffizi's disk array storage directly from louvre (bare metal server) w.r.t. accessing the same storage via uffizi (libvirt/kvm guest on louvre). The speed up is there for writes too, but is less significant. We need to find out whether that's "normal" and, if not, how to speed things up.
bonnie++ benchmarks for libvirt/kvm access (on a couple of shards):
zack@uffizi$ bonnie++ -d /srv/softwareheritage/objects/01/bonnie++
| Seq Out | | | Seq In | | Random | | Char | Block | Rewrite | Char | Block | Seeks | | K/s - CPU | K/s - CPU | K/s - CPU | K/s - CPU | K/s - CPU | /s - CPU | | 1378 - 98% | 755458 - 98% | 364821 - 58% | 2908 - 95% | 694353 - 42% | 257.7 - 36% | 6232us | 15146us | 390ms | 50966us | 338ms | 218ms | | Seq Create | | | Random Create | | | | Create | Read | Delete | Create | Read | Delete | | /sec - CPU | /sec - CPU | /sec - CPU | /sec - CPU | /sec - CPU | /sec - CPU | | 22208 - 79% | +++++ - +++% | 30949 - 88% | 23446 - 81% | +++++ - +++% | 29881 - 88% | 33782us | 130us | 195us | 22324us | 30us | 194us |
zack@uffizi$ bonnie++ -d /srv/softwareheritage/objects/a1/bonnie++
| Seq Out | | | Seq In | | Random | | Char | Block | Rewrite | Char | Block | Seeks | | K/s - CPU | K/s - CPU | K/s - CPU | K/s - CPU | K/s - CPU | /s - CPU | | 1351 - 98% | 686191 - 98% | 327914 - 54% | 3006 - 97% | 599862 - 45% | 204.9 - 44% | 9124us | 13713us | 302ms | 17296us | 329ms | 248ms | | Seq Create | | | Random Create | | | | Create | Read | Delete | Create | Read | Delete | | /sec - CPU | /sec - CPU | /sec - CPU | /sec - CPU | /sec - CPU | /sec - CPU | | 22364 - 80% | +++++ - +++% | 29042 - 88% | 21705 - 83% | +++++ - +++% | 25380 - 89% | 27162us | 210us | 224us | 19690us | 65us | 337us |
Same benchmarks accessing the same shards via louvre (bare metal):
zack@louvre$ mount|grep /mnt
/dev/mapper/vg--data-0 on /mnt type xfs (rw,relatime,attr2,inode64,noquota)
zack@louvre$ bonnie++ -d /mnt/bonnie
| Seq Out | | | Seq In | | Random | | Char | Block | Rewrite | Char | Block | Seeks | | K/s - CPU | K/s - CPU | K/s - CPU | K/s - CPU | K/s - CPU | /s - CPU | | 1404 - 99% | 633900 - 99% | 478570 - 75% | 2820 - 99% | 919112 - 91% | 345.1 - 27% | 7770us | 244ms | 359ms | 10967us | 513ms | 111ms | | Seq Create | | | Random Create | | | | Create | Read | Delete | Create | Read | Delete | | /sec - CPU | /sec - CPU | /sec - CPU | /sec - CPU | /sec - CPU | /sec - CPU | | 28184 - 88% | +++++ - +++% | +++++ - +++% | 28916 - 84% | +++++ - +++% | +++++ - +++% | 478us | 151us | 14701us | 444us | 15us | 43015us |
zack@louvre$ mount | grep /mnt
/dev/mapper/vg--data-a on /mnt type xfs (rw,relatime,attr2,inode64,noquota)
zack@louvre$ bonnie++ -d /mnt/bonnie
| Seq Out | | | Seq In | | Random | | Char | Block | Rewrite | Char | Block | Seeks | | K/s - CPU | K/s - CPU | K/s - CPU | K/s - CPU | K/s - CPU | /s - CPU | | 1317 - 99% | 655989 - 99% | 480398 - 75% | 2571 - 95% | 948253 - 92% | 350.6 - 25% | 9479us | 297ms | 10843ms | 38326us | 358ms | 136ms | | Seq Create | | | Random Create | | | | Create | Read | Delete | Create | Read | Delete | | /sec - CPU | /sec - CPU | /sec - CPU | /sec - CPU | /sec - CPU | /sec - CPU | | 28950 - 85% | +++++ - +++% | +++++ - +++% | 29147 - 88% | +++++ - +++% | +++++ - +++% | 478us | 112us | 13591us | 355us | 30us | 24317us |
Migrated from T283 (view on Phabricator)