Using an RBD image to store artifacts
The initial idea is described in this thread https://sympa.inria.fr/sympa/arc/swh-devel/2021-01/msg00026.html It is further discussed on the ceph user mailing list https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHQ5ZCHJISXIXOJSH3TU7DLYVYHRGTAT/ A well documented blog post with benchmarks and a billion objects dated 02/2020 https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond
"We've never managed 100TB+ in a single RBD volume. [...] Otherwise, yes RBD sounds very convenient for what you need."
Packing's obviously a good idea for storing these kinds of artifacts in Ceph, and hacking through the existing librbd might indeed be easier than building something up from raw RADOS, especially if you want to use stuff like rbd-mirror.
Migrated from T3014 (view on Phabricator)