Using libcephsqlite to store objects
https://github.com/ceph/ceph/pull/39191/files
I don't know enough about the internals of SQLite to figure out if that could work. Having a single writer with EXCLUSIVE + WAL to minimize latency is not too much of a concern (the target is writing at least 100MB/s). But having hundreds of concurrent readers is a requirement: there are many daemons searching the content of the archive. The "yet" in the documentation[0] makes me think it is a next step and hopefully not too difficult.
It is conceptually similar to a collection of RBD images containing a sequence of objects (SHA256 + content). To allow for fast random reads, a rocksdb index located elsewhere (not necessarily in Ceph since it can be rebuilt if lost) is updated when an object is added, using SHA256 as a key, the identifier of the image that contains it and the offset at which it is to be found.
Migrated from T3050 (view on Phabricator)