[cassandra] Evaluate using a LCS compaction strategy for the directory_entry and content tables
The STS compaction strategy always needs 50% of free disk space of the largest table to work correctly.
This is not sustainable with our big directory_entry table.
We should evaluate to migrate to a LCS compaction algorithm (https://cassandra.apache.org/doc/latest/cassandra/operating/compaction/lcs.html)
It's not the best compaction strategy for our workload, but it could solve our main issue which is the size.
The LCS only needs 10x the defined sst table size.
It generate an additional io cost on compaction but should increase the read throughput.
The sstable size should also be tuned to avoid having to many sstables for this gigantic table.
the default size is 160mb max per sstable which should result to 65535 sstables
The compaction strategy can be changed dynamically and it will be apply during the next recompaction which allow to test without rebuilding the table
https://docs.datastax.com/en/cassandra-oss/planning/planning/ossCapacityPlanning.html
https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/operations/opsConfigureCompaction.html