Factor out directory flattening from the revision processing pipeline
keep the directory flattening activated, but spawn it on (one or more) different machine(s), without waiting for it to terminate before moving to the next revision (writes different tables, that can be on a separate machine)