Make every Luigi task declare its memory usage
So they can be run on the same machine as the compression pipeline, with Luigi ensuring they never use more RAM than the machine allows
So they can be run on the same machine as the compression pipeline, with Luigi ensuring they never use more RAM than the machine allows