Luigi: automatically set -Xmx based on dataset size
Currently, -Xmx
for each task is definedby a max_ram
task parameter, that is manually tweaked. As memory requirement of each task negligibly differs from the total number of nodes times a script-specific integer, the Luigi workflow should be able to tweak it automatically.
Edited by vlorentz