Make batch size configurable for the deduplication worker
Currently, the deduplication_worker
uses a statically defined batch size
batch_size = (100000 if object_type in {SwhObjectType.DIRECTORY} else 10000)
This should be configurable
Edited by Rania Talbi