Skip to content

Make batch size configurable for the deduplication worker

Currently, the deduplication_worker uses a statically defined batch size

batch_size = (100000 if object_type in {SwhObjectType.DIRECTORY} else 10000)

This should be configurable

Edited by Rania Talbi