Skip to content

buffer: add a threshold for the estimated size of revision and release batches

Nicolas Dandrimont requested to merge generated-differential-D6446-source into generated-differential-D6446-target

The size of individual revisions and releases is essentially unbounded. This means that, when the buffer storage is used as a way of limiting memory use for an ingestion process, it is still possible to go beyond the expected memory use when adding a batch of revisions or releases with large messages or other metadata.

The duration of the database operations for revision_add or release_add is also commensurate to the size of the objects added in a batch, so using the buffer proxy to limit the time individual database operations takes was not effective.

Adding a threshold on estimated sizes for batches of revision and release objects makes this overuse of memory and of database transaction time much less likely.

Depends on !732 (closed) Related to swh-loader-git#3625 (closed)

Test Plan

new tests added for the new thresholds

Migrated from D6446 (view on Phabricator)

Merge request reports