Skip to content

buffer: add a threshold for the number of revision parents in one batch

Nicolas Dandrimont requested to merge generated-differential-D6445-source into generated-differential-D6445-target

The size of individual revisions is essentially unbounded. This means that, when the buffer storage is used as a way of limiting memory use for an ingestion process, it is still possible to go beyond the expected memory use when adding a batch of revisions with extensive histories.

The duration of the database operation for revision_add is also commensurate to the number of revision parents added in a batch, so using the buffer proxy to limit the time individual database operations takes was not effective.

Adding a threshold on cumulated number of revision parents per batch makes this overuse of memory and of database transaction time much less likely.

Test Plan

new test added

Migrated from D6445 (view on Phabricator)

Merge request reports