Skip to content

Add an overlay storage

vlorentz requested to merge generated-differential-D7838-source into master

I will use it to locally test incremental loads of large repositories which are already loaded in production, with read-only access of the production storage.

This may also be used in the future to access storages while migrating/replicating.

Test Plan

This is fully tested when the backend storages are empty (by inheriting the usual tests); and never tested when they are not (for now)

I also tried loading linux.git with this config:

storage:
  cls: pipeline
  steps:
    - cls: buffer
    - cls: filter
    - cls: overlay
      storages:
        - cls: memory
        - cls: remote
          url: http://webapp.internal.staging.swh.network:5002/

and it didn't use more than 4GB of RAM! (and my machined almost OOMed without the filter proxy), so it seems to work

Migrated from D7838 (view on Phabricator)

Merge request reports