swh.storage.filter: Add filtering storage implementation

Also add a sample_data fixture to read default test data from.

Depends on !838 (closed)

Test Plan

tox

---

Migrated from D2084 (view on Phabricator)

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/667/ See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/667/console

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/668/ See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/668/console

I don't think this is useful anymore. The postgresql storage does the filtering itself now (see !251 (closed)). Are you planning on removing this feature from the postgresql storage?

I don't think this is useful anymore. The postgresql storage does the filtering itself now (see !251 (closed)). Are you planning on removing this feature from the postgresql storage?

It always did that. But it does it server-side so we send lots of data on the wire (which has a cost).

It's for doing it client side as the current loaders do but it's entangled within its own loading logic (check swh.loader.core.loader.BufferedLoader).

This allows to untangle that part configuration wise.

Maybe the issue is with where i've put the modules. Maybe elsewhere swh.storage.client (or something) would be clearer...

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/671/ See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/671/console

BUILD has failed

swh/storage/filter.py:50: error: Argument 2 to "_filter_missing_ids" of "FilteringProxyStorage" has incompatible type "Generator[Any, None, None]"; expected "Sequence[bytes]"

hi ho!

mentioned in merge request !840 (closed)

! In !839 (closed), @ardumont wrote: I don't think this is useful anymore. The postgresql storage does the filtering itself now (see !251 (closed)). Are you planning on removing this feature from the postgresql storage?

It always did that.

Right, indeed

But it does it server-side so we send lots of data on the wire (which has a cost).

It's for doing it client side as the current loaders do but it's entangled within its own loading logic (check swh.loader.core.loader.BufferedLoader).

This allows to untangle that part configuration wise.

Right!

Accepting the diff, just make sure to fix the typo and the build

Merge request was accepted

approved this merge request

Actually, I still have some nitpicking to do: instead of having sample_data as a large fixture returning a dict of various value types; could you split it into smaller fixtures, each returning a single type of values?

Merge request was returned for changes

unapproved this merge request

Accepting, because @ardumont promised to fix it on IRC ;)

Merge request was accepted

approved this merge request

! In !839 (closed), @vlorentz wrote: Accepting, because @ardumont promised to fix it on IRC ;)

15:11 <+ardumont> i agree but there is an in-progress from douardda about the storage implem' tests
15:11 <+ardumont> and i think he will come up with something better
15:12 <vlorentz> are you *sure* this large fixture will be removed in the next ~2 weeks?
15:12 <+ardumont> most probably yes because *I*'ll make sure of it
15:12 <vlorentz> okay

Adapt according to review and mypy check

Build is green See https://jenkins.softwareheritage.org/job/DSTO/job/tox/673/ for more details.

Fix commit message