Since 43728c59, store_data consumes the entire iterator
of contents, and since 3b97703d, it does the same for
other object types.

This causes all the (new) objects of the loaded repository to be loaded
in memory at the same time before being sent to the storage, which can
cause OOM errors.

Instead, with this commit, objects are added one by one to the storage,
which restores the lazy behavior we had before these two commits using
the buffered storage proxy.

The deduplication logic of 'person' objects is an internal detail of
storage backends, so it's better not to rely on it.

Related to T2610

Related to T2610

flake8 hook has been removed from https://github.com/pre-commit/pre-commit-hooks
so now use the one from https://gitlab.com/pycqa/flake8

by moving import statements in functions.

Related to T2575.

Related to T645

It's a list with one item per file, so it makes sense it should be plural.

No migration needed, as this code wasn't released yet.

pytest wastes a lot of time in .hypothesis and .git; this commit excludes them.

The Debian loader currently does in the 'extrinsic' revision metadata;
so this avoids losing the URL when we stop writing extrinsic metadata
on revision objects.

Additionally, this makes all loaders write the 'url', which can be
useful later.

In addition to writing them on revisions, for now.

The authority is Software Heritage itself, because hashes and length are
computed by the loader, not the origin of the package.

It makes the function less big, and a future commit's tests will need to
override _load_directory to test the rest of _load_revision.

This is incorrect, and will become an error with the next version of swh-model.

Otherwise, when the list is too large, it cascades to a logging error...

snapshot_get is deprecated.

So the maintenance is easier (not necessarily code to package and deploy).

Instead of trying to download artifacts and then break on ingestion. Try to
prevent spurious downloads which cost too many wasted resources (both upstream
servers and our own infra) and many noisy issues in sentry.

Ideally, we should improve our archive support. In the mean time, this approach
allows more sensible treatment than the current approach (see previous points).
This will also allows to incrementally add support, and demonstrating it by
removing the new supported archive pattern in the current regexp in next
commits.

Related to T2510

Fixes build [1]

[1] https://jenkins.softwareheritage.org/job/DLDBASE/job/tests/1035/console

Related to T645

This also fixes the build [1]

[1] https://jenkins.softwareheritage.org/job/DLDBASE/job/tests/1028/console

Related to T2517

With a single implementation of the caching logic.

It errors in test_cran_parse_date, I don't understand why
it worked so far.

It will no longer return dicts in swh-storage v0.11.8.

1. origin_visit_add returns an iterable, not a sequence
2. starting with v0.11.6, _add() endpoints now require a List
   instead of an Iteralble.

All these will trigger type warning with swh.core > 0.2.0,
as it will no longer hide the signature of storage endpoints
from mypy

The next version of swh.core will allow mypy to detect the return
type of origin_visit_add is Iterable.

The RPC framework can't serialize them.

Also adding the StorageInterface type annotation, so that this becomes
type-checked with swh.core>v0.1.2 and swh.storage>v0.11.5.

Changed in swh-storage > v0.11.3