When unset, defaults to a NoopObjstorage. This can be useful for
testing, and it allows to use the `swh db init storage` without
a configuration file.

Warn the user if the storage instance actually use this noop objstorage.

It now needs types-click which is indeed a dependency of
swh.core[testing].

This allows creating the object references partitions even on storages
that are wrapped in any number of proxies, which is the case for both
production and staging (they're using the reference recording proxy).

Related to swh/infra/sysadm-environment#4811

The previous implementation would prevent the insertion of
object_references on Sundays.

Some circumstances generate duplicate entries that we should catch.

Our tests were incomplete, and didn't catch that the postgresql backend
wasn't able to store references from origins to snapshots (as there is
no `origin` object type in the postgresql schema)

Click 7.0 (which we use on buster) doesn't autogenerate subcommand
names, at least under some circumstances, so be explicit about them.

This does not matter much for normal loaders as they only do a single
`raw_extrinsic_metadata_add` before flushing; but the metadata loader
loads metadata from a bunch of origins (as many as there are origins in
the configured Kafka batch) one-by-one before flushing.

A future commit will add a counter for the size of 'raw_extrinsic_metadata',
which makes this code feel a bit repetitive

When branches_from parameter was greater than branch_name_exclude_prefix,
branches whose name is greater than the prefix but lesser than branches_from
could be returned.

So do not call snapshot_branch_get_range method for such cases and simply
call snapshot_branch_get_from_name.

Related to swh/infra/sysadm-environment#4863

Original issue has been fixed.

Due to the client-side filtering of branches returned by the CQL runner,
the number of returned branches could be off by one or greater than the
requested branches count.

So ensure to return the correct number of branches when filtering by
target types for instance.

This updates the PostgreSQL storage test fixtures to always create the
current partition, so that object_reference-related tests can pass.

Co-authored with @lunar

This will allow the takedown processing scripts to check whether the
objects they intend to remove are referenced by objects that are not
expected to be removed.

Ref. swh/devel/swh-storage#4680

This is a recurring mistake, that should be caught by other tests but
sometimes isn't.

A future commit will check statements look as expected; and not dealing
with two different syntaxes makes this task simpler

A future commit will check statements look as expected; and not dealing
with comments makes this task easier.

It's like directory_get_many, but allows building directories via
Directory.from_possibly_duplicated_entries().

It will be used by swh-scrubber in order not to crash on such directories.

Next version of swh.model will change a bit the API of this method. Make
storage compatible with both versions of the API.

In next version of swh.model, this class is expected to not handle the
"length" as an (optional) hash algo. Make this test compatible with both
versions of the MultiHash class.

This is needed to help a small swh.model (future) evolution that will
make these methods return TypedDict instead of raw dicts.

This is only to prevent a mypy breakage during the transition to the
next swh.model version (v6 -> v7).

A future revision will come to "fix" this code annotation once swh.model
v7 is published.

As a temporary workaround for https://github.com/redis/redis-py/issues/2629

swh_visit_find_by_date tries to do an index only scan on `origin_visit
(origin, date)`. However, by sorting the results on an extra column (the
visit id), the postgresql planner would end up doing an index scan,
which times out on origins with lots of visits.

Splitting the work across two steps (first find the date, which works
with index only scans, then find the highest visit id, which usually
only returns one row) fixes this issue.

The dict they returned are now proper classes with their own
documentation.