This should make it run up to 100 times faster, even on average directories.

Instead of fetching them one-by-one, with the very high latency this entails.

This is preliminary work to make `directory_ls` less painfully slow.

And fall back to concurrent insertion.

By reusing the 'steady state' main statement (which is quite large)
across calls.

This adds a new config option for the cassandra backend,
'directory_entries_insert_algo', with three possible values:

* 'one-per-one' is the default, and preserves the current naive behavior
* 'concurrent' and 'batch' are attempts at being more efficient

This will be used as a second pass on objects that failed with
older versions of the script.

They were inaccurate and a performance bottleneck.

We can/should use swh-counters instead, now.

resulting in OriginVisitStatus trying to put a snapshot id in the
metadata field

Related to T3539

Related to T3517

It will be used in the Cassandra experiment.

Currently we use the built-in counters of the Cassandra backend; but in
addition to being inaccurate, they seem to be a bottleneck.

This proxy will be a lightweight solution for counting object insertion,
without needing to run Kafka on the test cluster.

It was unclear this actually worked; I had to write this test to realize
the code wasn't buggy.

Also replaced a conditional that is always False (because Cassandra
always returns results in the order of the clustering key) with an
assertion, so the code is less confusing.

When called by a replayer, the visit.visit field is set; but
origin.next_visit_id was never incremented, so on the next loader
run, the visit id would be 1 even if there is already a visit
with that id.

Instead of calling content_find() for each object, which needs to make
two queries for each.

Given the latency of Cassandra queries, this should be a significant
speed-up (possibly up to 100 times faster, as this is the value of
PARTITION_KEY_RESTRICTION_MAX_SIZE).

This also changes the schema, because CQL does not allow doing `IN`
queries on compound partition keys.

Related to T3485

This fields allows having multiple version of the ExtID -> SWHID
mapping, for instance when the implementation of a loader changes in a
backwards-incompatible way.

For now, we don't change the API used to query or store ExtIDs. When
querying for the SWHIDs corresponding to a given external objects, all
versions are returned, and the client is expected to do the filtering.

The default ONE level is used to keep the previous behaviour

Related to T3396

This allows mypy to actually type-check calls to db methods.

This commit also fixes an issue found by mypy.

This will make it easier for users of swh-web to discover metadata
on a given SWHID, as you otherwise need to specify an authority to
fetch metadata.

We agreed a while ago they are IRIs, and we have some of them in the
postgresql database already.

This will be used by swh-web to allow downloading them
from a non-JSON endpoint.

All features work but snapshot_count_branches, because ScyllaDB does not
support user-defined aggregates yet.

Migration tests hang when run after the regular tests, but I can't
figure out why. This should not be an issue for now, as we won't run
Scylla tests on the CI.

This will remove further deprecation warnings from the tests, especially the ones from
other modules depending on the storage's pytest-plugin.

This also fixes some edge case configuration for the backfill and the storage rpc
backend which would have been broken if we switched to that new name prior to this.

Related to b487a21f

pytest-xdist expects the parameters to be in the same order
in all processes.

Before this commit, the only way to get Content objects from their sha1_git
was to call content_find for each object.
This was obviously neither convenient nor efficient.

Using this endpoint to batch calls reduces the runtime of the git-bare
vault cooker by 30%.

It spares a join with the content table, which should hopefully make
the vault (and possibly other users) faster when they don't need this
join.

to clean a bit the swh.storage namespace.

give a chance to one-object batches to be ingested, and reduce the
number of objects wrongly reported as non-ingested, e.g. during a
replayer session, where this situation can occur.

It renamed db_name to dbname, which is a breaking change.