When both hosts try to record the shard as mapped read-only at the same
time, one of them can fail. This situation cannot be recovered from
automatically.

To recover from the situation, first, add a retry to the
record_shard_mapped call, which should make it succeed from within the
main loop; second, when initializing the list of shards, if the shard is
mapped read-only and its state is "PACKED" (which should be pretty
rare), then mark it as mapped.

All our uses of it have been adopted upstream

Rather than repeating a hardcoded list of hashes everywhere.

The call was missing the conversion to sha256 before passing it to the
backend, and delete() must return True if the deletion has happened.

The previous implementation would only raise the exception with a
sha256, which is not what tests expect.

The main goal of this commit is adding support for PRIMARY_HASH==sha256
in the default implementation of list_content. To do so, we need to
introduce PRIMARY_HASH in the ObjStorageInterface, which pulls a bunch
of changes to appease mypy.

Supposedly this might make it easier to handle configurable PRIMARY_HASH?

Some tests rely on this correspondence.

This breaks many objstorage invariants (e.g. content-addressing) which
makes maintenance painful for little benefit.

Otherwise it uses the same /tmp/winery directory which makes CI unhappy
when running several test sessions concurrently.

And use a specific skip message.

The quick-start section was a bit outdated.

Make the throttler window of measurement a module-level constant, and
clean up stale rows that are twice as old as the window (10 minutes)
instead of one month old.

All throttler queries filter these tables using ranges of the updated
column, so adding a block range index makes these queries much more
efficient.

Ref. swh/infra/sysadm-environment#5388

We don’t want to see a full stack trace as it will very likely be noise
as it is a transient error. But we still add a new line with the
exception it self, though.

and slightly improve a few points.

This makes the multiplexer raise a proper ReadOnlyObjStorageError when
calling one of the write endpoints, and avoids calling on check_config
on every backend at __init__ time in that situation, which prevents us
from initializing the multiplexer if one of the backends times out.

Add support for an "retry" configuration entry that sets up and
configure an urllib3 Retry HTTPAdapter.

The request that we were making actually fails with a container shared
access signature, it only works for account shared signatures. Account
shared signatures give much more access than needed, so only parse the
URL to (lightly) check if it is valid.

Some objstorages are configured read-only implicitly by way of
permissions, so `check_config` raises a PermissionError instead of
returning False. Handle that situation explicitly.

Also handle RemoteExceptions with a warning, to support the old
situation where PermissionError was not properly wrapped by the RPC
layer.

`check_config` raises a PermissionError under some circumstances, and
it's nicer to be able to handle that directly than to have to unwrap a
RemoteException.

The `__del__` method gets called on object teardown in most cases.
Specifically, it can be called when `__init__` fails, in which case all
attributes aren't guaranteed to be set.

Closes #4744.

Under some circumstances (e.g. network instability, service restarts,
etc.), one of the object storages backing a multiplexer instance might be
temporarily unavailable.

Instead of immediately raising an exception, temporarily disable the
backend for further accesses and attempt reading the object from other
backends. Raise a specific `NoBackendsLeftError` if all the backends
have been disabled due to a transient exception.

Add a simple timing statsd probe for these methods.

For MultiplexerObjStorage.get(), also send a counter metrics
related to which backend did returned the object.

The counter have add a 'backend' tag with the name of the backend
that returned the object, and a 'backend_number' tag so it's easier
to figure how many fallbacks had to be reached to actually
retrieve a content object.

Note that it does not report timing probes for each backend, since
these metrics are already reported by the backend themselves.

It used to do nothing, which is inconsistent with the behavior of the
HTTPReadOnlyObjStorage.

Adapt a bit the multiplexer to handle this properly:

- keep a list of RW storage threads separated from the full list of
  backend storage threads
- the decision is made using a call to check_config(check_write=True)
  for each of the backend storages
- adapt a bit the API/behavior of the check_config() method to make it
  return False when called with 'check_write=True' for a RO obj storage,
- for the multiplexer, check_config(check_write=True) returns True if at
  least one backend objstorage is green,
- remove reference to deleted filters in the multiplexer doc string.

Add an 'Error' suffix so there is no confusion, especially with regard
to the new ReadOnlyProxyObjStorage proxy objstorage class (even if it's
not the same name, it can be very confusing).

Move the implementation in swh/objstorage/multiplexer.py since there is
nothing left in the swh/objstorage/multiplexer/ but this class.

Make the constructor able to instantiate the encapsulated objstorages so
we can get rid of the factory _construct_multiplexer_objstorage
function.

Get rid of the generic ObjStorageFilter complexity, since we only have a
single filter. This allows also to get rid of the
_construct_filtered_objstorage() factory function in factory.py; this later
is only deprecated for now for bw compat, but uses the new
ReadOnlyObjstorageProxy class instead.