Prior to this commit, a multiplexer instance would start during its
initialization to check the configuration of all its backends but also check
whether they can write or not even when not needed. As it exists read-only
backends for a multiplexer, that would just fail to start the service.

This evolves the objstorage to grow a check_write attribute (hence parametric
through configuration file). And then adapts accordingly the multiplexer
constructor so it can check the configuration of its backends according to
their respective configuration.

Refs. #4746
Refs. swh/infra/sysadm-environment#5334

Some objstorages are configured read-only implicitly by way of
permissions, so `check_config` raises a PermissionError instead of
returning False. Handle that situation explicitly.

Also handle RemoteExceptions with a warning, to support the old
situation where PermissionError was not properly wrapped by the RPC
layer.

`check_config` raises a PermissionError under some circumstances, and
it's nicer to be able to handle that directly than to have to unwrap a
RemoteException.

The `__del__` method gets called on object teardown in most cases.
Specifically, it can be called when `__init__` fails, in which case all
attributes aren't guaranteed to be set.

Closes #4744.

Under some circumstances (e.g. network instability, service restarts,
etc.), one of the object storages backing a multiplexer instance might be
temporarily unavailable.

Instead of immediately raising an exception, temporarily disable the
backend for further accesses and attempt reading the object from other
backends. Raise a specific `NoBackendsLeftError` if all the backends
have been disabled due to a transient exception.

Add a simple timing statsd probe for these methods.

For MultiplexerObjStorage.get(), also send a counter metrics
related to which backend did returned the object.

The counter have add a 'backend' tag with the name of the backend
that returned the object, and a 'backend_number' tag so it's easier
to figure how many fallbacks had to be reached to actually
retrieve a content object.

Note that it does not report timing probes for each backend, since
these metrics are already reported by the backend themselves.

It used to do nothing, which is inconsistent with the behavior of the
HTTPReadOnlyObjStorage.

Adapt a bit the multiplexer to handle this properly:

- keep a list of RW storage threads separated from the full list of
  backend storage threads
- the decision is made using a call to check_config(check_write=True)
  for each of the backend storages
- adapt a bit the API/behavior of the check_config() method to make it
  return False when called with 'check_write=True' for a RO obj storage,
- for the multiplexer, check_config(check_write=True) returns True if at
  least one backend objstorage is green,
- remove reference to deleted filters in the multiplexer doc string.

Add an 'Error' suffix so there is no confusion, especially with regard
to the new ReadOnlyProxyObjStorage proxy objstorage class (even if it's
not the same name, it can be very confusing).

Move the implementation in swh/objstorage/multiplexer.py since there is
nothing left in the swh/objstorage/multiplexer/ but this class.

Make the constructor able to instantiate the encapsulated objstorages so
we can get rid of the factory _construct_multiplexer_objstorage
function.

Get rid of the generic ObjStorageFilter complexity, since we only have a
single filter. This allows also to get rid of the
_construct_filtered_objstorage() factory function in factory.py; this later
is only deprecated for now for bw compat, but uses the new
ReadOnlyObjstorageProxy class instead.

this will allow using this name to add statsd probes ie.g. counting which
backend has been successful in a multiplexed backend.

Gunicorn access logs on RPC backends are pretty noisy by default, but
grouping successful requests into merged logs would make that more
bearable.

To use it, stick this in the gunicorn logging config:

```
filters:
  throttle_accesslog:
    (): swh.objstorage.backends.winery.gunicorn.ThrottledAccessLog
    interval: 60
    status_codes: [200]
handlers:
  gunicorn.access:
    level: INFO
    filters: [throttle_accesslog]
    handlers: [<your handler>]
```

Since recent (?) gunicorn versions, looks like worker_exit is called
rather than worker_int on graceful shutdowns.

`CREATE TABLE x (LIKE y INCLUDING ALL)` doesn't replicate the options
from the template table, so do it by hand.

We run winery writers in active/active mode on two hosts, but only
usually route requests to the host which is currently managing the
primary database. When the writer fails over, shards would keep being
locked in WRITING mode, even though they're idle.

Implement a RW shard idle timeout with a watchdog thread which gets
pinged every time an object is added. The timeout defaults to 5 minutes,
but can be adjusted with a configuration variable.

The init()/uninit() pattern was pretty confusing. Instead, use an
on_shutdown hook on the main storage, that calls down to the relevant
components' hooks, and can be called explicitly, or by gunicorn when it
shuts down a worker.

This test would always return False, even if the file was full of zeros

Now that we have connections to a single database, we don't need this
complexity anymore.

The semantics of creating and locking an arbitrary shard to record that
it has been mapped make little sense.

Now that databases are merged, we can add objects in a single
transaction instead of bouncing around with a 2-phase commit.

Instead of winery managing multiple databases by itself, creating quite
a complex database management lifecycle, simplify things drastically by
using the common SWH database management scaffolding.

Instead of separate databases, read-write shards now use separate tables
within the same database, created from the `shard_template` template
table. The throttler tables are also merged into the main database.

We completely drop the `shard_dsn` argument to all winery components.

When a packing operation is interrupted, the image can be left with
stale contents. Clean these stale contents up by zeroing the
image (either with blkdiscard or with ftruncate, as appropriate) before
writing the objects to the image.

`rbd image create` fails when the image already exists, so match that
behavior in the file-backed pool emulator.

These operations can be restarted from scratch, so it's much better to
release the lock on the affected shards when the operations fail.

Some ceph clusters (for instance, the one spawned by the ceph/demo
docker image) require some options to be passed to rbd device map. Add
support for configuring these options in all relevant functions.

This is useful to use a manually managed pool for all tests, e.g. when
using a toy ceph cluster generated by the ceph demo image.

Example usage (needs /etc/ceph to be empty!):

  docker run -d --net=host -v /etc/ceph:/etc/ceph -e MON_IP=192.168.1.201 -e CEPH_PUBLIC_NETWORK=192.168.1.0/24 -e CEPH_DEMO_UID=test-user quay.io/ceph/demo
  sudo chmod -R a+r /etc/ceph
  ceph config set global mon_allow_pool_size_one true
  ceph osd pool create winery-test-shards replicated --size=1 --yes-i-really-mean-it
  ceph osd pool create winery-test-shards-data replicated --size=1 --yes-i-really-mean-it
  ceph osd pool application enable winery-test-shards rbd
  ceph osd pool application enable winery-test-shards-data rbd

  CEPH_HARDCODE_POOL=winery-test-shards RBD_MAP_OPTIONS=ms_mode=prefer-secure pytest swh -k 'winery and not bench_real'

These were copy-and-pasted from the rbd function, generating nonsensical
names.

When an addition has been interrupted, a tmpfile can be lying around. A
lot of pathslicing objstorages are legacy, read-only partitions, so
removing these bogus files is not an option. Instead of breaking
iteration, just print a warning and keep on going.

Instead of failing fast when getting an ObjCorruptedError, try to read
from other objstorages (and report the error in logging).

Should work around Sentry issue SWH-INDEXER-YQ

Wrap object storage backends common code related to compressing an
an object to store in an utility method of base ObjStorage class.

Wrap object storage backends common code related to decompressing
a stored object in an utility method of base ObjStorage class in
order to centralize errors handling.

Catch possible exceptions decompressors might raise and raise an
ObjCorruptedError exception in such cases.

Related to swh-scrubber#4694.

Implement object check in base ObjStorage class instead of duplicating that code
in every object storage backend.

Add an ObjCorruptedError exception raised when an object corruption is detected.

Improve exception messages.

Normalize imports of exception classes.

Related to swh-scrubber#4694.