The quick-start section was a bit outdated.

Make the throttler window of measurement a module-level constant, and
clean up stale rows that are twice as old as the window (10 minutes)
instead of one month old.

All throttler queries filter these tables using ranges of the updated
column, so adding a block range index makes these queries much more
efficient.

Ref. swh/infra/sysadm-environment#5388

We don’t want to see a full stack trace as it will very likely be noise
as it is a transient error. But we still add a new line with the
exception it self, though.

and slightly improve a few points.

This makes the multiplexer raise a proper ReadOnlyObjStorageError when
calling one of the write endpoints, and avoids calling on check_config
on every backend at __init__ time in that situation, which prevents us
from initializing the multiplexer if one of the backends times out.

Add support for an "retry" configuration entry that sets up and
configure an urllib3 Retry HTTPAdapter.

The request that we were making actually fails with a container shared
access signature, it only works for account shared signatures. Account
shared signatures give much more access than needed, so only parse the
URL to (lightly) check if it is valid.

Some objstorages are configured read-only implicitly by way of
permissions, so `check_config` raises a PermissionError instead of
returning False. Handle that situation explicitly.

Also handle RemoteExceptions with a warning, to support the old
situation where PermissionError was not properly wrapped by the RPC
layer.

`check_config` raises a PermissionError under some circumstances, and
it's nicer to be able to handle that directly than to have to unwrap a
RemoteException.

The `__del__` method gets called on object teardown in most cases.
Specifically, it can be called when `__init__` fails, in which case all
attributes aren't guaranteed to be set.

Closes #4744.

Under some circumstances (e.g. network instability, service restarts,
etc.), one of the object storages backing a multiplexer instance might be
temporarily unavailable.

Instead of immediately raising an exception, temporarily disable the
backend for further accesses and attempt reading the object from other
backends. Raise a specific `NoBackendsLeftError` if all the backends
have been disabled due to a transient exception.

Add a simple timing statsd probe for these methods.

For MultiplexerObjStorage.get(), also send a counter metrics
related to which backend did returned the object.

The counter have add a 'backend' tag with the name of the backend
that returned the object, and a 'backend_number' tag so it's easier
to figure how many fallbacks had to be reached to actually
retrieve a content object.

Note that it does not report timing probes for each backend, since
these metrics are already reported by the backend themselves.

It used to do nothing, which is inconsistent with the behavior of the
HTTPReadOnlyObjStorage.

Adapt a bit the multiplexer to handle this properly:

- keep a list of RW storage threads separated from the full list of
  backend storage threads
- the decision is made using a call to check_config(check_write=True)
  for each of the backend storages
- adapt a bit the API/behavior of the check_config() method to make it
  return False when called with 'check_write=True' for a RO obj storage,
- for the multiplexer, check_config(check_write=True) returns True if at
  least one backend objstorage is green,
- remove reference to deleted filters in the multiplexer doc string.

Add an 'Error' suffix so there is no confusion, especially with regard
to the new ReadOnlyProxyObjStorage proxy objstorage class (even if it's
not the same name, it can be very confusing).

Move the implementation in swh/objstorage/multiplexer.py since there is
nothing left in the swh/objstorage/multiplexer/ but this class.

Make the constructor able to instantiate the encapsulated objstorages so
we can get rid of the factory _construct_multiplexer_objstorage
function.

Get rid of the generic ObjStorageFilter complexity, since we only have a
single filter. This allows also to get rid of the
_construct_filtered_objstorage() factory function in factory.py; this later
is only deprecated for now for bw compat, but uses the new
ReadOnlyObjstorageProxy class instead.

this will allow using this name to add statsd probes ie.g. counting which
backend has been successful in a multiplexed backend.

Gunicorn access logs on RPC backends are pretty noisy by default, but
grouping successful requests into merged logs would make that more
bearable.

To use it, stick this in the gunicorn logging config:

```
filters:
  throttle_accesslog:
    (): swh.objstorage.backends.winery.gunicorn.ThrottledAccessLog
    interval: 60
    status_codes: [200]
handlers:
  gunicorn.access:
    level: INFO
    filters: [throttle_accesslog]
    handlers: [<your handler>]
```

Since recent (?) gunicorn versions, looks like worker_exit is called
rather than worker_int on graceful shutdowns.

`CREATE TABLE x (LIKE y INCLUDING ALL)` doesn't replicate the options
from the template table, so do it by hand.

We run winery writers in active/active mode on two hosts, but only
usually route requests to the host which is currently managing the
primary database. When the writer fails over, shards would keep being
locked in WRITING mode, even though they're idle.

Implement a RW shard idle timeout with a watchdog thread which gets
pinged every time an object is added. The timeout defaults to 5 minutes,
but can be adjusted with a configuration variable.

The init()/uninit() pattern was pretty confusing. Instead, use an
on_shutdown hook on the main storage, that calls down to the relevant
components' hooks, and can be called explicitly, or by gunicorn when it
shuts down a worker.

This test would always return False, even if the file was full of zeros

Now that we have connections to a single database, we don't need this
complexity anymore.

The semantics of creating and locking an arbitrary shard to record that
it has been mapped make little sense.

Now that databases are merged, we can add objects in a single
transaction instead of bouncing around with a 2-phase commit.

Instead of winery managing multiple databases by itself, creating quite
a complex database management lifecycle, simplify things drastically by
using the common SWH database management scaffolding.

Instead of separate databases, read-write shards now use separate tables
within the same database, created from the `shard_template` template
table. The throttler tables are also merged into the main database.

We completely drop the `shard_dsn` argument to all winery components.

When a packing operation is interrupted, the image can be left with
stale contents. Clean these stale contents up by zeroing the
image (either with blkdiscard or with ftruncate, as appropriate) before
writing the objects to the image.

`rbd image create` fails when the image already exists, so match that
behavior in the file-backed pool emulator.

These operations can be restarted from scratch, so it's much better to
release the lock on the affected shards when the operations fail.

Some ceph clusters (for instance, the one spawned by the ceph/demo
docker image) require some options to be passed to rbd device map. Add
support for configuring these options in all relevant functions.

This is useful to use a manually managed pool for all tests, e.g. when
using a toy ceph cluster generated by the ceph demo image.

Example usage (needs /etc/ceph to be empty!):

  docker run -d --net=host -v /etc/ceph:/etc/ceph -e MON_IP=192.168.1.201 -e CEPH_PUBLIC_NETWORK=192.168.1.0/24 -e CEPH_DEMO_UID=test-user quay.io/ceph/demo
  sudo chmod -R a+r /etc/ceph
  ceph config set global mon_allow_pool_size_one true
  ceph osd pool create winery-test-shards replicated --size=1 --yes-i-really-mean-it
  ceph osd pool create winery-test-shards-data replicated --size=1 --yes-i-really-mean-it
  ceph osd pool application enable winery-test-shards rbd
  ceph osd pool application enable winery-test-shards-data rbd

  CEPH_HARDCODE_POOL=winery-test-shards RBD_MAP_OPTIONS=ms_mode=prefer-secure pytest swh -k 'winery and not bench_real'