WIP backend: add retrying on IO error with remote services

Wrapper function on objects that let choose methods to retry on IO errors.

This function would not live here of course by rather in core if useful.

Not tested yet, had some troubles finding how to decorate methods of objects by decorating only the object, and decorate where it is used rather than where defined.

Inspiration taken from RetryingProxyStorage. First, need feedback on the approach, before discussing implementation.

Related to #2862 (closed)

Test Plan

local tests then proper pytest

---

Migrated from D4630 (view on Phabricator)

Build was aborted

Patch application report for D4630 (id=16427)

Rebasing onto 4e1fb5e0...

First, rewinding head to replay your work on top of it...
Applying: backend: add retrying on IO error with remote services
Using index info to reconstruct a base tree...
M	swh/vault/backend.py
Falling back to patching base and 3-way merge...
Auto-merging swh/vault/backend.py

Changes applied before test

commit ebd1aaef2425859fc70202361072d1859b8f99a1
Author: tenma <tenma+swh@mailbox.org>
Date:   Mon Nov 30 11:30:12 2020 +0100

    backend: add retrying on IO error with remote services
    
    Decorator-like function on objects that let choose methods to retry on IO
    error.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/58/ See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/58/console

@tenma This is partially (for the storage part) superseded by swh/infra/puppet/puppet-swh-site!276 (closed). So might be, this need to be closed now or in planned changes.

Another strategy preferred: using retrying proxies for storage (existing and activated) and scheduler (to be done).

Merge request was abandoned

closed