swh api clients often fail with a BadStatusLine exception

The swh backend API clients (swh.storage.api.client and swh.obstorage.api.client) often fail with a BadStatusLine exception from deep in the bowels of urllib3.

This is especially significant under high loads (e.g. when rehashing millions of objects through the objstorage api).

Basically, there is a race condition between the client, which eagerly pipelines requests, and the server, which reaps children every few hundred thousand requests because of a memory leak (swh/devel/swh-storage#757 (closed)).

The HTTP implementation of the WSGI runners (gunicorn, uwsgi) is not supposed to be exposed to external clients; We should probably bite the bullet and front them with an actual reverse proxy such as nginx or apache (ugh).

Migrated from T760 (view on Phabricator)

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information