docker/conf: Use storage HTTP proxy in vault worker service (!260) · Merge requests · Platform / Development / swh-environment

Antoine Lambert requested to merge anlambert/swh-environment:docker-vault-worker-use-storage-proxy into master Apr 26, 2023
It prevents connection errors when cooking a large bundle.
Stack trace obtained with previous configuration
swh-vault-worker_1                   | [2023-04-26 13:53:11,444: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHCookingTask[1f8d2234-1266-43cb-947d-e5e3dd81b39e] received
swh-vault-worker_1                   | [2023-04-26 13:53:11,455: DEBUG/ForkPoolWorker-1] Loading config file /cooker.yml
swh-vault-worker_1                   | [2023-04-26 13:53:52,027: DEBUG/ForkPoolWorker-1] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@632ee1e680f6', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 19.2.1', 'product': 'RabbitMQ', 'version': '3.6.16'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
swh-vault-worker_1                   | [2023-04-26 13:53:52,028: DEBUG/ForkPoolWorker-1] using channel_id: 1
swh-vault-worker_1                   | [2023-04-26 13:53:52,030: DEBUG/ForkPoolWorker-1] Channel open
swh-vault-worker_1                   | [2023-04-26 13:53:52,032: INFO/ForkPoolWorker-1] Task swh.vault.cooking_tasks.SWHCookingTask[1f8d2234-1266-43cb-947d-e5e3dd81b39e] succeeded in 40.57624794500589s: None
swh-vault-worker_1                   | [2023-04-26 14:19:16,830: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHCookingTask[eacd1318-f2d1-4b90-ac4e-2c17413712ef] received
swh-vault-worker_1                   | [2023-04-26 14:19:16,832: DEBUG/ForkPoolWorker-1] Loading config file /cooker.yml
swh-vault-worker_1                   | hint: Using 'master' as the name for the initial branch. This default branch name
swh-vault-worker_1                   | hint: is subject to change. To configure the initial branch name to use in all
swh-vault-worker_1                   | hint: of your new repositories, which will suppress this warning, call:
swh-vault-worker_1                   | hint: 
swh-vault-worker_1                   | hint:    git config --global init.defaultBranch <name>
swh-vault-worker_1                   | hint: 
swh-vault-worker_1                   | hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
swh-vault-worker_1                   | hint: 'development'. The just-created branch can be renamed via this command:
swh-vault-worker_1                   | hint: 
swh-vault-worker_1                   | hint:    git branch -m <name>
swh-vault-worker_1                   | Initialized empty Git repository in /tmp/swh-vault-gitbare-exwx29kk/clone.git/
swh-vault-worker_1                   | [2023-04-26 14:41:00,273: ERROR/ForkPoolWorker-1] Bundle cooking failed.
swh-vault-worker_1                   | Traceback (most recent call last):
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
swh-vault-worker_1                   |     chunked=chunked,
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
swh-vault-worker_1                   |     conn.request(method, url, **httplib_request_kw)
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connection.py", line 244, in request
swh-vault-worker_1                   |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
swh-vault-worker_1                   |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
swh-vault-worker_1                   |     self._send_request(method, url, body, headers, encode_chunked)
swh-vault-worker_1                   |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
swh-vault-worker_1                   |     self.endheaders(body, encode_chunked=encode_chunked)
swh-vault-worker_1                   |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
swh-vault-worker_1                   |     self._send_output(message_body, encode_chunked=encode_chunked)
swh-vault-worker_1                   |   File "/usr/local/lib/python3.7/http/client.py", line 1075, in _send_output
swh-vault-worker_1                   |     self.send(chunk)
swh-vault-worker_1                   |   File "/usr/local/lib/python3.7/http/client.py", line 997, in send
swh-vault-worker_1                   |     self.sock.sendall(data)
swh-vault-worker_1                   | ConnectionResetError: [Errno 104] Connection reset by peer
swh-vault-worker_1                   | 
swh-vault-worker_1                   | During handling of the above exception, another exception occurred:
swh-vault-worker_1                   | 
swh-vault-worker_1                   | Traceback (most recent call last):
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 499, in send
swh-vault-worker_1                   |     timeout=timeout,
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 788, in urlopen
swh-vault-worker_1                   |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 550, in increment
swh-vault-worker_1                   |     raise six.reraise(type(error), error, _stacktrace)
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 769, in reraise
swh-vault-worker_1                   |     raise value.with_traceback(tb)
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
swh-vault-worker_1                   |     chunked=chunked,
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request
swh-vault-worker_1                   |     conn.request(method, url, **httplib_request_kw)
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connection.py", line 244, in request
swh-vault-worker_1                   |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
swh-vault-worker_1                   |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
swh-vault-worker_1                   |     self._send_request(method, url, body, headers, encode_chunked)
swh-vault-worker_1                   |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
swh-vault-worker_1                   |     self.endheaders(body, encode_chunked=encode_chunked)
swh-vault-worker_1                   |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
swh-vault-worker_1                   |     self._send_output(message_body, encode_chunked=encode_chunked)
swh-vault-worker_1                   |   File "/usr/local/lib/python3.7/http/client.py", line 1075, in _send_output
swh-vault-worker_1                   |     self.send(chunk)
swh-vault-worker_1                   |   File "/usr/local/lib/python3.7/http/client.py", line 997, in send
swh-vault-worker_1                   |     self.sock.sendall(data)
swh-vault-worker_1                   | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
swh-vault-worker_1                   | 
swh-vault-worker_1                   | During handling of the above exception, another exception occurred:
swh-vault-worker_1                   | 
swh-vault-worker_1                   | Traceback (most recent call last):
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 261, in raw_verb
swh-vault-worker_1                   |     return getattr(self.session, verb)(self._url(endpoint), **opts)
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 635, in post
swh-vault-worker_1                   |     return self.request("POST", url, data=data, json=json, **kwargs)
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 587, in request
swh-vault-worker_1                   |     resp = self.send(prep, **send_kwargs)
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 701, in send
swh-vault-worker_1                   |     r = adapter.send(request, **kwargs)
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 547, in send
swh-vault-worker_1                   |     raise ConnectionError(err, request=request)
swh-vault-worker_1                   | requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
swh-vault-worker_1                   | 
swh-vault-worker_1                   | During handling of the above exception, another exception occurred:
swh-vault-worker_1                   | 
swh-vault-worker_1                   | Traceback (most recent call last):
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/vault/cookers/base.py", line 134, in cook
swh-vault-worker_1                   |     self.prepare_bundle()
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/vault/cookers/git_bare.py", line 163, in prepare_bundle
swh-vault-worker_1                   |     self.load_objects()
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/vault/cookers/git_bare.py", line 411, in load_objects
swh-vault-worker_1                   |     self.load_directories(directory_ids)
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/vault/cookers/git_bare.py", line 634, in load_directories
swh-vault-worker_1                   |     obj_ids,
swh-vault-worker_1                   |   File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 748, in next
swh-vault-worker_1                   |     raise value
swh-vault-worker_1                   |   File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 121, in worker
swh-vault-worker_1                   |     result = (True, func(*args, **kwds))
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/vault/cookers/git_bare.py", line 633, in <lambda>
swh-vault-worker_1                   |     lambda obj_id: self.load_directory(obj_id, raw_manifests.get(obj_id)),
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/vault/cookers/git_bare.py", line 641, in load_directory
swh-vault-worker_1                   |     self.storage.directory_get_entries, obj_id
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/classes.py", line 47, in stream_results_optional
swh-vault-worker_1                   |     res = f(*args, page_token=None, **kwargs)
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 190, in meth_
swh-vault-worker_1                   |     return self._post(meth._endpoint_path, post_data)
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 279, in _post
swh-vault-worker_1                   |     **opts,
swh-vault-worker_1                   |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 263, in raw_verb
swh-vault-worker_1                   |     raise self.api_exception(e)
swh-vault-worker_1                   | swh.storage.exc.StorageAPIError: An unexpected error occurred in the api backend: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
swh-vault-worker_1                   | [2023-04-26 14:41:00,281: INFO/ForkPoolWorker-1] Task swh.vault.cooking_tasks.SWHCookingTask[eacd1318-f2d1-4b90-ac4e-2c17413712ef] succeeded in 1303.4485696100019s: None
docker/conf: Use storage HTTP proxy in vault worker service

Merge request reports
