Skip to content

api/serializers: Add Exception type encoder

Antoine Lambert requested to merge generated-differential-D4515-source into generated-differential-D4515-target

I stumbled across that error while working on the Web API POC based on OpenAPI due to uffizi.internal.softwareheritage.org host currently not available.

INFO:werkzeug:127.0.0.1 - - [18/Nov/2020 17:33:25] "POST /content/data HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
OSError: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.7/http/client.py", line 1244, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1290, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1239, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/urllib3/connection.py", line 172, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fcf9a993ba8>: Failed to establish a new connection: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/urllib3/connectionpool.py", line 767, in urlopen
    **response_kw
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/urllib3/connectionpool.py", line 767, in urlopen
    **response_kw
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/urllib3/connectionpool.py", line 767, in urlopen
    **response_kw
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='uffizi.internal.softwareheritage.org', port=5003): Max retries exceeded with url: /content/get (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcf9a993ba8>: Failed to establish a new connection: [Errno 113] No route to host'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/anlambert/swh/swh-environment/swh-core/swh/core/api/__init__.py", line 254, in raw_verb
    return getattr(self.session, verb)(self._url(endpoint), **opts)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/requests/sessions.py", line 578, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='uffizi.internal.softwareheritage.org', port=5003): Max retries exceeded with url: /content/get (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcf9a993ba8>: Failed to establish a new connection: [Errno 113] No route to host'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "<decorator-gen-10>", line 2, in content_get_data
    
  File "/home/anlambert/swh/swh-environment/swh-core/swh/core/api/negotiation.py", line 147, in _negotiate
    return f.negotiator(*args, **kwargs)
  File "/home/anlambert/swh/swh-environment/swh-core/swh/core/api/negotiation.py", line 81, in __call__
    result = self.func(*args, **kwargs)
  File "/home/anlambert/swh/swh-environment/swh-core/swh/core/api/__init__.py", line 456, in _f
    return obj_meth(**kw)
  File "/home/anlambert/swh/swh-environment/swh-storage/swh/storage/metrics.py", line 24, in d
    return f(*a, **kw)
  File "/home/anlambert/swh/swh-environment/swh-storage/swh/storage/postgresql/storage.py", line 274, in content_get_data
    return self.objstorage.content_get(content)
  File "/home/anlambert/swh/swh-environment/swh-storage/swh/storage/objstorage.py", line 41, in content_get
    data = self.objstorage.get(obj_id)
  File "/home/anlambert/swh/swh-environment/swh-objstorage/swh/objstorage/multiplexer/multiplexer_objstorage.py", line 278, in get
    return storage.get(obj_id)
  File "/home/anlambert/swh/swh-environment/swh-objstorage/swh/objstorage/multiplexer/multiplexer_objstorage.py", line 105, in call
    return self.get_result_from_mailbox(mailbox)
  File "/home/anlambert/swh/swh-environment/swh-objstorage/swh/objstorage/multiplexer/multiplexer_objstorage.py", line 83, in get_result_from_mailbox
    raise result["result"] from None
  File "/home/anlambert/swh/swh-environment/swh-objstorage/swh/objstorage/multiplexer/multiplexer_objstorage.py", line 28, in run
    ret = getattr(self.storage, command)(*args, **kwargs)
  File "/home/anlambert/swh/swh-environment/swh-objstorage/swh/objstorage/api/client.py", line 54, in get
    return self._proxy.post("content/get", {"obj_id": obj_id})
  File "/home/anlambert/swh/swh-environment/swh-core/swh/core/api/__init__.py", line 272, in post
    **opts,
  File "/home/anlambert/swh/swh-environment/swh-core/swh/core/api/__init__.py", line 256, in raw_verb
    raise self.api_exception(e)
swh.objstorage.exc.ObjStorageAPIError: An unexpected error occurred in the api backend: HTTPConnectionPool(host='uffizi.internal.softwareheritage.org', port=5003): Max retries exceeded with url: /content/get (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcf9a993ba8>: Failed to establish a new connection: [Errno 113] No route to host'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/flask/app.py", line 1822, in handle_user_exception
    return handler(e)
  File "/home/anlambert/swh/swh-environment/swh-storage/swh/storage/api/server.py", line 48, in my_error_handler
    return error_handler(exception, encode_data)
  File "/home/anlambert/swh/swh-environment/swh-core/swh/core/api/__init__.py", line 399, in error_handler
    response = encoder(exception_to_dict(exception))
  File "/home/anlambert/swh/swh-environment/swh-core/swh/core/api/__init__.py", line 375, in encode_data_server
    encoded_data = ENCODERS[content_type](data, extra_encoders=extra_type_encoders)
  File "/home/anlambert/swh/swh-environment/swh-core/swh/core/api/serializers.py", line 228, in msgpack_dumps
    return msgpack.packb(data, use_bin_type=True, default=encode_types)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack
    
  File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
    
  File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack
    
  File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack
    
  File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack
    
  File "msgpack/_packer.pyx", line 258, in msgpack._cmsgpack.Packer._pack
    
  File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack
    
TypeError: can not serialize 'ConnectionError' object

A RemoteException could not be properly serialized due to the corner case that the wrapped expection has been constructed with another exception argument (here ObjStorageAPIError(ConnectionError())).

So handle that special case by adding a dedicated encoder to avoid remote exception serialization error.

Migrated from D4515 (view on Phabricator)

Merge request reports