Skip to content

server/app: Fix aiohttp >= 3.7 exception related errors

Antoine Lambert requested to merge generated-differential-D4466-source into generated-differential-D4466-target

Finally found the bug in swh-graph since the bump to aiohttp 3.7.

Using pytest -sv --log-cli-level DEBUG --swh/graph/tests/test_api_client.py::test_param_validation, the following stack trace is displayed:

ERROR    aiohttp.server:web_protocol.py:393 Unhandled exception
Traceback (most recent call last):
  File "/home/anlambert/swh/swh-environment/swh-graph/swh/graph/server/app.py", line 61, in node_of_swhid
    return self.backend.swhid2node[swhid]
  File "/home/anlambert/swh/swh-environment/swh-graph/swh/graph/swhid.py", line 297, in __getitem__
    return self._find(swhid_str)[0]  # return element, ignore position
  File "/home/anlambert/swh/swh-environment/swh-graph/swh/graph/swhid.py", line 285, in _find
    raise KeyError(swhid_str)
KeyError: 'swh:1:ori:fff0000000000000000000000000000000000021'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 422, in _handle_request
    resp = await self._request_handler(request)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/aiohttp/web_app.py", line 499, in _handle
    resp = await handler(request)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 118, in impl
    return await handler(request)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/aiohttp_utils/negotiation.py", line 216, in middleware
    response = yield from handler(request)
  File "/home/anlambert/swh/swh-environment/swh-core/swh/core/api/asynchronous.py", line 71, in middleware_handler
    return await handler(request)
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/aiohttp/web_urldispatcher.py", line 948, in _iter
    resp = await method()
  File "/home/anlambert/swh/swh-environment/swh-graph/swh/graph/server/app.py", line 128, in get
    await self.prepare_response()
  File "/home/anlambert/swh/swh-environment/swh-graph/swh/graph/server/app.py", line 165, in prepare_response
    self.src_node = self.node_of_swhid(src)
  File "/home/anlambert/swh/swh-environment/swh-graph/swh/graph/server/app.py", line 63, in node_of_swhid
    raise aiohttp.web.HTTPNotFound(body=f"SWHID not found: {swhid}")
aiohttp.web_exceptions.HTTPNotFound: Not Found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 487, in start
    resp, reset = await task
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 428, in _handle_request
    status=exc.status, reason=exc.reason, text=exc.text, headers=exc.headers
  File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/aiohttp/web_response.py", line 650, in text
    return self._body.decode(self.charset or "utf-8")
AttributeError: 'StringPayload' object has no attribute 'decode'

Since aiohttp 3.7, the internal handling of aiohttp.web.HTTPException has changed and a aiohttp.web_response.Response is now constructed from the exception text then returned (see reference code).

aiohttp exception constructions in swh-graph were passing error message through the body keyword parameter instead of the text one, leading to an error related to unicode decoding as body is expected to be bytes.

Using the text keyword parameter when constructing aiohttp exceptions now ensures that response body will be properly encoded and thus avoid the decoding errors.

Closes T2768

Migrated from D4466 (view on Phabricator)

Merge request reports