Some unknown SWHID errors crash the graph server
HTTP logs:
Mar 29 12:45:26 granet swh[2170078]: INFO:aiohttp.access:192.168.100.111 [29/Mar/2022:12:45:26 +0000] "GET /graph/visit/nodes/swh:1:snp:2758ba0f8125d4a0a6bd2befbfbddad381aef915?edges=snp:rev,snp:rel,rel:rev&direction=forward&max_edges=0&return_types=rev HTTP/1.1" 200 257 "-" "python-requests/2.27.1"
Mar 29 12:45:26 granet swh[2170078]: INFO:aiohttp.access:192.168.100.111 [29/Mar/2022:12:45:26 +0000] "GET /graph/visit/nodes/swh:1:snp:6647d5a4677e69a37bcf13d44d428c98ba20a509?edges=snp:rev,snp:rel,rel:rev&direction=forward&max_edges=0&return_types=rev HTTP/1.1" 200 308 "-" "python-requests/2.27.1"
Mar 29 12:45:26 granet swh[2170078]: INFO:aiohttp.access:192.168.100.111 [29/Mar/2022:12:45:26 +0000] "GET /graph/visit/nodes/swh:1:snp:b4d3cb72786b432c883c2880280306a2c33cdd30?edges=snp:rev,snp:rel,rel:rev&direction=forward&max_edges=0&return_types=rev HTTP/1.1" 200 257 "-" "python-requests/2.27.1"
Mar 29 12:45:26 granet swh[2170078]: INFO:aiohttp.access:192.168.100.111 [29/Mar/2022:12:45:26 +0000] "GET /graph/visit/nodes/swh:1:snp:f50e0acf8e964ca04977d2c8f3032af977bb8869?edges=snp:rev,snp:rel,rel:rev&direction=forward&max_edges=0&return_types=rev HTTP/1.1" 200 308 "-" "python-requests/2.27.1"
Mar 29 12:45:27 granet swh[2170078]: INFO:aiohttp.access:192.168.100.111 [29/Mar/2022:12:45:27 +0000] "GET /graph/visit/nodes/swh:1:snp:037ac774b7f57d0be7b0e8f4af211fcd2c1cb843?edges=snp:rev,snp:rel,rel:rev&direction=forward&max_edges=0&return_types=rev HTTP/1.1" 200 410 "-" "python-requests/2.27.1"
Mar 29 12:45:27 granet swh[2170078]: INFO:aiohttp.access:192.168.100.111 [29/Mar/2022:12:45:27 +0000] "GET /graph/visit/nodes/swh:1:snp:05d5c8aa16a0488b363d7986c18e8fc2d0e1e750?edges=snp:rev,snp:rel,rel:rev&direction=forward&max_edges=0&return_types=rev HTTP/1.1" 200 359 "-" "python-requests/2.27.1"
Mar 29 12:45:28 granet swh[2170078]: INFO:aiohttp.access:192.168.100.111 [29/Mar/2022:12:45:28 +0000] "GET /graph/visit/nodes/swh:1:snp:1a8893e6a86f444e8be8e7bda6cb34fb1735a00e?edges=snp:rev,snp:rel,rel:rev&direction=forward&max_edges=0&return_types=rev HTTP/1.1" 200 155 "-" "python-requests/2.27.1"
Mar 29 12:45:28 granet swh[2170078]: INFO:aiohttp.access:192.168.100.111 [29/Mar/2022:12:45:28 +0000] "GET /graph/visit/nodes/swh:1:snp:47d7d8afaf824c7bfef1d9bae5157d5795f5d16d?edges=snp:rev,snp:rel,rel:rev&direction=forward&max_edges=0&return_types=rev HTTP/1.1" 200 308 "-" "python-requests/2.27.1"
Mar 29 12:45:28 granet swh[2170078]: ERROR:root:An error occurred while calling o422540.visit_nodes.
Mar 29 12:45:28 granet swh[2170078]: : java.lang.IllegalArgumentException: Unknown SWHID: swh:1:snp:af3d189d1f742c42239460da00acc92c1d867427
Mar 29 12:45:28 granet swh[2170078]: at org.softwareheritage.graph.maps.NodeIdMap.getNodeId(NodeIdMap.java:152)
Mar 29 12:45:28 granet swh[2170078]: at org.softwareheritage.graph.maps.NodeIdMap.getNodeId(NodeIdMap.java:157)
Mar 29 12:45:28 granet swh[2170078]: at org.softwareheritage.graph.Graph.getNodeId(Graph.java:280)
Mar 29 12:45:28 granet swh[2170078]: at org.softwareheritage.graph.Entry$QueryHandler.visit_nodes(Entry.java:136)
Mar 29 12:45:28 granet swh[2170078]: at jdk.internal.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
Mar 29 12:45:28 granet swh[2170078]: at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Mar 29 12:45:28 granet swh[2170078]: at java.base/java.lang.reflect.Method.invoke(Method.java:566)
Mar 29 12:45:28 granet swh[2170078]: at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
Mar 29 12:45:28 granet swh[2170078]: at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
Mar 29 12:45:28 granet swh[2170078]: at py4j.Gateway.invoke(Gateway.java:282)
Mar 29 12:45:28 granet swh[2170078]: at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
Mar 29 12:45:28 granet swh[2170078]: at py4j.commands.CallCommand.execute(CallCommand.java:79)
Mar 29 12:45:28 granet swh[2170078]: at py4j.GatewayConnection.run(GatewayConnection.java:238)
Mar 29 12:45:28 granet swh[2170078]: at java.base/java.lang.Thread.run(Thread.java:829)
Mar 29 12:45:28 granet swh[2170078]: Traceback (most recent call last):
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/swh/graph/backend.py", line 163, in java_call_iterator
Mar 29 12:45:28 granet swh[2170078]: async for value in reader:
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/swh/graph/backend.py", line 114, in read_node_ids
Mar 29 12:45:28 granet swh[2170078]: with (await asyncio.wait_for(open_thread, timeout=2)) as f:
Mar 29 12:45:28 granet swh[2170078]: File "/usr/lib/python3.7/asyncio/tasks.py", line 423, in wait_for
Mar 29 12:45:28 granet swh[2170078]: raise futures.TimeoutError()
Mar 29 12:45:28 granet swh[2170078]: concurrent.futures._base.TimeoutError
Mar 29 12:45:28 granet swh[2170078]: During handling of the above exception, another exception occurred:
Mar 29 12:45:28 granet swh[2170078]: Traceback (most recent call last):
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/swh/core/api/asynchronous.py", line 71, in middleware_handler
Mar 29 12:45:28 granet swh[2170078]: return await handler(request)
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/aiohttp/web_urldispatcher.py", line 892, in _iter
Mar 29 12:45:28 granet swh[2170078]: resp = await method()
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/swh/graph/server/app.py", line 164, in get
Mar 29 12:45:28 granet swh[2170078]: await self.stream_response()
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/swh/graph/server/app.py", line 220, in stream_response
Mar 29 12:45:28 granet swh[2170078]: self.return_types,
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/swh/graph/backend.py", line 84, in traversal
Mar 29 12:45:28 granet swh[2170078]: async for line in method(*args):
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/swh/graph/backend.py", line 172, in java_call_iterator
Mar 29 12:45:28 granet swh[2170078]: raise task_exc
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/swh/graph/backend.py", line 141, in java_call
Mar 29 12:45:28 granet swh[2170078]: await loop.run_in_executor(None, lambda: func(*args, **kwargs))
Mar 29 12:45:28 granet swh[2170078]: File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
Mar 29 12:45:28 granet swh[2170078]: result = self.fn(*self.args, **self.kwargs)
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/swh/graph/backend.py", line 141, in <lambda>
Mar 29 12:45:28 granet swh[2170078]: await loop.run_in_executor(None, lambda: func(*args, **kwargs))
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/py4j/java_gateway.py", line 1286, in __call__
Mar 29 12:45:28 granet swh[2170078]: answer, self.gateway_client, self.target_id, self.name)
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/py4j/protocol.py", line 328, in get_return_value
Mar 29 12:45:28 granet swh[2170078]: format(target_id, ".", name), value)
Mar 29 12:45:28 granet swh[2170078]: py4j.protocol.Py4JJavaError: An error occurred while calling o422540.visit_nodes.
Mar 29 12:45:28 granet swh[2170078]: : java.lang.IllegalArgumentException: Unknown SWHID: swh:1:snp:af3d189d1f742c42239460da00acc92c1d867427
Mar 29 12:45:28 granet swh[2170078]: at org.softwareheritage.graph.maps.NodeIdMap.getNodeId(NodeIdMap.java:152)
Mar 29 12:45:28 granet swh[2170078]: at org.softwareheritage.graph.maps.NodeIdMap.getNodeId(NodeIdMap.java:157)
Mar 29 12:45:28 granet swh[2170078]: at org.softwareheritage.graph.Graph.getNodeId(Graph.java:280)
Mar 29 12:45:28 granet swh[2170078]: at org.softwareheritage.graph.Entry$QueryHandler.visit_nodes(Entry.java:136)
Mar 29 12:45:28 granet swh[2170078]: at jdk.internal.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
Mar 29 12:45:28 granet swh[2170078]: at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Mar 29 12:45:28 granet swh[2170078]: at java.base/java.lang.reflect.Method.invoke(Method.java:566)
Mar 29 12:45:28 granet swh[2170078]: at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
Mar 29 12:45:28 granet swh[2170078]: at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
Mar 29 12:45:28 granet swh[2170078]: at py4j.Gateway.invoke(Gateway.java:282)
Mar 29 12:45:28 granet swh[2170078]: at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
Mar 29 12:45:28 granet swh[2170078]: at py4j.commands.CallCommand.execute(CallCommand.java:79)
Mar 29 12:45:28 granet swh[2170078]: at py4j.GatewayConnection.run(GatewayConnection.java:238)
Mar 29 12:45:28 granet swh[2170078]: at java.base/java.lang.Thread.run(Thread.java:829)
Mar 29 12:45:28 granet swh[2170078]: ERROR:aiohttp.server:Error handling request
Mar 29 12:45:28 granet swh[2170078]: Traceback (most recent call last):
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 418, in start
Mar 29 12:45:28 granet swh[2170078]: resp = await task
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/sentry_sdk/integrations/aiohttp.py", line 123, in sentry_app_handle
Mar 29 12:45:28 granet swh[2170078]: reraise(*_capture_exception(hub))
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/sentry_sdk/_compat.py", line 54, in reraise
Mar 29 12:45:28 granet swh[2170078]: raise value
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/sentry_sdk/integrations/aiohttp.py", line 113, in sentry_app_handle
Mar 29 12:45:28 granet swh[2170078]: response = await old_handle(self, request)
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/aiohttp/web_app.py", line 458, in _handle
Mar 29 12:45:28 granet swh[2170078]: resp = await handler(request)
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 119, in impl
Mar 29 12:45:28 granet swh[2170078]: return await handler(request)
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/aiohttp_utils/negotiation.py", line 224, in middleware
Mar 29 12:45:28 granet swh[2170078]: render_result = renderer(request, data)
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/swh/core/api/asynchronous.py", line 160, in <lambda>
Mar 29 12:45:28 granet swh[2170078]: request, data, extra_encoders=self.extra_type_encoders
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/swh/core/api/asynchronous.py", line 37, in render_msgpack
Mar 29 12:45:28 granet swh[2170078]: return msgpack_dumps(data, extra_encoders=extra_encoders)
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/swh/core/api/serializers.py", line 275, in msgpack_dumps
Mar 29 12:45:28 granet swh[2170078]: default=encode_types,
Mar 29 12:45:28 granet swh[2170078]: File "/opt/swhgraph_venv/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb
Mar 29 12:45:28 granet swh[2170078]: return Packer(**kwargs).pack(o)
Mar 29 12:45:28 granet swh[2170078]: File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
Mar 29 12:45:28 granet swh[2170078]: File "msgpack/_packer.pyx", line 298, in msgpack._cmsgpack.Packer.pack
Mar 29 12:45:28 granet swh[2170078]: File "msgpack/_packer.pyx", line 295, in msgpack._cmsgpack.Packer.pack
Mar 29 12:45:28 granet swh[2170078]: File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
Mar 29 12:45:28 granet swh[2170078]: File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
Mar 29 12:45:28 granet swh[2170078]: File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer._pack
Mar 29 12:45:28 granet swh[2170078]: TypeError: can not serialize 'JavaObject' objectMigrated from T4115 (view on Phabricator)