graph popular-4-shell dataset issues
Issue is about a missing graph.stats [1].
The full listing of the content of the dataset retrieved from s3 through swh graph download cli [2]
[1]
2024-10-16T07:58:52.278376585Z swh graph -C /etc/swh/config.yml rpc-serve -h 0.0.0.0 -p 5009 -g /srv/graph/2024-08-23_popular-4-shell/compressed/graph
2024-10-16T07:58:52.497311726Z WARNING:swh.core.sentry:Sentry DSN not provided, events will not be sent.
2024-10-16T07:58:52.657811199Z INFO:swh.graph.grpc_server:Starting gRPC server: /usr/local/bin/swh-graph-grpc-serve --bind ':50091' /srv/graph/2024-08-23_popular-4-shell/compressed/graph
2024-10-16T07:58:52.661146554Z [2024-10-16T07:58:52Z INFO swh_graph_grpc_serve] Loading graph
2024-10-16T07:58:52.662285655Z [2024-10-16T07:58:52Z INFO swh_graph_grpc_serve] Starting server
2024-10-16T07:58:53.664697506Z [2024-10-16T07:58:53Z ERROR swh_graph_grpc_server] Could not open /srv/graph/2024-08-23_popular-4-shell/compressed/graph.stats: No such file or directory (os error 2)
2024-10-16T07:58:53.664728614Z [2024-10-16T07:58:53Z INFO swh_graph_grpc_server] 200 OK - http://localhost:50091/swh.graph.TraversalService/Stats - 251.705µs
2024-10-16T07:58:53.665382801Z Started GRPC using dataset from /srv/graph/2024-08-23_popular-4-shell/compressed/graph
2024-10-16T07:58:53.665393490Z ['/usr/local/bin/swh-graph-grpc-serve', '--bind', ':50091', '/srv/graph/2024-08-23_popular-4-shell/compressed/graph']
2024-10-16T07:58:53.670134058Z Traceback (most recent call last):
2024-10-16T07:58:53.670154696Z File "/opt/swh/.local/bin/swh", line 8, in <module>
2024-10-16T07:58:53.670164394Z sys.exit(main())
2024-10-16T07:58:53.670183519Z ^^^^^^
2024-10-16T07:58:53.670193077Z File "/opt/swh/.local/lib/python3.11/site-packages/swh/core/cli/__init__.py", line 175, in main
2024-10-16T07:58:53.670221069Z return swh(auto_envvar_prefix="SWH")
2024-10-16T07:58:53.670224986Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-16T07:58:53.670228953Z File "/opt/swh/.local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
2024-10-16T07:58:53.670310694Z return self.main(*args, **kwargs)
2024-10-16T07:58:53.670314411Z ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-16T07:58:53.670317657Z File "/opt/swh/.local/lib/python3.11/site-packages/click/core.py", line 1078, in main
2024-10-16T07:58:53.670362089Z rv = self.invoke(ctx)
2024-10-16T07:58:53.670366337Z ^^^^^^^^^^^^^^^^
2024-10-16T07:58:53.670370615Z File "/opt/swh/.local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
2024-10-16T07:58:53.670499703Z return _process_result(sub_ctx.command.invoke(sub_ctx))
2024-10-16T07:58:53.670504081Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-16T07:58:53.670508409Z File "/opt/swh/.local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
2024-10-16T07:58:53.670606911Z return _process_result(sub_ctx.command.invoke(sub_ctx))
2024-10-16T07:58:53.670611188Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-16T07:58:53.670615607Z File "/opt/swh/.local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
2024-10-16T07:58:53.670696275Z return ctx.invoke(self.callback, **ctx.params)
2024-10-16T07:58:53.670700653Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-16T07:58:53.670704871Z File "/opt/swh/.local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
2024-10-16T07:58:53.670723956Z return __callback(*args, **kwargs)
2024-10-16T07:58:53.670728204Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-16T07:58:53.670732502Z File "/opt/swh/.local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
2024-10-16T07:58:53.670752800Z return f(get_current_context(), *args, **kwargs)
2024-10-16T07:58:53.670757118Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-16T07:58:53.670777105Z File "/opt/swh/.local/lib/python3.11/site-packages/swh/graph/cli.py", line 163, in serve
2024-10-16T07:58:53.670789187Z aiohttp.web.run_app(app, host=host, port=port)
2024-10-16T07:58:53.670793825Z File "/opt/swh/.local/lib/python3.11/site-packages/aiohttp/web.py", line 526, in run_app
2024-10-16T07:58:53.670835673Z loop.run_until_complete(main_task)
2024-10-16T07:58:53.670839199Z File "/usr/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
2024-10-16T07:58:53.670911803Z return future.result()
2024-10-16T07:58:53.670930968Z ^^^^^^^^^^^^^^^
2024-10-16T07:58:53.670940837Z grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
2024-10-16T07:58:53.670949433Z status = StatusCode.INTERNAL
2024-10-16T07:58:53.670957718Z details = "Could not open .stats file"
2024-10-16T07:58:53.670966754Z debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"Could not open .stats file", grpc_status:13, created_time:"2024-10-16T07:58:53.664721851+00:00"}"
2024-10-16T07:58:53.670975150Z >
2024-10-16T07:58:53.763054656Z Error in sys.excepthook:
2024-10-16T07:58:53.763086315Z
2024-10-16T07:58:53.763096483Z Original exception was:
2024-10-16T08:00:14.98372448Z Stream closed EOF for swh/graph-rpc-popular-20240823-7857d4964f-lj968 (graph-rpc-popular-20240823)[2]
root@graph-rpc-popular-20240823-b76bcf5f7-m27m7:/# ls -lah /srv/dataset/2024-08-23_popular-4-shell/compressed/
total 79M
drwxr-xr-x 1 1000 1000 2.4K Oct 16 07:54 .
drwxr-xr-x 1 1000 1000 20 Oct 16 07:52 ..
-rw-r--r-- 1 1000 1000 0 Oct 16 07:54 .graph-is-initialized
-rw-r--r-- 1 1000 1000 359K Oct 16 07:53 graph-labelled.ef
-rw-r--r-- 1 1000 1000 280K Oct 16 07:53 graph-labelled.labeloffsets
-rw-r--r-- 1 1000 1000 24M Oct 16 07:53 graph-labelled.labels
-rw-r--r-- 1 1000 1000 170 Oct 16 07:53 graph-labelled.properties
-rw-r--r-- 1 1000 1000 360K Oct 16 07:53 graph-transposed-labelled.ef
-rw-r--r-- 1 1000 1000 295K Oct 16 07:53 graph-transposed-labelled.labeloffsets
-rw-r--r-- 1 1000 1000 24M Oct 16 07:54 graph-transposed-labelled.labels
-rw-r--r-- 1 1000 1000 181 Oct 16 07:53 graph-transposed-labelled.properties
-rw-r--r-- 1 1000 1000 230K Oct 16 07:53 graph-transposed.ef
-rw-r--r-- 1 1000 1000 1.3M Oct 16 07:53 graph-transposed.graph
-rw-r--r-- 1 1000 1000 269K Oct 16 07:53 graph-transposed.offsets
-rw-r--r-- 1 1000 1000 199 Oct 16 07:53 graph-transposed.properties
-rw-r--r-- 1 1000 1000 8 Oct 16 07:53 graph.edges.count.txt
-rw-r--r-- 1 1000 1000 126 Oct 16 07:53 graph.edges.stats.txt
-rw-r--r-- 1 1000 1000 247K Oct 16 07:53 graph.ef
-rw-r--r-- 1 1000 1000 1.8M Oct 16 07:53 graph.graph
-rw-r--r-- 1 1000 1000 6 Oct 16 07:53 graph.labels.count.txt
-rw-r--r-- 1 1000 1000 90K Oct 16 07:53 graph.labels.csv.zst
-rw-r--r-- 1 1000 1000 304K Oct 16 07:53 graph.labels.fcl.bytearray
-rw-r--r-- 1 1000 1000 35K Oct 16 07:53 graph.labels.fcl.pointers
-rw-r--r-- 1 1000 1000 16 Oct 16 07:53 graph.labels.fcl.properties
-rw-r--r-- 1 1000 1000 6.5K Oct 16 07:53 graph.labels.pthash
-rw-r--r-- 1 1000 1000 140K Oct 16 07:53 graph.labels.pthash.order
-rw-r--r-- 1 1000 1000 5.2M Oct 16 07:53 graph.node2swhid.bin
-rw-r--r-- 1 1000 1000 91K Oct 16 07:53 graph.node2type.bin
-rw-r--r-- 1 1000 1000 7 Oct 16 07:53 graph.nodes.count.txt
-rw-r--r-- 1 1000 1000 53 Oct 16 07:53 graph.nodes.stats.txt
-rw-r--r-- 1 1000 1000 266K Oct 16 07:53 graph.offsets
-rw-r--r-- 1 1000 1000 5 Oct 16 07:53 graph.persons.count.txt
-rw-r--r-- 1 1000 1000 304K Oct 16 07:53 graph.persons.csv.zst
-rw-r--r-- 1 1000 1000 3.8K Oct 16 07:53 graph.persons.pthash
-rw-r--r-- 1 1000 1000 199 Oct 16 07:53 graph.properties
-rw-r--r-- 1 1000 1000 969K Oct 16 07:53 graph.property.author_id.bin
-rw-r--r-- 1 1000 1000 1.9M Oct 16 07:53 graph.property.author_timestamp.bin
-rw-r--r-- 1 1000 1000 485K Oct 16 07:53 graph.property.author_timestamp_offset.bin
-rw-r--r-- 1 1000 1000 969K Oct 16 07:53 graph.property.committer_id.bin
-rw-r--r-- 1 1000 1000 1.9M Oct 16 07:53 graph.property.committer_timestamp.bin
-rw-r--r-- 1 1000 1000 485K Oct 16 07:53 graph.property.committer_timestamp_offset.bin
-rw-r--r-- 1 1000 1000 31K Oct 16 07:53 graph.property.content.is_skipped.bits
-rw-r--r-- 1 1000 1000 1.9M Oct 16 07:53 graph.property.content.length.bin
-rw-r--r-- 1 1000 1000 5.7M Oct 16 07:53 graph.property.message.bin
-rw-r--r-- 1 1000 1000 1.9M Oct 16 07:53 graph.property.message.offset.bin
-rw-r--r-- 1 1000 1000 3.8K Oct 16 07:53 graph.property.tag_name.bin
-rw-r--r-- 1 1000 1000 1.9M Oct 16 07:53 graph.property.tag_name.offset.bin
-rw-r--r-- 1 1000 1000 88K Oct 16 07:53 graph.pthash
-rw-r--r-- 1 1000 1000 1.9M Oct 16 07:53 graph.pthash.order
drwxr-xr-x 1 1000 1000 2.4K Oct 16 07:53 logs
drwxr-xr-x 1 1000 1000 54 Oct 16 07:54 meta