Skip to content

compressed_graph: Fix data race to .obl files in Transpose command

vlorentz requested to merge vlorentz/swh-graph:obl-race into master
$ cat /tmp/pytest-of-dev/pytest-208/test_compressgraph_100_0/compressed_graph/logs/graph-1676375678050-transpose.log
Starting compression step TRANSPOSE at 2023-02-14 12:54:38.050488
Running: java it.unimi.dsi.big.webgraph.Transform transposeOffline /tmp/pytest-of-dev/pytest-208/test_compressgraph_100_0/compressed_graph/graph /tmp/pytest-of-dev/pytest-208/test_compressgraph_100_0/compressed_graph/graph-transposed 1000 /tmp/pytest-of-dev/pytest-208/test_compressgraph_100_0/compressed_graph/tmp
Picked up JAVA_TOOL_OPTIONS: -Xmx30120210432 -XX:PretenureSizeThreshold=512M -XX:MaxNewSize=4G -XX:+UseLargePages -XX:+UseTransparentHugePages -XX:+UseNUMA -XX:+UseTLAB -XX:+ResizeTLAB -Dlogback.configurationFile=/tmp/pytest-of-dev/pytest-208/test_compressgraph_100_0/compressed_graph/tmp/logback.xml -Djava.io.tmpdir=/tmp/pytest-of-dev/pytest-208/test_compressgraph_100_0/compressed_graph/tmp
2023-02-14 12:54:38,375 201 INFO [main] i.u.d.b.w.Transform - Loading graph...
2023-02-14 12:54:38,376 202 INFO [main] i.u.d.b.w.Transform - Completed.
2023-02-14 12:54:38,380 206 INFO [main] i.u.d.b.w.Transform - Elapsed: 1ms [27 bytes, ∞ bytes/s, 0.00 ns/byte]
2023-02-14 12:54:38,380 206 INFO [main] i.u.d.b.w.Transform - Loading offsets...
Exception in thread "main" java.io.EOFException
	at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2911)
	at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3406)
	at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:932)
	at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:375)
	at it.unimi.dsi.fastutil.io.BinIO.loadObject(BinIO.java:108)
	at it.unimi.dsi.big.webgraph.BVGraph.loadInternal(BVGraph.java:1495)
	at it.unimi.dsi.big.webgraph.BVGraph.load(BVGraph.java:1300)
	at it.unimi.dsi.big.webgraph.BVGraph.load(BVGraph.java:1337)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at it.unimi.dsi.big.webgraph.ImmutableGraph.load(ImmutableGraph.java:703)
	at it.unimi.dsi.big.webgraph.ImmutableGraph.load(ImmutableGraph.java:643)
	at it.unimi.dsi.big.webgraph.Transform.load(Transform.java:1997)
	at it.unimi.dsi.big.webgraph.Transform.main(Transform.java:2182)
Edited by vlorentz

Merge request reports