Draft: Update the example dataset to add a few dandling nodes in the graph

Add a few contents and directories that are not referenced, to help testing edge cases.

Jenkins job DGRPH/gitlab-builds #1750 failed .
See Console Output and Coverage Report for more details.

I'm afraid you can't just update the dataset like that. You need to regenerate the compressed graph (swh graph compress CLI) then update every single test to take the new nodes into account, update statistics, and tests that rely on the MPH+permutation to have a specific order.

#4814

So using generate_dataset --compress does produce the compresed dataset, but there are a few add/remove elements of the diffstat I am not sure about:

	new file:   swh/graph/example_dataset/compressed/example-bfs.roots.txt
	deleted:    swh/graph/example_dataset/compressed/example.node2type.map
	deleted:    swh/graph/example_dataset/compressed/example.nodes.csv.zst
	deleted:    swh/graph/example_dataset/compressed/example.property.content.is_skipped.bits
	deleted:    swh/graph/example_dataset/compressed/meta/export.json

• Is the meta/export.json necessary somewhere? How is it generated?
• are the other deleted files an issue for the purpose of a usable example dataset?

Looks like the node2type.map is required for the java tool... How do I regen it?

meta/export.json is generated by the Luigi pipeline, which is what I used to compress the graph so far.

example-bfs.roots.txt should be removed, I forgot to add it to the list of files to clean up at the end.

node2type.map is not needed anymore as I updated the Java code to support the new format (node2type.bin). Try re-running make java? See https://gitlab.softwareheritage.org/swh/devel/swh-graph/-/blob/master/java/src/main/java/org/softwareheritage/graph/maps/NodeTypesMap.java

property.content.is_skipped.bits is required, I'm afraid. You can either revert ef28a3f2 or run swh graph reindex to create it from property.content.is_skipped.bin

nodes.csv.zst is not generated anymore, I don't think anything uses it.

thanks for the feedback; so the only missing piece is the property.content.is_skipped.bits; I should add the required step to the "generate_dataset.py" script right?

I guess. Ideally this won't last for long. Could you add a comment in webgraph.py next to the NODE_PROPERTIES command definition so I don't forget when I switch it back to Rust?

fixed all the missing parts I think. Did not modified the generate_dataset script yet (not sure I will)

Also, is it ok to use swh/graph/example_dataset/compressed/example.stats in the tests (instead of hardcoded values)?

use it for what?

to compare stats values (eg. https://gitlab.softwareheritage.org/swh/devel/swh-graph/-/blob/master/swh/graph/tests/test_cli.py?ref_type=heads#L66 )

This test apparently checks the compression pipeline did not miss (or invent new) nodes; so this would defeat the point of the test.

the computed value depends on what the example_dataset is made of, so hardcoded values will need to be updated each time the dataset is updated; so the idea of using expected stats as provided by the dataset itself.

then compute the stats from example_dataset/__init__.py instead of reading them from compressed/example.stats.

why? why add more code for that? the idea is to check there is no subsequent change of behavior the compression pipeline, so using the stats file is semantically/practically equivalent to using handwritten values in the test file IMHO.

The .stats and .properties files are both generated by the compression pipeline (at different steps). If the compression pipeline had a bug writing nodes=0 in .stats, then .properties would contain nodes=0 too and the test would not catch the bug.

Ah, I get it. You want to compare the nodes= value in the graph compressed by test_cli.py with the nodes= value in the graph in swh/graph/example_dataset/compressed/. That's fine.

added 1 commit

• 468f69ac - Update the example dataset to add a few dandling nodes in the graph

Compare with previous version

Jenkins job DGRPH/gitlab-builds #1765 failed .
See Console Output and Coverage Report for more details.

marked this merge request as draft

mentioned in issue #4844