Add simple command to print the names of the Software Heritage
graph datasets that can be downloaded with the following command:

$ swh graph download --name <dataset_name> <target_directory>

vlorentz authored 3 months ago and vlorentz committed 3 months ago

The cgroup was created when the Command is initialized, and calling .run() on it
twice caused creation of two RunningCommand objects.

As RunningCommand is responsible for deleting the cgroup (and polling information
from it), this means that one of the RunningCommand deleted the cgroup before the
other one tried to read it, causing a crash.

This happens for example in CountPaths when direction=backward, because the
zstdcat command running on the topological order is used twice (once to read the
header, once to read the rest of the file)

Include update to `pyo3-0.23.0`

The --locked option forces Cargo to use the lockfile declared by swh-graph,
which was carefully written to avoid conflicts between two Arrow versions
that are used as transitive dependencies through different packages.

When Cargo tries to update one, it ends up with incompatible versions,
causing a compilation failure.

Currently, this only happens when enabling the `orc` features (which
commands in the documentation do not enable), but similar issues could
appear in the future with the default versions.

This will be useful to generate graphs for test cases

Otherwise it captures `'de` though `D` and that makes the function pretty hard to use, as its return value cannot alive the deserializer.

In this case, serde_json only yields owned Strings, so
`<&str>::deserialize()` fails. We need a visitor, which has a default
implementation of visit_string that defers to visit_str, so it works.

This reverts commit 3478faf8.

We need orc-rust < 0.4.0 for now, which does not depend on the same
arrow version as ar_row/dataset-writer

This reverts commit 260d641a.

orc-rust v0.4.0 and v0.5.0 break ar_row:

* https://github.com/datafusion-contrib/orc-rust/pull/13
* https://github.com/datafusion-contrib/datafusion-orc/issues/137

vlorentz authored 5 months ago and vlorentz committed 5 months ago

This can be used to get a textual representation of small graphs, which can then
be shared and used in tests or examples.

They will be used to generate generates databases for the swh-provenance-grpc's tests

It is absent from graph.stats files computed by the Rust pipeline

This file used to be generated by the Java compression pipeline

It seems to be slightly faster on ZSTD-compressed filesystems, as it does not need to read `graph.pthash.order` and `graph.node2swhid.bin` all the time to parse the input.