Instead of recomputing a subset of it using a less efficient algorithm, which takes
days to find origins for the last revisions

Which creates an export, compresses a graph, uploads both to S3, and computes
a couple of derived datasets.

This may be used as an entry point for the task that needs to run every ~3 months

It should be needed in `src/properties/*.rs` to serialize/deserialize `Vec*` structs,
but for some reason it isn't for Rust >= 1.82 (or >= 1.81?)

Add simple command to print the names of the Software Heritage
graph datasets that can be downloaded with the following command:

$ swh graph download --name <dataset_name> <target_directory>

vlorentz authored 3 months ago and vlorentz committed 3 months ago

The cgroup was created when the Command is initialized, and calling .run() on it
twice caused creation of two RunningCommand objects.

As RunningCommand is responsible for deleting the cgroup (and polling information
from it), this means that one of the RunningCommand deleted the cgroup before the
other one tried to read it, causing a crash.

This happens for example in CountPaths when direction=backward, because the
zstdcat command running on the topological order is used twice (once to read the
header, once to read the rest of the file)

Include update to `pyo3-0.23.0`

The --locked option forces Cargo to use the lockfile declared by swh-graph,
which was carefully written to avoid conflicts between two Arrow versions
that are used as transitive dependencies through different packages.

When Cargo tries to update one, it ends up with incompatible versions,
causing a compilation failure.

Currently, this only happens when enabling the `orc` features (which
commands in the documentation do not enable), but similar issues could
appear in the future with the default versions.

This will be useful to generate graphs for test cases

Otherwise it captures `'de` though `D` and that makes the function pretty hard to use, as its return value cannot alive the deserializer.

In this case, serde_json only yields owned Strings, so
`<&str>::deserialize()` fails. We need a visitor, which has a default
implementation of visit_string that defers to visit_str, so it works.

This reverts commit 3478faf8.

We need orc-rust < 0.4.0 for now, which does not depend on the same
arrow version as ar_row/dataset-writer

This reverts commit 260d641a.

orc-rust v0.4.0 and v0.5.0 break ar_row:

* https://github.com/datafusion-contrib/orc-rust/pull/13
* https://github.com/datafusion-contrib/datafusion-orc/issues/137

vlorentz authored 5 months ago and vlorentz committed 5 months ago

This can be used to get a textual representation of small graphs, which can then
be shared and used in tests or examples.

They will be used to generate generates databases for the swh-provenance-grpc's tests