Move aggregate-dataset, blobs-datasets, filenames, and origin contributors to swh-datasets,
and provenance generation to its own repository

Antoine Lambert authored 3 months ago and vlorentz committed 2 months ago

Some parameters have been renamed so some examples were not working anymore.

Antoine Lambert authored 3 months ago and vlorentz committed 3 months ago

The softwaheritage S3 bucket is public so that option is required
to download graph dataset files with the aws CLI.

Antoine Lambert authored 3 months ago and vlorentz committed 3 months ago

It is required to run swh-graph Python tests.

The --locked option forces Cargo to use the lockfile declared by swh-graph,
which was carefully written to avoid conflicts between two Arrow versions
that are used as transitive dependencies through different packages.

When Cargo tries to update one, it ends up with incompatible versions,
causing a compilation failure.

Currently, this only happens when enabling the `orc` features (which
commands in the documentation do not enable), but similar issues could
appear in the future with the default versions.

They used to be installed by everyone as part of installing the gRPC server,
but are not since we moved the gRPC server to its own crate

vlorentz authored 7 months ago and vlorentz committed 7 months ago

This is most of the time spent in the Traverse() endpoint, but it wasn't counted so far
because we logged the request as soon as we start sending the response, even though
this happens before the traversal even starts.

This is easier to install, as there is no need for users to remember
--features=grpc-server.

This also avoids declaring every dependency of the gRPC server as 'optional' and
adding it to the list of dependencies of the 'grpc-server' feature; and I expect
this set of dependencies to get larger now that the gRPC server is starting to
be used in production (eg. statsd metrics).

Finally, this allows Cargo to remove building the gRPC server from the critical
path to building swh_graph_provenance when building all crates (eg. in CI),
which should make it a little faster to build.

It was a bug in the Java gRPC server to allow it without the "node." prefix because
FieldMasks should match the fully-qualified field name in the output, and these
two endpoints return a Path (with a Node field) instead of directly a Node stream
like Traverse does.

This makes the EDGE_LABELS run time go from 2.5 days to 1 day; and TRANSPOSE_EDGE_LABELS probably from 3.5 to 1-1.5 day.

1. seems to be the de facto standard
2. already used by webgraph
3. prints which module emitted each log line
4. can be configured to default to INFO level, so we don't need to pass
   -vv to every Rust executable to get progress reporting
5. env vars allow configuring even when commands are called through Luigi

This wasn't the right syntax for commenting out a block of text in ReST

It is not available in Rust, and contained a reference to the now-removed
documentation of the Java API

vlorentz authored 11 months ago and vlorentz committed 8 months ago

These endpoints are currently not implemented.

Based on observing ardumont read it and try to follow it.

The tutorial is adapted from the existing Java tutorial.

build-essentials -> build-essential
protobuf-compiler is required by cargo install swh-graph

When cgroups are available