Bump development tools: mypy, codespell, isort, ...

Move all tools configuration in pyproject.toml.

Remove no longer needed mypy overrides.

Move aggregate-dataset, blobs-datasets, filenames, and origin contributors to swh-datasets,
and provenance generation to its own repository

Antoine Lambert authored 2 months ago and vlorentz committed 2 months ago

The dockerfile for swh-graph was outdated as it was still expected
to use removed java components that have been rewritten in rust.

Antoine Lambert authored 2 months ago and vlorentz committed 2 months ago

Some parameters have been renamed so some examples were not working anymore.

instead of crashing, because the JOIN did not find a sha1

Antoine Lambert authored 2 months ago and vlorentz committed 2 months ago

The softwaheritage S3 bucket is public so that option is required
to download graph dataset files with the aws CLI.

For some reason, this now happens all the time on my machine.

This may also be the cause of the random hangs we have had on Jenkins

Otherwise, it crashes with: RuntimeError: The grpc package installed is at version 1.67.0, but the generated code in swh/graph/grpc/swhgraph_pb2_grpc.py depends on grpcio>=1.69.0. Please upgrade your grpc module to grpcio>=1.69.0 or downgrade your generated code using grpcio-tools<=1.67.0.

This allows to remove version constraints on some dependencies (namely
mypy-protobuf).

Antoine Lambert authored 2 months ago and vlorentz committed 2 months ago

It is required to run swh-graph Python tests.

Instead of recomputing a subset of it using a less efficient algorithm, which takes
days to find origins for the last revisions

Which creates an export, compresses a graph, uploads both to S3, and computes
a couple of derived datasets.

This may be used as an entry point for the task that needs to run every ~3 months

It should be needed in `src/properties/*.rs` to serialize/deserialize `Vec*` structs,
but for some reason it isn't for Rust >= 1.82 (or >= 1.81?)

Add simple command to print the names of the Software Heritage
graph datasets that can be downloaded with the following command:

$ swh graph download --name <dataset_name> <target_directory>

vlorentz authored 3 months ago and vlorentz committed 3 months ago

The cgroup was created when the Command is initialized, and calling .run() on it
twice caused creation of two RunningCommand objects.

As RunningCommand is responsible for deleting the cgroup (and polling information
from it), this means that one of the RunningCommand deleted the cgroup before the
other one tried to read it, causing a crash.

This happens for example in CountPaths when direction=backward, because the
zstdcat command running on the topological order is used twice (once to read the
header, once to read the rest of the file)