For example, swh-datasets needed this hack to get a `Rust` class that
will use its local executables: https://gitlab.softwareheritage.org/swh/devel/swh-datasets/-/blob/1aab21ff2ea9bc394210d92bec5d518a08edbd8c/swh/datasets/shell.py

with this change, it can be simplified to only adding the
`base_rust_executable_dir` to the arguments passed to the constructor.

Co-authored-by: Valentin Lorentz <vlorentz@softwareheritage.org>

Bump development tools: mypy, codespell, isort, ...

Move all tools configuration in pyproject.toml.

Remove no longer needed mypy overrides.

Move aggregate-dataset, blobs-datasets, filenames, and origin contributors to swh-datasets,
and provenance generation to its own repository

Antoine Lambert authored 2 months ago and vlorentz committed 2 months ago

The dockerfile for swh-graph was outdated as it was still expected
to use removed java components that have been rewritten in rust.

Antoine Lambert authored 2 months ago and vlorentz committed 2 months ago

Some parameters have been renamed so some examples were not working anymore.

instead of crashing, because the JOIN did not find a sha1

Antoine Lambert authored 2 months ago and vlorentz committed 2 months ago

The softwaheritage S3 bucket is public so that option is required
to download graph dataset files with the aws CLI.

For some reason, this now happens all the time on my machine.

This may also be the cause of the random hangs we have had on Jenkins

Otherwise, it crashes with: RuntimeError: The grpc package installed is at version 1.67.0, but the generated code in swh/graph/grpc/swhgraph_pb2_grpc.py depends on grpcio>=1.69.0. Please upgrade your grpc module to grpcio>=1.69.0 or downgrade your generated code using grpcio-tools<=1.67.0.

This allows to remove version constraints on some dependencies (namely
mypy-protobuf).

Antoine Lambert authored 2 months ago and vlorentz committed 2 months ago

It is required to run swh-graph Python tests.

Instead of recomputing a subset of it using a less efficient algorithm, which takes
days to find origins for the last revisions

Which creates an export, compresses a graph, uploads both to S3, and computes
a couple of derived datasets.

This may be used as an entry point for the task that needs to run every ~3 months