This reverts commit d9abf3c2.

So we can publish swh-graph on crates.io

For some reason, datagrams are not always received in order on Jenkins,
even though they go through loopback interfaces should provide this guarantee.

This commit removes the check for datagram order, as it does not matter much
in practice, anyway.

This will be used by swh-provenance, as its tables cannot be queried
efficiently by Athena, and it's faster to upload all tables in parallel
(which we is harder if each table has to have its own Task class).

Graphs compressed with the Rust implementation (2023-09-06 and newer) sort labels
by name, while the Java implementation (2022-12-07 and older) sorted them by
base64(name).

mimalloc is already used in swh-provenance, provides performance on par with jemalloc
(even better by some reports, though with more memory overhead), and compiles
significantly faster than tikv-jemalloctor.

When adding end-to-end tests for graph compression, the new steps were
not integrated in the example dataset generation script, making it not
capable of generating a compressed example graph.

For example, swh-datasets needed this hack to get a `Rust` class that
will use its local executables: https://gitlab.softwareheritage.org/swh/devel/swh-datasets/-/blob/1aab21ff2ea9bc394210d92bec5d518a08edbd8c/swh/datasets/shell.py

with this change, it can be simplified to only adding the
`base_rust_executable_dir` to the arguments passed to the constructor.

Co-authored-by: Valentin Lorentz <vlorentz@softwareheritage.org>

Bump development tools: mypy, codespell, isort, ...

Move all tools configuration in pyproject.toml.

Remove no longer needed mypy overrides.

Move aggregate-dataset, blobs-datasets, filenames, and origin contributors to swh-datasets,
and provenance generation to its own repository

Antoine Lambert authored 2 months ago and vlorentz committed 2 months ago

The dockerfile for swh-graph was outdated as it was still expected
to use removed java components that have been rewritten in rust.

Antoine Lambert authored 2 months ago and vlorentz committed 2 months ago

Some parameters have been renamed so some examples were not working anymore.

instead of crashing, because the JOIN did not find a sha1

Antoine Lambert authored 2 months ago and vlorentz committed 2 months ago

The softwaheritage S3 bucket is public so that option is required
to download graph dataset files with the aws CLI.

For some reason, this now happens all the time on my machine.

This may also be the cause of the random hangs we have had on Jenkins

Otherwise, it crashes with: RuntimeError: The grpc package installed is at version 1.67.0, but the generated code in swh/graph/grpc/swhgraph_pb2_grpc.py depends on grpcio>=1.69.0. Please upgrade your grpc module to grpcio>=1.69.0 or downgrade your generated code using grpcio-tools<=1.67.0.