Related to swh/meta#4960

vlorentz authored 2 years ago and vlorentz committed 2 years ago

Copying it 1k or 10k for each thread is wasteful, even though they are
lightweight copies.

This brings the runtime down from 133 to ~100 hours.

Jérémy Bobbio (Lunar) authored 2 years ago and Jérémy Bobbio (Lunar) committed 2 years ago

Related to swh/meta#4959

Each thread handled one 96th of the node id range. But nodes
are not homogeneously randomized across that range, so some
threads had a lot more work to do than others, causing them
to end weeks after, while most CPU cores idled.

By splitting the range this way, threads should have more
homogeneous workloads.

This is much more performant on the dir layer: it only takes 27 hours on
swh1.enst.fr instead of an ETA of one or two years, most of it spent on
this particular line:

```
LazyLongIterator successorAncestors = graph.successors(successorNodeId);
```

even when replacing all the code that used `successorAncestors`, it was
still the major cause of the huge expected runtime.

Jérémy Bobbio (Lunar) authored 2 years ago and vlorentz committed 2 years ago

protobuf earlier 4.12.* versions crashed when `swhgraph_pb2.py`
was discovered by pytest. This has been fixed in the 4.12.11 release.
See: https://github.com/protocolbuffers/protobuf/issues/10151

Preventing protobuf 4.12.* to be used makes `pip` install grpcio-tools
version 1.49.0 which fails to build on Debian bookworm. In order to
allow more fixed versions of grpcio-tools to be used, we bump the
dependency on protobuf to version 4.12.11 or later.

vlorentz authored 2 years ago and vlorentz committed 2 years ago

to help debugging

This will be used as a metric for 'popularity' of directories,
which will be used to weigh results of PopularContents (which
counts the most popular names used to refer to each content)

vlorentz authored 2 years ago and vlorentz committed 2 years ago

It caused NodeProperties to OOM because it tried to allocate array 22 times
larger than needed.

vlorentz authored 2 years ago and vlorentz committed 2 years ago

It is 5 times faster

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

vlorentz authored 2 years ago and vlorentz committed 2 years ago

To be consistent with other CSV files in the license dataset.

Because the main table already contains (origin_id, person_id) pairs,
it is redundant to repeat every pair in this file.

They tend to be forgotten; this will make sure that running 'tox' will
run Java tests as well as Python tests.

It will also make it easier to run Java tests on the CI