This is much more performant on the dir layer: it only takes 27 hours on
swh1.enst.fr instead of an ETA of one or two years, most of it spent on
this particular line:

```
LazyLongIterator successorAncestors = graph.successors(successorNodeId);
```

even when replacing all the code that used `successorAncestors`, it was
still the major cause of the huge expected runtime.

Jérémy Bobbio (Lunar) authored 2 years ago and vlorentz committed 2 years ago

protobuf earlier 4.12.* versions crashed when `swhgraph_pb2.py`
was discovered by pytest. This has been fixed in the 4.12.11 release.
See: https://github.com/protocolbuffers/protobuf/issues/10151

Preventing protobuf 4.12.* to be used makes `pip` install grpcio-tools
version 1.49.0 which fails to build on Debian bookworm. In order to
allow more fixed versions of grpcio-tools to be used, we bump the
dependency on protobuf to version 4.12.11 or later.

vlorentz authored 2 years ago and vlorentz committed 2 years ago

to help debugging

This will be used as a metric for 'popularity' of directories,
which will be used to weigh results of PopularContents (which
counts the most popular names used to refer to each content)

vlorentz authored 2 years ago and vlorentz committed 2 years ago

It caused NodeProperties to OOM because it tried to allocate array 22 times
larger than needed.

vlorentz authored 2 years ago and vlorentz committed 2 years ago

It is 5 times faster

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

vlorentz authored 2 years ago and vlorentz committed 2 years ago

To be consistent with other CSV files in the license dataset.

Because the main table already contains (origin_id, person_id) pairs,
it is redundant to repeat every pair in this file.

They tend to be forgotten; this will make sure that running 'tox' will
run Java tests as well as Python tests.

It will also make it easier to run Java tests on the CI

two paths + reimport tasks at the package level so they are automatically
picked up by luigi without passing all module names on the CLI

whitelist_externals is deprecated since v3.18, and removed since v4.0

RPCServerApp inherits from aiohttp.web.Application, and GraphServerApp
does not use any of the features added by RPCServerApp