Luigi workflows contain plenty of ad-hoc bash scripts like 'zstdcat | java | pv | zstdmt > output.tmp'.

This is a source of problems, because:

1. a command failing in the script won't fail the script (bash eats errors), so the Python code believes the temporary output was successfully written, commits it, and proceeds: swh/devel/swh-graph#4773
2. most arguments aren't properly escaped
3. a future change to CountPaths will need to build a pipeline dynamically (`zstdcat` in one case, `zstdcat | cat - <(cat other_file.txt | sed)` in another, `zstdcat | cat <(cat other_file.txt | sed) -` in another one), which was be pretty unreadable with string substitutions

vlorentz authored 2 years ago and vlorentz committed 2 years ago

With node filters, some steps will produce empty sets of files,
which makes Luigi crash.

Steps already produce stamp files anyway, so it is good to list
them here anyway.

vlorentz authored 2 years ago and vlorentz committed 2 years ago

A future commit will pass '--object-types *', which really needs to be escaped.

vlorentz authored 2 years ago and vlorentz committed 2 years ago

eg. graph '2022-12-07-history-hosting' is compressed using the '2022-12-07'
dataset export, but applies its own filters on it

vlorentz authored 2 years ago and vlorentz committed 2 years ago

A future commit will make downloads parallel, and sys.exit() plays badly
with multiprocessing.dummy.

It defaults to 90% of the total RAM, which means Java effectively never frees
any memory, and it prevents running more than a couple of tests in parallel.

vlorentz authored 2 years ago and vlorentz committed 2 years ago

It's too easy to accidentally use the wrong version of the graph
when there is a default port

Since 20ea7592, the file doesn't store
the graph anymore (no origin), just a map between person ids and their
name.

GitLab will display the content of the README file when browsing the
repository. But in case the file is a symlink, it will display the
path pointed by the symlink. There is a 6 year old issue about this:
https://gitlab.com/gitlab-org/gitlab/-/issues/15093

We can workaround the issue by having the content at the root of the
repository and a symlink to this file in the `docs/` directory.

Tested in swh/devel/swh-py-template!27

run_script() currently doesn't detect commands crashing, so psql not being
installed made it produce an empty file

It's more correct/resilient, and buffering improves performance.

vlorentz authored 2 years ago and vlorentz committed 2 years ago

And add buffering.

Both changes together reduce the runtime from 133 to ~24 hours.

Related to swh/meta#4960

vlorentz authored 2 years ago and vlorentz committed 2 years ago

Copying it 1k or 10k for each thread is wasteful, even though they are
lightweight copies.

This brings the runtime down from 133 to ~100 hours.

Jérémy Bobbio (Lunar) authored 2 years ago and Jérémy Bobbio (Lunar) committed 2 years ago

Related to swh/meta#4959

Each thread handled one 96th of the node id range. But nodes
are not homogeneously randomized across that range, so some
threads had a lot more work to do than others, causing them
to end weeks after, while most CPU cores idled.

By splitting the range this way, threads should have more
homogeneous workloads.

This is much more performant on the dir layer: it only takes 27 hours on
swh1.enst.fr instead of an ETA of one or two years, most of it spent on
this particular line:

```
LazyLongIterator successorAncestors = graph.successors(successorNodeId);
```

even when replacing all the code that used `successorAncestors`, it was
still the major cause of the huge expected runtime.

Jérémy Bobbio (Lunar) authored 2 years ago and vlorentz committed 2 years ago

protobuf earlier 4.12.* versions crashed when `swhgraph_pb2.py`
was discovered by pytest. This has been fixed in the 4.12.11 release.
See: https://github.com/protocolbuffers/protobuf/issues/10151

Preventing protobuf 4.12.* to be used makes `pip` install grpcio-tools
version 1.49.0 which fails to build on Debian bookworm. In order to
allow more fixed versions of grpcio-tools to be used, we bump the
dependency on protobuf to version 4.12.11 or later.

vlorentz authored 2 years ago and vlorentz committed 2 years ago

to help debugging

This will be used as a metric for 'popularity' of directories,
which will be used to weigh results of PopularContents (which
counts the most popular names used to refer to each content)