This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

vlorentz authored 2 years ago and vlorentz committed 2 years ago

To be consistent with other CSV files in the license dataset.

Because the main table already contains (origin_id, person_id) pairs,
it is redundant to repeat every pair in this file.

They tend to be forgotten; this will make sure that running 'tox' will
run Java tests as well as Python tests.

It will also make it easier to run Java tests on the CI

two paths + reimport tasks at the package level so they are automatically
picked up by luigi without passing all module names on the CLI

whitelist_externals is deprecated since v3.18, and removed since v4.0

RPCServerApp inherits from aiohttp.web.Application, and GraphServerApp
does not use any of the features added by RPCServerApp

It's slightly more accurate

Origin pseudo-SWHIDs are kind of useless on their own, as there is no public API
to resolve them yet; so we might as well produce integers instead.

A future commit will add a mapping table to URLs, which are actually
useful.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496

It is easier to keep track of it in Grafana.

RunAll is not practical, because we need to mount the graph on a tmpfs
at some point. Let's keep that step manual for now, which means
RunExportCompressUpload and DeanonymizeOriginContributors will be
called separately.

I cannot find how to make WebGraph log to stderr instead of stdout,
so it seems to be the only way.

It ensures luigi will be available after having installed swh
development environment.

It can be cumbersome to set paths for all (recursives) dependencies of
the task we want to run; this CLI endpoint takes care of most of them.

It is going to get large, with the future addition of tasks to generate
the license dataset and the citation dataset.

This triggers a bug in ListOriginContributors, causing it to include
"null" as a contributor.
A future commit will fix this.

This Java script (and related Luigi tasks) traverse the graph in
topological order, building up the set of all contributors to a
node and its ancestors, then dump the value of this set for every
origin node they encounter.