-
-
-
-
v3.0.020a76683 · ·
v3.0.0 Breaking changes: * Use CRLF in output CSV instead of LF * FindEarliestRevision: switch from TSV to CSV and rename columns * origin_contributors: Add final task checking integrity of the dataset * origin_contributors: Change table format/layout to be more compact, improve performance, add contribution years Minor changes: * Match swh-dataset's rename of 'object_type' to 'object_types' * luigi: Make grpc API globally configurable, and remove the default value * Bump requirements on protobuf New derived datasets: * Import the "blobs datasets" (license, citation) generation script as a luigi workflow * Add scripts to find the most popular name(s)/path of content nodes * Add script to count the total number of paths to any node * Add ListEarliestRevisions, which computes the earliest revision of all dir/cnt objects at once New features: * Export naive_graph_client and remote_graph_client fixtures in pytest plugin * Add INITIAL_ORIGIN and FORKED_ORIGIN to example dataset * Make example dataset available for other modules * Add export_{started,ended}_at to Stats response * Make Luigi tasks declare their RAM usage (and auto-tune when possible) * Add support for compressing the graph with only some node types * Add step stamp to each step's list of output files * Add support for making the graph dataset name differ from the export name * webgraph.py: Display log path in error messages * Add a script to count paths leading to each node * TopoSort: Default to DFS instead of BFS * TopoSort: Add support for running forward * luigi: Add an option to define the maximum RAM used by graph compression Documentation: * Move the doc for the example dataset to its own page * Include the representation of the example dataset in the documentation * Add some more style to the example dataset graph * Remove the figure from the example dataset documentation * docs/compression: Fix inaccuracies in the dependency graph * DownloadGraphFromS3: Fix incorrect docstring Bug fixes: * getMessage: Fix crash on origins with no URL property * luigi/misc_datasets: Fix _clean_s3_directory() when directory is empty * Add a flyweight copy() to SwhGraphProperties to make it threadsafe * FindEarliestRevision: Fix crash on revisions with no committer timestamp * compressed_graph: Fix data race to .obl files in Transpose command * Check in constructor instead of size64() * NodeIdMap: Fix incorrect implementation of size64() * TopoSort: Fix discard of the last node while looking for leaves Performance improvements: * FindEarliestRevision: Run traversals in parallel * FindEarliestRevision, TopoSort: Use Apache Commons CSV * luigi/compressed_graph: Tune -Xmx per task * TopoSort: Various optimizations Misc: * assembly: Remove some transitive dependencies from the final uber jar * luigi: Rewrite compression pipeline as small Luigi tasks -
-
-
-
-
-
-
v2.3.094b1d2c1 · ·
v2.3.0 * Add tools/swh-graph-lookup/swh-graph-lookup.py * Add Luigi workflow to generate the compressed graph * Add scripts to generate a topological order and list origin contributors * Fix minor crashes
-
v2.2.0edcca81a · ·
v2.2.0 * FindEarliestRevision: Add earliest_ts and rev_occurrences columns * pre-commit, tox: Bump pre-commit, codespell, black and flake8 * docs/grpc-api.rst: Update to match to current code * docs/grpc-api.rst: Add Python examples
-
-
-
v2.1.248e9fbbb · ·
v2.1.2 * Apply 'max_matching_nodes' restriction after 'return_types' filter * http_client: Add max_matching_nodes parameter to visit_nodes()