-
v5.1.04102a5ea · ·
v5.1.0 No changes to the Python code in this release, only Rust and Javz. Additions (Rust): * stdlib: add fs_ls_tree, implementing a recursive ls of a FS tree * Switch NODE_PROPERTIES to Rust implementation * Switch EXTRACT_PERSONS to Rust implementation * Add 'compare-graphs' tool Soundness fixes (Rust): * Check CSV inputs are not a single non-header line, when expecting a header * Switch default port of the Rust gRPC server to 50091 Compilation fixes (Rust): * Lower MSRV to 1.79 Documentation: * rust doc: Move Crash Course to its own page + add Tutorial * Update gRPC documentation to run the Rust implementation * Fix documentation of DirEntry::permission * Fix docstring to mention inputs are CSV * Add diagnostic hint for when properties are not loaded Internal changes: * stdlib: port find_latest_snp() to typed labeled successor iterator * Remove unused Java utils and SWH-specific Java compression code * Make DefaultUnderlyingGraph and SwhLabeling newtypes * Fix warnings * Use ar_row's own decimal -> timestamp decoding * pytest.ini: Ignore Rust's target/
-
v5.0.073120521 · ·
v5.0.0 No changes to the Python code in this release, only Rust. This is the first release of swh-graph on crates.io. Breaking changes (Rust): * Rename "labelled" -> "labeled", for consistency with webgraph * Rename SWHType to NodeType * Moved find_root_dir to the "stdlib" * Make labeled_{predecessor,successors} return typed labels * Rename Rust all "::new_with" constructors to just "::with" * Rust: implement FromStr (rather than TryInto<&str>) for NodeType Additions (Rust): * Added a "stdlib" of algorithms (find_latest_snp, path resolution functions, generic node visit operator) * GraphBuilder: Add support for Visit (and Branch) labels * Add support for multi-arcs in VecGraph and GraphBuilder * Add `.flatten_labels()` on labeled arc iterator * Rewrite EDGE_LABELS compression step in Rust * Subgraph: add builder based on NodeConstraint * Add tool to dump edges from a compressed graph Improvements (Rust): * Switch to released versions of webgraph and ar_row * Various performance improvements in properties compression * GraphBuilder: add BuiltGraph type alias for done() return type * Rewrite CountPaths to produce sharded Parquet files directly * Small improvement of the rust executable dir handling in check_config() Soundness fixes (Rust): * Fixed all bugs NODE_PROPERTIES compression (or at least, have a strict subset of the Java implementation's bugs). In particular: * properties: Update arrays atomically and remove LongArrayBitVector * Remove redundant 'datasets' path component * Rust: use create_new() to create on-disk maps * Subgraph: fix has_arc(), which was transposing the passed arc Crash/compilation fixes (Rust): * Fix detection of missing node2type.bin/content.is_skipped.bits file * Fix loading node2type.bin larger than 2^31 * 8 bytes * Fix debian requirements * blobs_dataset: Make Datafusion write to a single file * root_directory.rs: do not require (unneeded) SwhBackwardGraph trait Documentation: * Subgraph: document that num_nodes/arcs() return non-filtered values * Add a 'minimal build for tests' section in rust/README.md * review *.rs file headers: add missing Copyright decl -
v4.0.05c51b6a5 · ·
v4.0.0 Breaking: * Switch default gRPC server from Java to Rust * Remove sorted list of nodes from graph output Bug fixes: * Fix pyo3 extension build on recent Maturin versions * compute-directory-frontier: Use correct timestamp to decide if node is frontier Ergonomics: * Add CLI to regenerate graph files for the current version * Add CLI to download the graph (and decompress .zst files) Performance optimizations: * Start BFS from (sorted) origins instead of random nodes * http_rpc_server.VisitEdgesView: Do not fetch edge labels * provenance: Switch from glibc malloc to jemalloc * contents-in-directories: Do not traverse nodes not reachable from a frontier directory * Make dependency on swh-storage optional Rust rewrite: * Look for rust swh-graph-grpc-serve execuctable in user's PATH and 'rust_executable_dir' config * EXTRACT_NODES, MAPS, COMPOSE_ORDERS, TRANSPOSE: Switch to Rust implementation * Rewrite ListOriginContributors in Rust * Rewrite ListFilesByName in Rust * Rewrite MPHTranslate in Rust * Add support for computing node ids from SWHIDs * Update list of temporary files to clean after compression is done * java: Add support for reading is_skipped.bits and node2type.bin (produced by the Rust compression pipeline) Other improvements: * CountPaths: Add support for 2024-05-16 graph * provenance: Add support for '--node-filter all' * Document how to get the URL of an origin node * Document dependency on protoc * grpc-server: Add --masked-nodes option * pytest_plugin: Move server config to its own fixture * model: adapt to the renaming of model.TargetType to model.SnapshotTargetType
-
v3.4.08791e8e2 · ·
v3.4.0 - Add MPHF to the Python extension - provenance: - Add support for Hive partitioning on sha1_git column - Remove remaining references to topological_order_dir - Replace .csv.zst output with .parquet - Compress paths with zstd - Order rows in final files by the column they will be queried on - Move find_frontiers_from_root_directory to frontier-directories-in-revisions - Make dependency on 'arrow' optional, even when 'dataset-writer' is used - docs: Fix reference to SWHIDs - webgraph: Set RUST_MIN_STACK to avoid stack overflows - naive client: add max_matching_nodes for neighbors method - Add timestamp and is_full_visit bit to ori->snp edges' label - Update webgraph - Fix task dependencies - Switch LLP compression step to use the Rust implementation - Fix sort_batch_size and input_batch_size values being swapped in 'permute' and 'transpose' commands - transform: Log computed configuration - Rewrite PopularContentPaths in Rust - Misc. fixes and code improvements
-
v3.3.15867279b · ·
v3.3.1 * permute-and-symmetrize: Sort arc lists in parallel * rust and java: Enforce max_edges *before* traversing edges
-
v3.3.075acaa6b · ·
v3.3.0 * Rewrite most dataset generation scripts from Java to Rust * Make most dataset generation scripts produce sharded files instead of a single .csv.zst that cannot be processed in parallel * Improve ergonomics of Rust library * Switch some early compression steps to Rust (BV, BFS, {PERMUTE,TRANSPOSE,SIMPLIFY}_BFS) * Replace Athena with datafusion * Finish Rust rewrite of the gRPC server (but Java remains the default) -
-
v3.1.0159f5343 · ·
v3.1.0 Misc: * Prevent timestamps in node properties from being shifted according to the timezone WriteNodeProperties is being run in. * Add scripts to generate an index for swh-provenance HTTP API: * Raise an error when RemoteGraphClient URL is wrong * Fix DeprecationWarnings in http_client * http_rpc_server: Remove duplicate request * Properly handle empty results in HTTP client CLI: * compress: Hide traceback on subprocess exception * Print the end of the log Rust: * java utils: add Mph2Cmph to convert from java to rust version of MPH files * initial skeleton to ship a rust crate * Implement the first Rust data structures * codespell: move excluded word list from toml to precommit conf * Add example BFS implementation * added Node2Type, its tests, and an bin to convert the .node2swhid.bin file to the new .node2type.bin Luigi: * Avoid silencing exceptions thrown by worker threads * Document how to run Luigi tasks * luigi: Add missing output files to ExtractNodes. * luigi, docs: MAPS step does not depend on LLP * luigi: Increase MPH_LABELS memory allowance * Tune MPH maximum memory to avoid OOMs * Honor JAVA_HOME when locating the default Java binary PopularContentPaths: * Avoid concurrent updates to ProgressLogger * Fix missing 'sha1' column for contents with no path * Skip loading the forward graph * Allow zstdcat more memory * Print corrupt records Internal: * Prevent flake8 from finding issues in build/ directory * Migrate to copier-based swh-py-template * Move the jar under swh/graph so we can get rid of deprecated "date_files" in setup * Fix documentation build
-
-
-
-
v3.0.020a76683 · ·
v3.0.0 Breaking changes: * Use CRLF in output CSV instead of LF * FindEarliestRevision: switch from TSV to CSV and rename columns * origin_contributors: Add final task checking integrity of the dataset * origin_contributors: Change table format/layout to be more compact, improve performance, add contribution years Minor changes: * Match swh-dataset's rename of 'object_type' to 'object_types' * luigi: Make grpc API globally configurable, and remove the default value * Bump requirements on protobuf New derived datasets: * Import the "blobs datasets" (license, citation) generation script as a luigi workflow * Add scripts to find the most popular name(s)/path of content nodes * Add script to count the total number of paths to any node * Add ListEarliestRevisions, which computes the earliest revision of all dir/cnt objects at once New features: * Export naive_graph_client and remote_graph_client fixtures in pytest plugin * Add INITIAL_ORIGIN and FORKED_ORIGIN to example dataset * Make example dataset available for other modules * Add export_{started,ended}_at to Stats response * Make Luigi tasks declare their RAM usage (and auto-tune when possible) * Add support for compressing the graph with only some node types * Add step stamp to each step's list of output files * Add support for making the graph dataset name differ from the export name * webgraph.py: Display log path in error messages * Add a script to count paths leading to each node * TopoSort: Default to DFS instead of BFS * TopoSort: Add support for running forward * luigi: Add an option to define the maximum RAM used by graph compression Documentation: * Move the doc for the example dataset to its own page * Include the representation of the example dataset in the documentation * Add some more style to the example dataset graph * Remove the figure from the example dataset documentation * docs/compression: Fix inaccuracies in the dependency graph * DownloadGraphFromS3: Fix incorrect docstring Bug fixes: * getMessage: Fix crash on origins with no URL property * luigi/misc_datasets: Fix _clean_s3_directory() when directory is empty * Add a flyweight copy() to SwhGraphProperties to make it threadsafe * FindEarliestRevision: Fix crash on revisions with no committer timestamp * compressed_graph: Fix data race to .obl files in Transpose command * Check in constructor instead of size64() * NodeIdMap: Fix incorrect implementation of size64() * TopoSort: Fix discard of the last node while looking for leaves Performance improvements: * FindEarliestRevision: Run traversals in parallel * FindEarliestRevision, TopoSort: Use Apache Commons CSV * luigi/compressed_graph: Tune -Xmx per task * TopoSort: Various optimizations Misc: * assembly: Remove some transitive dependencies from the final uber jar * luigi: Rewrite compression pipeline as small Luigi tasks -
-
-
-
-