Skip to content
Snippets Groups Projects

Don't materialize vec of values while building MPHs

Merged vlorentz requested to merge lazy-par-mph into master

Instead, pthash will iterate on the values (while we read them from disk) and only materialize a vec of hashes, which is much smaller than a vec of values.

When building the SWHIDs MPH, this reduces memory usage from 3.6 to 0.6TB.

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
Please register or sign in to reply
Loading