Skip to content

winery: big ol'pile o'changes

Nicolas Dandrimont requested to merge olasd/swh-objstorage:mr/winery-benchmark-changes into master

This is a bunch of small and big changes to winery that were introduced for the benchmarking to actually run to some form of completion. Sorry for the haphazard pile of changes; I suggest a commit-per-commit review.

I've tried to separate the actual logic changes from the benchmarking related changes, with mixed results.

Highlights of this changeset for the actual winery changes:

  • migration to psycopg3 (to use the binary protocol, which seems to double the postgresql write performance)
  • migration to COPY for shard packing (reduces memory usage of the packing process by 99.9%: only one object is held in memory, instead of the whole shard)
  • disabling of autovacuum for shard tables (as they're short-lived)
  • introduction of automagic postgresql connection pooling

Highlights of the benchmark changes:

  • less expensive object fetching (use a randomly sorted tablesample instead of randomly sorting all of the signature2shard table (!) on each iteration)
  • add a stats worker (even though I believe this was the eventual downfall of the benchmark run)

Merge request reports