Skip to content
Snippets Groups Projects

Winery: Implement asynchronous packing

Merged Nicolas Dandrimont requested to merge olasd/swh-objstorage:mr/async-packing into master

Highlights of this pile of changes, in commit order:

  • avoid double-init() in WineryWriter (would create new shards and never use them)
  • replace python-sh with direct calls to subprocess
  • some hardening of the test fixtures for postgresql and ceph rbd so that they teardown properly in "all" (famous last words) cases
  • cleaner separation of parameters in the benchmark code (between ro workers and rw workers)
  • implement shard lifecycle with an enum instead of a pair of booleans
  • add an asynchronous packing function
  • hook the asynchronous packing into benchmarks

A few commits to improve mypy coverage have been sprinkled throughout.

TODO:

  • add a CLI entry point for packing
  • future: get rid of all the Winery classes passing around bare kwargs dicts.

Implements #4741 (closed)

Edited by Nicolas Dandrimont

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Jérémy Bobbio (Lunar)
  • Jérémy Bobbio (Lunar)
  • Jérémy Bobbio (Lunar)
  • 121 121 self.stats = Stats(kwargs.get("output_dir"))
    122 122 self.args = kwargs
    123 123 self.shard = shard
    124 self.init()
    125
    126 def init(self):
    127 124 self.rw = RWShard(self.shard, **self.args)
    128 125 self.ro = ROShard(self.shard, **self.args)
    129 126
    130 def uninit(self):
    131 del self.ro
    132 self.rw.uninit()
    133
  • Jérémy Bobbio (Lunar)
  • Jérémy Bobbio (Lunar)
    Jérémy Bobbio (Lunar) @lunar started a thread on an outdated change in commit 58a4325d
  • 145 145 count += 1
    146 logger.info("Worker(rw, %s): packing %s objects", os.getpid(), count)
    147 packer = self.winery.packers[0]
    148 packer.join()
    149 assert packer.exitcode == 0
    146 self.finalize(count)
    150 147 elapsed = time.time() - start
    151 148 logger.info("Worker(rw, %s): finished (%.2fs)", os.getpid(), elapsed)
    152 149
    153 150 return "rw"
    154 151
    152 def keep_going(self):
    153 return len(self.winery.packers) == 0
    154
    155 def finalize(self, count):
    156 logger.info("Worker(rw, %s): packing %s objects", os.getpid(), count)
  • Jérémy Bobbio (Lunar)
  • Jérémy Bobbio (Lunar)
  • Jérémy Bobbio (Lunar)
  • Jérémy Bobbio (Lunar)
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Please register or sign in to reply
    Loading