Winery: Implement asynchronous packing
Highlights of this pile of changes, in commit order:
- avoid double-init() in WineryWriter (would create new shards and never use them)
- replace python-sh with direct calls to subprocess
- some hardening of the test fixtures for postgresql and ceph rbd so that they teardown properly in "all" (famous last words) cases
- cleaner separation of parameters in the benchmark code (between ro workers and rw workers)
- implement shard lifecycle with an enum instead of a pair of booleans
- add an asynchronous packing function
- hook the asynchronous packing into benchmarks
A few commits to improve mypy coverage have been sprinkled throughout.
TODO:
-
add a CLI entry point for packing -
future: get rid of all the Winery classes passing around bare kwargs dicts.
Implements #4741 (closed)
Edited by Nicolas Dandrimont
Merge request reports
Activity
Filter activity
assigned to @olasd
Jenkins job DOBJS/gitlab-builds #73 succeeded .
See Console Output and Coverage Report for more details.added 18 commits
- 9d2a89aa...98f99783 - 8 earlier commits
- f08f4a06 - winery: add logging for database admin ops
- e8ae69b1 - winery: import psycopg2.errors when needed
- 9869a715 - winery.sharedbase: improve shard lock handling
- 1391a493 - winery: add logging to shard packing code
- 732f874d - winery: Packer does not need to be a class
- 6f4e262e - winery: use a custom postgresql fixture to tear down multiple databases
- 4447b54b - winery: Add support for packing asynchronously
- f9f922c3 - winery: don't use f-strings in benchmark logging
- 2701f1c6 - winery benchmark: split RO and RW workers in separate classes
- d8adfe66 - winery benchmark: make RW worker more modular
Toggle commit listadded 6 commits
- 723a31bb - winery: Add support for packing asynchronously
- 82aca122 - winery: don't use f-strings in benchmark logging
- 99881b97 - winery benchmark: split RO and RW workers in separate classes
- d8010d14 - winery benchmark: make RW worker more modular
- 7b8b8317 - winery benchmark: split storage arguments, benchmark arguments, and
- d0d3942a - winery benchmark: more granular control over RW workers
Toggle commit listJenkins job DOBJS/gitlab-builds #75 succeeded .
See Console Output and Coverage Report for more details.added 7 commits
- 7fa5b31e - winery: don't use f-strings in benchmark logging
- b967c046 - winery benchmark: split RO and RW workers in separate classes
- 58a4325d - winery benchmark: make RW worker more modular
- c6f84f1f - winery benchmark: split storage arguments, benchmark arguments, and
- e8d1a680 - winery benchmark: more granular control over RW workers
- d408b76d - winery: replace shard lifecycle booleans with an enum
- 23254215 - winery.sharedbase: Improve type annotations
Toggle commit listJenkins job DOBJS/gitlab-builds #76 succeeded .
See Console Output and Coverage Report for more details.- Resolved by Nicolas Dandrimont
- Resolved by Jérémy Bobbio (Lunar)
- Resolved by Nicolas Dandrimont
121 121 self.stats = Stats(kwargs.get("output_dir")) 122 122 self.args = kwargs 123 123 self.shard = shard 124 self.init() 125 126 def init(self): 127 124 self.rw = RWShard(self.shard, **self.args) 128 125 self.ro = ROShard(self.shard, **self.args) 129 126 130 def uninit(self): 131 del self.ro 132 self.rw.uninit() 133 - Comment on lines -130 to -133
This might have landed in the wrong commit (1391a493) given the message, but not a real issue.
- Resolved by Nicolas Dandrimont
145 145 count += 1 146 logger.info("Worker(rw, %s): packing %s objects", os.getpid(), count) 147 packer = self.winery.packers[0] 148 packer.join() 149 assert packer.exitcode == 0 146 self.finalize(count) 150 147 elapsed = time.time() - start 151 148 logger.info("Worker(rw, %s): finished (%.2fs)", os.getpid(), elapsed) 152 149 153 150 return "rw" 154 151 152 def keep_going(self): 153 return len(self.winery.packers) == 0 154 155 def finalize(self, count): 156 logger.info("Worker(rw, %s): packing %s objects", os.getpid(), count) - Resolved by Nicolas Dandrimont
- Resolved by Nicolas Dandrimont
- Resolved by Nicolas Dandrimont
- Resolved by Nicolas Dandrimont