Winery settings refactorings

The final goal of this series of refactorings is to make the shards pool configurable (between a RBD-based pool and a file-based pool).

As all the winery components had been written to take a bunch of indiscriminate kwargs and picking through them, adding more settings would be a pain, so I went ahead and moved settings to a clearer (I hope) hierarchy, with the aim of reducing the number of kwargs passed around.

Eventually this should allow performing the "file-based pool" refactoring.

This only makes sense in a commit-by-commit review, I think...

Jenkins job DOBJS/gitlab-builds #325 succeeded in 3 min 43 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

added 3 commits

• c1234013 - winery: move RBD settings to the new settings framework
• 72f9df6a - winery: make RBD shard manager a function rather than a method of the pool
• b04bddf6 - winery: Make the directory-based shards pool usable

Compare with previous version

Jenkins job DOBJS/gitlab-builds #326 succeeded in 4 min 10 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

b04bddf6 - winery: Make the directory-based shards pool usable is a bit of a mess, but the mocking of the ceph pool in the tests was really tangled with the rest of the code, so I didn't manage to unpick the "moving of the code" and the "making it available through configuration", unfortunately.

added 4 commits

• ff145b87 - winery: close read-only (and read-write) shards on shutdown
• 0222d0eb - winery: On shutdown, ensure that packer processes are waited for
• b44b9841 - winery: in storage fixture, use rbd_pool_name
• 0a7e197f - winery: run deletion tests on both pool backends

Compare with previous version

Jenkins job DOBJS/gitlab-builds #327 succeeded in 2 min 44 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

added 1 commit

• bef4d9a6 - winery: bring documentation up to date with respect to the reality of the code

Compare with previous version

Jenkins job DOBJS/gitlab-builds #328 succeeded in 4 min 17 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

I think I would prefer these types to be explicitly names XXXSettings (here PackerSettings) so it's clear what it is when manipulated outside of this settings module.

So, that's what I did first.

But then I ended up having to import 25 names from the settings module (especially in objstorage), so I moved to from . import settings, which made everything look like settings.FooSettings, which annoyed me. from .settings import * made pylance highlight everything as "maybe this is an unbound variable" which annoyed me too.

I'm happy to defer to your opinion either way, I agree these names aren't very satisfactory.

Reading some of the following commits in this MR, I wonder if I would not go to an object-based api for settings rather than a dict-based one (aka use attr instead of TypedDict I suppose)...

It could make things a bit more elegant as well (get rid of the xxx_settings_with_defaults which I find not so pretty).

That, too, was my first instinct. I was concerned that it would make typing annoying for the polyglot shards pool RBDShardsPool/DirectoryShardsPool, so I went with the lighterweight TypedDict instead.

I yearn for a typed config parser with a sensible interface, but...

"list of shard" or "list of shards"?

"or a RBD image" missing "or a regular file" as well?

changed this line in version 6 of the diff

"swh-perfect hash" -> "swh-perfecthash" could come with this commit, I supposse

changed this line in version 6 of the diff

approved this merge request

Thanks a lot for this

added 11 commits

• a8b2e459 - 1 earlier commit
• c156e5b5 - winery: make throttler optional
• 914a2259 - winery: move RBD settings to the new settings framework
• ec5f9e05 - winery: make RBD shard manager a function rather than a method of the pool
• f7927dae - winery: Make the directory-based shards pool usable
• 1eafb823 - winery: close read-only (and read-write) shards on shutdown
• 6337570a - winery: On shutdown, ensure that packer processes are waited for
• 1f90aae1 - winery: in storage fixture, use rbd_pool_name
• 2a42ea79 - winery: run deletion tests on both pool backends
• a714ab06 - winery settings: remove legacy kwargs
• 77f139bd - winery: bring documentation up to date with respect to the reality of the code

Compare with previous version

Jenkins job DOBJS/gitlab-builds #329 succeeded in 4 min 28 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

added 2 commits

• 67a32cc2 - winery: bring documentation up to date with respect to the reality of the code
• 198abe69 - winery: improve typing and documentation of SharedBase

Compare with previous version

Jenkins job DOBJS/gitlab-builds #330 succeeded in 3 min 25 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

merged manually