Draft: Use Nickel for loader configuration
While I worked on a couple other configurations with @yannham (see !47 (closed)), this is a first, completely working, completely tested, somewhat proper, complete use of Nickel in our project. Specifically, this replaces the loader config template with a Nickel configuration. This allows us, for example, to
- replace the
envsubst
-based templating mechanism, - simplify a bit the loader start script,
- have runtime type checking (is that the correct word?), as exemplified by the Redis and objstorage port parameters (try setting them to something invalid in the
config.template
-derived config file, orSWH_LOADER_OBJSTORAGE
to something other thannoop
orremote
).
The idea is as follows: the Nickel configuration consists of two files,
-
loader.ncl
that essentially contains a nickelized copy of the loader config template file with typed parameters and default values, plus a custom contract that allows us to check whether a string is a valid literal for a port number, plus some small boilerplate, -
config.ncl
which is the kind of user-facing part of the configuration and contains all parameters that we actually might want to change. In this case, these are hostnames, ports, and the switch determining whether the object storage is a no-op storage or a remote storage.
This configuration is then evaluated with necessary values passed in mostly via mostly standard environment variables (thus maintaining the environment variable / env file "message passing" system), and written out as a YAML config file in the loader start script.
Note that there's still many things that could be improved:
the handling of the objstorage no-op / remote switch is inconsistent (we should pass it the same way as the other environment variables),- types can be improved
(for example, many config parameters are actually positive naturally numbers, for which a type exists in Nickel (,std.number.PosNat
) - ...
but this should give a very first idea of how Nickel could work in our project.
We could do many more things, if we wanted to:
- replace the other configuration files with Nickel configurations,
- replace the
config.template
config file with a Nickel configuration,- if we did this, we could then add a simple switch that allows us to create a fully local configuration and a Slurm configuration without code duplication,
- create the Bash scripts with Nickel (and thus have configuration items that are not used in the service template files, but in the start / stop / wait / etc. scripts also controlled by Nickel with all its advantages),
- ... the possibilities are endless
😁
To test this, all you have to do is get the Nickel binary here (yes, it's not so lightweight), and set the NICKEL_BINARY
env var in your config.template
-derived file to the path of the binary. Then everything should work out of the box, both locally and using Slurm.
Finally, this is in draft mode, just to prevent it from being merged from now: this MR is just a demo and should ideally serve as the basis for a discussion on whether we should pursue the use of Nickel in this project or not.