POC elastic worker infrastructure
In order to validate the feasibility and the possible caveats to implements an elastic workers infrastructure, we will implement a poc managing the workers for the gitlab repositories
-
First we need to refresh and land the kubernetes branch on the swh-environment to have a working example -
Refresh the rancher VM on uffizi to test the solution in a pseudo real environment (created from scratch, cf. terraform/staging) -
Create workers and register them in the rancher cluster -
POC image building / deployment process (manual push on docker hub) -
POC worker autoscaling according to message in queues -
POC worker autoscaling according to available ressources on the cluster -
cluster / elastic workers monitoring (nb of running workers, statsd, ...) -
Plug standard log ingestion with elastic workers -
[1] Draft note on https://hedgedoc.softwareheritage.org/4ZHT03kRT7mHYOEm1MSueQ
Migrated from T3592 (view on Phabricator)