Skip to content

POC elastic worker infrastructure

In order to validate the feasibility and the possible caveats to implements an elastic workers infrastructure, we will implement a poc managing the workers for the gitlab repositories

  • First we need to refresh and land the kubernetes branch on the swh-environment to have a working example

  • Refresh the rancher VM on uffizi to test the solution in a pseudo real environment (created from scratch, cf. terraform/staging)

  • Create workers and register them in the rancher cluster

  • POC image building / deployment process (manual push on docker hub)

  • POC worker autoscaling according to message in queues

  • POC worker autoscaling according to available ressources on the cluster

  • cluster / elastic workers monitoring (nb of running workers, statsd, ...)

  • Plug standard log ingestion with elastic workers

  • [1] Draft note on https://hedgedoc.softwareheritage.org/4ZHT03kRT7mHYOEm1MSueQ

Migrated from T3592 (view on Phabricator)

Edited by Antoine R. Dumont