upgrade-swh-service: Make reference names unique

As reference namespace is global, we need to make an extra effort to name reference uniquely.

Only top level commit is worth reviewing as other commits are opened as distinct MRs (!341 (merged), !342 (merged))

swh/infra/sysadm-environment#4794 (closed)

docs/sysadm/deployment/upgrade-swh-service.rst

-.. _upgrade-swh-service:
+.. _deployment-upgrade-swh-service:
 
 Upgrade swh service
 ===================
@@ -27,7 +27,7 @@ Then follows the actual :ref:`deployment with debian packaging
 <deployment-with-debian-packaging>`. It concludes with the :ref:`deployment with
 kubernetes<deployment-with-kubernetes>` chapter.
 
-.. _distinct-services:
+.. _deployment-upgrade-swh-service-distinct-services:
 
 Distinct Services
 -----------------
@@ -38,7 +38,7 @@ Distinct Services
 - rpc services (scheduler, objstorage, storage, web, ...)
 - journal client services (search, scheduler, indexer)
 
-.. _code-and-publish-a-release:
+.. _deployment-upgrade-swh-service-code-and-publish-a-release:
 
 
 Code and publish a release
@@ -50,7 +50,7 @@ Code an evolution or a bugfix in the impacted git repository (usually the master
 branch). Open a diff for review. Land it when accepted. And then release it following
 the :ref:`tag and push <tag-and-push>` part.
 
-.. _tag-and-push:
+.. _deployment-upgrade-swh-service-tag-and-push:
 
 Tag and push
 ~~~~~~~~~~~~
@@ -63,7 +63,7 @@ When ready, `git tag` and `git push` the new tag of the module. Then let jenkins
    $ git tag -a vA.B.C  # (optionally) `git tag -a -s` to sign the tag too
    $ git push origin --follow-tags
 
-.. _publish-artifacts:
+.. _deployment-upgrade-swh-service-publish-artifacts:
 
 Publish artifacts
 ~~~~~~~~~~~~~~~~~
@@ -73,16 +73,18 @@ the tag just pushed). It then builds the debian package and pushes it to our :re
 debian repositories <howto-debian-packaging>`.
 
 
-.. _troubleshoot:
+.. _deployment-upgrade-swh-service-troubleshoot:
 
 Troubleshoot
 ~~~~~~~~~~~~
 
 If jenkins fails for some reason, fix the module be it :ref:`python code
-<code-and-publish-a-release>` or the :ref:`debian packaging <troubleshoot-debian-package>`.
+<deployment-upgrade-swh-service-code-and-publish-a-release>` or the
+:ref:`debian packaging
+<deployment-upgrade-swh-service-troubleshoot-debian-package>`.
 
 
-.. _deployment-with-debian-packaging:
+.. _deployment-upgrade-swh-service-with-debian-packaging:
 
 
 Deployment with debian packaging
@@ -90,7 +92,7 @@ Deployment with debian packaging
 
 This mostly involves deploying new version of debian packages to static nodes.
 
-.. _upgrade-services:
+.. _deployment-upgrade-swh-service-upgrade-services:
 
 Upgrade services
 ~~~~~~~~~~~~~~~~
@@ -122,7 +124,7 @@ nodes through the following group names:
 
 See :ref:`deploy-new-lister` for a practical example.
 
-.. _troubleshoot-debian-package:
+.. _deployment-upgrade-swh-service-troubleshoot-debian-package:
 
 Debian package troubleshoot
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -144,7 +146,7 @@ Lather, rinse, repeat until it's all green!
 Deploy
 ------
 
-.. _nominal-case:
+.. _deployment-upgrade-swh-service-nominal-case:
 
 Nominal case
 ~~~~~~~~~~~~
@@ -171,7 +173,7 @@ command, something like:
 [3] pergamon is already *clush* configured to allow multiple ssh connections in parallel
 on our managed infrastructure nodes.
 
-.. _configuration-change-required:
+.. _deployment-upgrade-swh-service-configuration-change-required:
 
 Configuration change required
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -187,7 +189,7 @@ Or force a puppet run:
 
 Note: *-t* is not optional
 
-.. _long-standing-upgrade:
+.. _deployment-upgrade-swh-service-long-standing-upgrade:
 
 Long-standing upgrade
 ~~~~~~~~~~~~~~~~~~~~~
@@ -218,7 +220,7 @@ Then:
    $ sudo clush -b -w @swh-workers 'systemctl start cron.service; puppet agent --enable'
 
 
-.. _deployment-with-kubernetes:
+.. _deployment-upgrade-swh-service-with-kubernetes:
 
 Deployment with Kubernetes
 --------------------------
@@ -227,11 +229,24 @@ This new deployment involves docker images which are exposing script/services wh
 running in a virtual python frozen environment. Those versioned images are then
 referenced in a specific helm chart which is deployed in a kubernetes rancher cluster.
 
-Those docker images are built out of a declared Dockerfile in in the `swh-apps`_
+That cluster runs on machines nodes (with :ref:`specific labels
+<deployment-upgrade-swh-service-labels-on-nodes>`) onto which are scheduled
+pods with containers inside. Those containers are the ones spawning the docker
+image as applications.
+
+Those docker images are built out of a declared Dockerfile in the `swh-apps`_
 repository.
 
-Add a new app
-~~~~~~~~~~~~~
+You can either:
+
+- :ref:`Add a new application<deployment-upgrade-swh-service-add-new-swh-application>`
+- :ref:`Update an application<deployment-upgrade-swh-service-update-swh-application>`
+- :ref:`Release a new version of an application<deployment-upgrade-swh-service-build-and-publish-docker-image-app>`
+
+.. _deployment-upgrade-swh-service-add-new-swh-application:
+
+Add new swh application
+~~~~~~~~~~~~~~~~~~~~~~~
 
 From the repository `swh-apps`_, create a new Dockerfile.
 
@@ -242,20 +257,66 @@ applications can serve as template:
 - rpc service: use `graphql <https://gitlab.softwareheritage.org/swh/infra/swh-apps/-/blob/master/apps/swh-graphql/>`_
 - journal client: use `storage replayer <https://gitlab.softwareheritage.org/swh/infra/swh-apps/-/blob/master/apps/swh-storage-replayer>`_
 
-.. _update-app-frozen-requirements:
+It's time to build and publish a docker image. It's a multiple steps process that can be
+executed locally starting with the :ref:`frozen set of dependencies requirements to
+generate <update-app-frozen-requirements>`.
+
+.. _deployment-upgrade-swh-service-update-swh-application:
+
+Update swh application
+~~~~~~~~~~~~~~~~~~~~~~
+
+If you need to update the swh application, edit its ``swh-apps/apps/$app/Dockerfile`` or
+``swh-apps/apps/$app/entrypoint.sh`` to adapt according to change.
+
+Note:
+If a new requirement is necessary, update the ``swh-apps/apps/$app/requirements.txt``
+(source of the generated ``requirements-frozen.txt``). Note that those should be kept to
+a minimal and it may be that such change should happen upstream in the swh modules
+instead.
+
+Once your update is done, commit and push the change, then :ref:`build and publish the
+new docker image <deployment-upgrade-swh-service-build-and-publish-docker-image-app>`.
+
+.. _deployment-upgrade-swh-service-build-and-publish-docker-image-app:
+
+Build and publish docker image (recommended)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Use the `dedicated jenkins job
+<https://jenkins.softwareheritage.org/job/swh-apps/job/build-docker-image/build?delay=0sec>`_
+to update the app's frozen requirements, build the docker image with that set and
+publish that image to the swh gitlab registry.
+
+Once the application image is published in the registry, you need to
+:ref:`update the impacted chart
+<deployment-upgrade-swh-service-update-impacted-chart>`.
+
+.. _deployment-upgrade-swh-service-update-impacted-chart:
+
+Update impacted chart
+~~~~~~~~~~~~~~~~~~~~~
+
+In the `swh-chart`_ repository, update the `values file
+<https://gitlab.softwareheritage.org/swh/infra/ci-cd/swh-charts/-/blob/production/values-swh-application-versions.yaml>`_
+with the corresponding new changed version.
+
+Check that the nodes are properly labelled to receive the application. Then :ref:`ArgoCD
+<argocd-config>` will be in charge of deploying the changes in a rolling upgrade
+fashion.
+
+.. _deployment-upgrade-swh-service-update-app-frozen-requirements:
 
 Update app's frozen requirements
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Once the application is registered. We need to build the frozen environment:
-
-We'll first need a "build-deps" container with some dependencies set (due to some
+We'll first need a "app-manager" container with some dependencies set (due to some
 limitations in our stack):
 
 .. code::
 
    $ cd swh-apps/scripts
-   $ docker build -t build-deps .
+   $ docker build -t app-manager .
 
 Out of this container, we are able to generate the frozen requirements for the
 $APP_NAME (e.g. *loader_{git, svn, cvs, ...}*, *lister*, *indexer* ...):
@@ -263,18 +324,19 @@ $APP_NAME (e.g. *loader_{git, svn, cvs, ...}*, *lister*, *indexer* ...):
 .. code::
 
    $ cd swh-apps
-   $ docker run --rm -v $PWD:/src build-deps generate-frozen-requirements $APP_NAME
+   $ docker run --rm -v $PWD:/src app-manager generate-frozen-requirements $APP_NAME
 
 You have built your frozen requirements that can be committed. Next, we will
-:ref:`generate the image updated with that frozen environment <generate-image>`.
+:ref:`generate the image updated with that frozen environment
+<deployment-upgrade-swh-service-generate-image>`.
 
-.. _generate-image:
+.. _deployment-upgrade-swh-service-generate-image:
 
 Generate image
 ~~~~~~~~~~~~~~
 
 Build the docker image with the frozen environment and then :ref:`publish it
-<publish-image>`:
+<deployment-upgrade-swh-service-publish-image>`:
 
 .. code::
 
@@ -298,7 +360,7 @@ You must have a gitlab account and generate a personal access token with at leas
 `write` access to the `gitlab registry
 <https://gitlab.softwareheritage.org/swh/infra/swh-apps/container_registry/>`_.
 
-.. _publish-image:
+.. _deployment-upgrade-swh-service-publish-image:
 
 Publish image
 ~~~~~~~~~~~~~
@@ -313,29 +375,50 @@ then push the image:
    $ docker push $FULL_IMAGE
    $ docker push $FULL_IMAGE_LATEST
 
-Do not forget to :ref:`commit the changes and tag <commit-changes-and-tag>`.
+Do not forget to :ref:`commit the changes and tag
+<deployment-upgrade-swh-service-commit-changes-and-tag>`.
 
-Finally, let's :ref:`update the impacted chart <update-impacted-chart>` with the new
-docker image version.
+Finally, let's :ref:`update the impacted chart
+<deployment-upgrade-swh-service-update-impacted-chart>` with the new docker
+image version.
 
-.. _commit-changes-and-tag:
+.. _deployment-upgrade-swh-service-commit-changes-and-tag:
 
 Commit and tag
 ~~~~~~~~~~~~~~
 
 Commit and tag the changes.
 
-.. _update-impacted-chart:
+.. _deployment-upgrade-swh-service-labels-on-nodes:
 
-Update impacted chart
-~~~~~~~~~~~~~~~~~~~~~
+Labels on nodes
+~~~~~~~~~~~~~~~
 
-In the `swh-chart`_ repository, update the `values file
-<https://gitlab.softwareheritage.org/swh/infra/ci-cd/swh-charts/-/blob/production/values-swh-application-versions.yaml>`_
-with the corresponding new changed version.
+For now, we are using dedicated labels on nodes to run specific applications:
+
+- swh/rpc=true: rpc services, e.g. graphql
+- swh/cooker=true: cooker worker
+- swh/indexer=true: indexer journal client
+- swh/lister=true: lister worker
+- swh/loader=true: loader worker
+- swh/loader-metadata=true: loader-metadata worker
+
+In the following example:
+
+- cluster in {archive-staging-rke2, archive-production-rke2})
+- $node is an actual node hostname e.g. rancher-node-staging-rke2-worker[1, ...] or
+  rancher-node-metal0{1,2} (for production)
+- $new-label is a label of the form: ``swh/$service=true``
+
+To check the actual list of labels
+
+    kubectl --context $cluster get nodes --show-labels
+
+To install a label on a node:
+
+    kubectl --context $cluster label --overwrite node \
+      $node $new-label
 
-:ref:`ArgoCD <argocd-config>` will be in charge of deploying the changes in a rolling
-upgrade fashion.
 
 .. _swh-apps: https://gitlab.softwareheritage.org/swh/infra/swh-apps/
 .. _swh-chart: https://gitlab.softwareheritage.org/infra/ci-cd/swh-charts