[Add forge now] implement a CI pipeline

• triggered on issue creation

see https://hedgedoc.softwareheritage.org/ToS40kN1RuWqdOSx0xJ0aA

changed milestone to %Automate add forge now [Roadmap - Collect]

added priority:Normal label

assigned to @guillaume

Add Forge Now Requests Automation

POC summary (swh-environment and minikube)

Gitlab in swh-environment

Update the loader configuration to add add_forge_now queue.

diff --git a/docker/conf/loader.yml b/docker/conf/loader.yml
index efbdd25..a379bbf 100644
--- a/docker/conf/loader.yml
+++ b/docker/conf/loader.yml
@@ -54,3 +54,4 @@ celery:
     - save_code_now:swh.loader.svn.tasks.DumpMountAndLoadSvnRepository
     - save_code_now:swh.loader.svn.tasks.MountAndLoadSvnRepository
     - save_code_now:swh.loader.package.archive.tasks.LoadTarball
+    - add_forge_now:swh.loader.git.tasks.UpdateGitRepository

Create an override file to bind ports, declare dns aliases and create gitlab/gitlab-runner.

docker/docker-compose.override.yml

version: "2.1"
services:
  amqp:
    container_name: amqp
    networks:
      default:
        aliases:
          - amqp.local
    ports:
      - 15672:15672
      - 5672:5672
  swh-scheduler:
    container_name: swh-scheduler
    volumes:
      - "../swh-scheduler:/src/swh-scheduler"
    networks:
      default:
        aliases:
          - scheduler.local
          - saatchi.local
  swh-scheduler-db:
    container_name: swh-scheduler-db
    networks:
      default:
        aliases:
          - scheduler-db.local
    ports:
      - 5432:5432
  gitlab:
    image: 'gitlab/gitlab-ce:latest'
    restart: always
    hostname: 'gitlab'
    container_name: gitlab
    environment:
      GITLAB_ROOT_PASSWORD: turlututu
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://gitlab.local'
    ports:
      - '80:80'
      - '443:443'
    volumes:
      - 'gitlab_config:/etc/gitlab'
      - 'gitlab_logs:/var/log/gitlab'
      - 'gitlab_data:/var/opt/gitlab'
    networks:
      default:
        aliases:
          - gitlab.local
  gitlab-runner:
    image: gitlab/gitlab-runner:ubuntu
    container_name: gitlab-runner
    restart: always
    depends_on:
      - gitlab
    volumes:
      - /run/docker.sock:/var/run/docker.sock
      - 'gitlab_gitlab-runner:/etc/gitlab-runner'

volumes:
  gitlab_config:
  gitlab_logs:
  gitlab_data:
  gitlab_gitlab-runner:

Add gitlab external_url in /etc/hosts.

ᐅ awk 'NR==2' /etc/hosts
127.0.1.1       gsamson-laptop.softwareheritage.org     gsamson-laptop gitlab.local

Create gitlab group and user

Create group Infra.

| Admin Area > Groups

Create a new user.

| Admin Area > Users

Invite the new user in group Infra.

| Admin Area > Groups > Infra

Create the project repository add-forge-now-requests.

[Docker executor] Create, register and configure a runner.

Disable shared and group runners for the new project.

| Infra > add-forge-now-requests > CI/CD Settings > Runners

Create a new project runner with a pipeline tag.

| Infra > add-forge-now-requests > CI/CD Settings > Runners

Register the runner.

ᐅ dck exec -ti gitlab-runner bash
root@05bd528e5d1b:/# gitlab-runner register  --url http://gitlab.local  --token xxxxxxxxxxxxxxxxxxx
Runtime platform                                    arch=amd64 os=linux pid=31 revision=782e15da version=16.2.0
Running in system-mode.

Enter the GitLab instance URL (for example, https://gitlab.com/):
[http://gitlab.local]:
Verifying runner... is valid                        runner=yy3nKeCPf
Enter a name for the runner. This is stored only in the local config.toml file:
[05bd528e5d1b]: addForgeNowRequests Runner
Enter an executor: docker-autoscaler, docker+machine, instance, docker-windows, parallels, ssh, virtualbox, custom, docker, shell, kubernetes:
docker
Enter the default Docker image (for example, ruby:2.7):
ruby:2.7
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Configuration (with the authentication token) was saved in "/etc/gitlab-runner/config.toml"

Allow runner executor to access swh-environment docker network.

ᐅ dck exec -ti gitlab-runner bash
root@05bd528e5d1b:/# echo '    network_mode = "docker_default"' >> /etc/gitlab-runner/config.toml
root@05bd528e5d1b:/# cat /etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "addForgeNowRequests Runner"
  url = "http://gitlab.local"
  id = 1
  token = "xxxxxxxxxxxxxxxxxxxxxxxx"
  token_obtained_at = 2023-07-26T13:53:25Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    tls_verify = false
    image = "ruby:2.7"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false12
    volumes = ["/cache"]
    shm_size = 0
    network_mode = "docker_default"

Check the runner configuration was reloaded.

ᐅ dck logs gitlab-runner --tail 1
gitlab-runner  | Configuration loaded                                builds=0 max_builds=1

Check the runner connection status.

[Kubernetes executor] Create, register and configure a runner.

Create a new project runner with a pipeline tag.

| Infra > add-forge-now-requests > CI/CD Settings > Runners

Create and deploy a runner Helm chart.

ᐅ helm repo add gitlab https://charts.gitlab.io
ᐅ helm show values gitlab/gitlab-runner > gitlab-runner-chart-values.yaml

Update the gitlab-runner-chart-values.yaml file with appropriate values.

## GitLab Runner Image
##
## By default it's using registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v{VERSION}
## where {VERSION} is taken from Chart.yaml from appVersion field
##
## ref: https://gitlab.com/gitlab-org/gitlab-runner/container_registry/29383?orderBy=NAME&sort=asc&search[]=alpine-v&search[]=
##
## Note: If you change the image to the ubuntu release
##       don't forget to change the securityContext;
##       these images run on different user IDs.
##
image:
  registry: registry.gitlab.com
  image: gitlab-org/gitlab-runner
  # tag: alpine-v11.6.0

## When using GitLab Runner Helm Chart with gitlab-runner-ubi-images (https://gitlab.com/gitlab-org/ci-cd/gitlab-runner-ubi-images/container_registry)
## the installation fails because dumb-init is not packaged in the image. However, the tini is present.
## This configuration will allow gitlab-runner-ubi-images users to explicitly enabled the use of `tini` instead of `dumb-init`
useTini: false

## Specify a imagePullPolicy for the main runner deployment
## 'Always' if imageTag is 'latest', else set to 'IfNotPresent'
##
## Note: it does not apply to job containers launched by this executor.
## Use `pull_policy` in [runners.kubernetes] to change it.
##
## ref: https://kubernetes.io/docs/concepts/containers/images/#pre-pulled-images
##
imagePullPolicy: IfNotPresent

## Specifying ImagePullSecrets on a Pod
## Kubernetes supports specifying container image registry keys on a Pod.
## ref: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
##
# imagePullSecrets:
#   - name: "image-pull-secret"

## Timeout, in seconds, for liveness and readiness probes of a runner pod.
# probeTimeoutSeconds: 1

## How many runner pods to launch.
##
# replicas: 1

## How many old ReplicaSets for this Deployment you want to retain
# revisionHistoryLimit: 10

## The GitLab Server URL (with protocol) that want to register the runner against
## ref: https://docs.gitlab.com/runner/commands/index.html#gitlab-runner-register
##
# gitlabUrl: http://gitlab.your-domain.com/
gitlabUrl: http://gitlab.local/

## DEPRECATED: The Registration Token for adding new Runners to the GitLab Server.
##
## ref: https://docs.gitlab.com/ee/ci/runners/new_creation_workflow.html
##
# runnerRegistrationToken: ""
runnerRegistrationToken: "xxxxxxxxxxxxxxxxxxxxxxxxxx"

## The Runner Token for adding new Runners to the GitLab Server. This must
## be retrieved from your GitLab Instance. It is token of already registered runner.
## ref: (we don't yet have docs for that, but we want to use existing token)
##
# runnerToken: ""
#
runnerToken: "xxxxxxxxxxxxxxxxxxxxxxxx"

## Unregister all runners before termination
##
## Updating the runner's chart version or configuration will cause the runner container
## to be terminated and created again. This may cause your Gitlab instance to reference
## non-existant runners. Un-registering the runner before termination mitigates this issue.
## ref: https://docs.gitlab.com/runner/commands/index.html#gitlab-runner-unregister
##
# unregisterRunners: true

## When stopping the runner, give it time to wait for its jobs to terminate.
##
## Updating the runner's chart version or configuration will cause the runner container
## to be terminated with a graceful stop request. terminationGracePeriodSeconds
## instructs Kubernetes to wait long enough for the runner pod to terminate gracefully.
## ref: https://docs.gitlab.com/runner/commands/#signals
terminationGracePeriodSeconds: 3600

## Set the certsSecretName in order to pass custom certficates for GitLab Runner to use
## Provide resource name for a Kubernetes Secret Object in the same namespace,
## this is used to populate the /home/gitlab-runner/.gitlab-runner/certs/ directory
## ref: https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates-targeting-the-gitlab-server
##
# certsSecretName:

## Configure the maximum number of concurrent jobs
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
concurrent: 10

## Defines in seconds how often to check GitLab for a new builds
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
checkInterval: 30

## Configure GitLab Runner's logging level. Available values are: debug, info, warn, error, fatal, panic
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
# logLevel:

## Configure GitLab Runner's logging format. Available values are: runner, text, json
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
# logFormat:

## Configure GitLab Runner's Sentry DSN.
## ref https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
# sentryDsn:

## A custom bash script that will be executed prior to the invocation
## gitlab-runner process
#
#preEntrypointScript: |
#  echo "hello"

## Specify whether the runner should start the session server.
## Defaults to false
## ref:
##
## When sessionServer is enabled, the user can either provide a public publicIP
## or rely on the external IP auto discovery
## When a serviceAccountName is used with the automounting to the pod disable,
## we recommend the usage of the publicIP
sessionServer:
  enabled: false
  # annotations: {}
  # timeout: 1800
  # internalPort: 8093
  # externalPort: 9000
  # publicIP: ""
  # loadBalancerSourceRanges:
  #   - 1.2.3.4/32

## For RBAC support:
rbac:
  create: true

  ## Define list of rules to be added to the rbac role permissions.
  ## Each rule supports the keys:
  ## - apiGroups: default "" (indicates the core API group) if missing or empty.
  ## - resources: default "*" if missing or empty.
  ## - verbs: default "*" if missing or empty.
  ##
  ## Read more about the recommended rules on the following link
  ##
  ## ref: https://docs.gitlab.com/runner/executors/kubernetes.html#configuring-executor-service-account
  ##
  rules: []
  # - resources: ["configmaps", "pods", "pods/attach", "secrets", "services"]
  #   verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]
  # - apiGroups: [""]
  #   resources: ["pods/exec"]
  #   verbs: ["create", "patch", "delete"]

  ## Run the gitlab-bastion container with the ability to deploy/manage containers of jobs
  ## cluster-wide or only within namespace
  clusterWideAccess: false

  ## Use the following Kubernetes Service Account name if RBAC is disabled in this Helm chart (see rbac.create)
  ##
  # serviceAccountName: default

  ## Specify annotations for Service Accounts, useful for annotations such as eks.amazonaws.com/role-arn
  ##
  ## ref: https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html
  ##
  # serviceAccountAnnotations: {}

  ## Use podSecurity Policy
  ## ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/
  podSecurityPolicy:
    enabled: false
    resourceNames:
    - gitlab-runner

  ## Specify one or more imagePullSecrets used for pulling the runner image
  ##
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account
  ##
  # imagePullSecrets: []

## Configure integrated Prometheus metrics exporter
##
## ref: https://docs.gitlab.com/runner/monitoring/#configuration-of-the-metrics-http-server
##
metrics:
  enabled: false

  ## Define a name for the metrics port
  ##
  portName: metrics

  ## Provide a port number for the integrated Prometheus metrics exporter
  ##
  port: 9252

  ## Configure a prometheus-operator serviceMonitor to allow autodetection of
  ## the scraping target. Requires enabling the service resource below.
  ##
  serviceMonitor:
    enabled: false

    ## Provide additional labels to the service monitor ressource
    ##
    ## labels: {}

    ## Define a scrape interval (otherwise prometheus default is used)
    ##
    ## ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
    ##
    # interval: ""

    ## Specify the scrape protocol scheme e.g., https or http
    ##
    # scheme: "http"

    ## Supply a tls configuration for the service monitor
    ##
    ## ref: https://github.com/helm/charts/blob/master/stable/prometheus-operator/crds/crd-servicemonitor.yaml
    ##
    # tlsConfig: {}

    ## The URI path where prometheus metrics can be scraped from
    ##
    # path: "/metrics"

    ## A list of MetricRelabelConfigs to apply to samples before ingestion
    ##
    ## ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs
    ##
    # metricRelabelings: []

    ## A list of RelabelConfigs to apply to samples before scraping
    ##
    ## ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
    ##
    ## relabelings: []

## Configure a service resource e.g., to allow scraping metrics via
## prometheus-operator serviceMonitor
service:
  enabled: false

  ## Provide additonal labels for the service
  ##
  # labels: {}

  ## Provide additonal annotations for the service
  ##
  # annotations: {}

  ## Define a specific ClusterIP if you do not want a dynamic one
  ##
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#choosing-your-own-ip-address
  ##
  # clusterIP: ""

  ## Define a list of one or more external IPs for this service
  ##
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#external-ips
  ##
  # externalIPs: []

  ## Provide a specific loadbalancerIP e.g., of an external Loadbalancer
  ##
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer
  ##
  # loadBalancerIP: ""

  ## Provide a list of source IP ranges to have access to this service
  ##
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#aws-nlb-support
  ##
  # loadBalancerSourceRanges: []

  ## Specify the service type e.g., ClusterIP, NodePort, Loadbalancer or ExternalName
  ##
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types
  ##
  type: ClusterIP

  ## Specify the services metrics nodeport if you use a service of type nodePort
  ##
  # metrics:

    ## Specify the node port under which the prometheus metrics of the runner are made
    ## available.
    ##
    ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#nodeport
    ##
    # nodePort: ""

  ## Provide a list of additional ports to be exposed by this service
  ##
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#defining-a-service
  ##
  # additionalPorts: []

## Configuration for the Pods that the runner launches for each new job
##
runners:
  # runner configuration, where the multi line strings is evaluated as
  # template so you can specify helm values inside of it.
  #
  # tpl: https://helm.sh/docs/howto/charts_tips_and_tricks/#using-the-tpl-function
  # runner configuration: https://docs.gitlab.com/runner/configuration/advanced-configuration.html
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        image = "ubuntu:22.04"
        [[runners.kubernetes.host_aliases]]
          ip = "128.93.73.161"
          hostnames = ["gitlab.local", "scheduler.local", "saatchi.local", "amqp.local", "scheduler-db.local"]
  tags: "AddForgeNowRequest"
  name: "AddForgeNowRequest"
  privileged: true

  ## Which executor should be used
  ##
  # executor: kubernetes

  ## DEPRECATED: Specify whether the runner should be locked to a specific project: true, false.
  ##
  ## ref: https://docs.gitlab.com/ee/ci/runners/new_creation_workflow.html
  ##
  # locked: true

  ## DEPRECATED: Specify the tags associated with the runner. Comma-separated list of tags.
  ##
  ## ref: https://docs.gitlab.com/ee/ci/runners/new_creation_workflow.html
  ##
  # tags: ""

  ## Specify the name for the runner.
  ##
  # name: ""

  ## DEPRECATED:Specify the maximum timeout (in seconds) that will be set for job when using this Runner
  ##
  ## ref: https://docs.gitlab.com/ee/ci/runners/new_creation_workflow.html
  ##
  # maximumTimeout: ""

  ## DEPRECATED: Specify if jobs without tags should be run.
  ##
  ## ref: https://docs.gitlab.com/ee/ci/runners/new_creation_workflow.html
  ##
  # runUntagged: true

  ## DEPRECATED: Specify whether the runner should only run protected branches.
  ##
  ## ref: https://docs.gitlab.com/ee/ci/runners/new_creation_workflow.html
  ##
  # protected: true

  ## The name of the secret containing runner-token and runner-registration-token
  # secret: gitlab-runner

  ## Distributed runners caching
  ## ref: https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching
  ##
  ## If you want to use s3 based distributing caching:
  ## First of all you need to uncomment General settings and S3 settings sections.
  ##
  ## Create a secret 's3access' containing 'accesskey' & 'secretkey'
  ## ref: https://aws.amazon.com/blogs/security/wheres-my-secret-access-key/
  ##
  ## $ kubectl create secret generic s3access \
  ##   --from-literal=accesskey="YourAccessKey" \
  ##   --from-literal=secretkey="YourSecretKey"
  ## ref: https://kubernetes.io/docs/concepts/configuration/secret/
  ##
  ## If you want to use gcs based distributing caching:
  ## First of all you need to uncomment General settings and GCS settings sections.
  ##
  ## Access using credentials file:
  ## Create a secret 'google-application-credentials' containing your application credentials file.
  ## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscachegcs-section
  ## You could configure
  ## $ kubectl create secret generic google-application-credentials \
  ##   --from-file=gcs-application-credentials-file=./path-to-your-google-application-credentials-file.json
  ## ref: https://kubernetes.io/docs/concepts/configuration/secret/
  ##
  ## Access using access-id and private-key:
  ## Create a secret 'gcsaccess' containing 'gcs-access-id' & 'gcs-private-key'.
  ## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscachegcs-section
  ## You could configure
  ## $ kubectl create secret generic gcsaccess \
  ##   --from-literal=gcs-access-id="YourAccessID" \
  ##   --from-literal=gcs-private-key="YourPrivateKey"
  ## ref: https://kubernetes.io/docs/concepts/configuration/secret/
  ##
  ## If you want to use Azure-based distributed caching:
  ## First, uncomment General settings.
  ##
  ## Create a secret 'azureaccess' containing 'azure-account-name' & 'azure-account-key'
  ## ref: https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction
  ##
  ## $ kubectl create secret generic azureaccess \
  ##   --from-literal=azure-account-name="YourAccountName" \
  ##   --from-literal=azure-account-key="YourAccountKey"
  ## ref: https://kubernetes.io/docs/concepts/configuration/secret/

  cache: {}
    ## S3 the name of the secret.
    # secretName: s3access
    ## Use this line for access using gcs-access-id and gcs-private-key
    # secretName: gcsaccess
    ## Use this line for access using google-application-credentials file
    # secretName: google-application-credentials
    ## Use this line for access using Azure with azure-account-name and azure-account-key
    # secretName: azureaccess

## Specify the name of the scheduler which used to schedule runner pods.
## Kubernetes supports multiple scheduler configurations.
## ref: https://kubernetes.io/docs/reference/scheduling
# schedulerName: "my-custom-scheduler"

## Configure securitycontext for the main container
## ref: http://kubernetes.io/docs/user-guide/security-context/
##
securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: false
  runAsNonRoot: true
  privileged: false
  capabilities:
    drop: ["ALL"]

## Configure securitycontext valid for the whole pod
## ref: http://kubernetes.io/docs/user-guide/security-context/
##
podSecurityContext:
  runAsUser: 100
  # runAsGroup: 65533
  fsGroup: 65533
  # supplementalGroups: [65533]

  ## Note: values for the ubuntu image:
  # runAsUser: 999
  # fsGroup: 999

## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources: {}
  # limits:
  #   memory: 256Mi
  #   cpu: 200m
  # requests:
  #   memory: 128Mi
  #   cpu: 100m

## Affinity for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
##
affinity: {}

## Node labels for pod assignment
## Ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}
  # Example: The gitlab runner manager should not run on spot instances so you can assign
  # them to the regular worker nodes only.
  # node-role.kubernetes.io/worker: "true"

## List of node taints to tolerate (requires Kubernetes >= 1.6)
## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
tolerations: []
  # Example: Regular worker nodes may have a taint, thus you need to tolerate the taint
  # when you assign the gitlab runner manager with nodeSelector or affinity to the nodes.
  # - key: "node-role.kubernetes.io/worker"
  #   operator: "Exists"

## Configure environment variables that will be present when the registration command runs
## This provides further control over the registration process and the config.toml file
## ref: `gitlab-runner register --help`
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html
##
# envVars:
#   - name: RUNNER_EXECUTOR
#     value: kubernetes

## list of hosts and IPs that will be injected into the pod's hosts file
hostAliases:
  - ip: 128.93.73.161
    hostnames:
      - 'gitlab.local'
      - 'scheduler.local'
      - 'saatchi.local'
      - 'amqp.local'
      - 'scheduler-db.local'
  # Example:
  # - ip: "127.0.0.1"
  #   hostnames:
  #   - "foo.local"
  #   - "bar.local"
  # - ip: "10.1.2.3"
  #   hostnames:
  #   - "foo.remote"
  #   - "bar.remote"

## Annotations to be added to manager pod
##
podAnnotations: {}
  # Example:
  # iam.amazonaws.com/role: <my_role_arn>

## Labels to be added to manager pod
##
podLabels: {}
  # Example:
  # owner.team: <my_cool_team>

## HPA support for custom metrics:
## This section enables runners to autoscale based on defined custom metrics.
## In order to use this functionality, Need to enable a custom metrics API server by
## implementing "custom.metrics.k8s.io" using supported third party adapter
## Example: https://github.com/directxman12/k8s-prometheus-adapter
##
#hpa: {}
  # minReplicas: 1
  # maxReplicas: 10
  # metrics:
  # - type: Pods
  #   pods:
  #     metricName: gitlab_runner_jobs
  #     targetAverageValue: 400m

## Configure priorityClassName for manager pod. See k8s docs for more info on how pod priority works:
##  https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/
priorityClassName: ""

## Secrets to be additionally mounted to the containers.
## All secrets are mounted through init-runner-secrets volume
## and placed as readonly at /init-secrets in the init container
## and finally copied to an in-memory volume runner-secrets that is
## mounted at /secrets.
secrets: []
  # Example:
  # - name: my-secret
  # - name: myOtherSecret
  #   items:
  #     - key: key_one
  #       path: path_one

## Additional config files to mount in the containers in `/configmaps`.
##
## Please note that a number of keys are reserved by the runner.
## See https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/main/templates/configmap.yaml
## for a current list.
configMaps: {}

## Additional volumeMounts to add to the runner container
##
volumeMounts: []
  # Example:
  # - name: my-volume
  #   mountPath: /mount/path

## Additional volumes to add to the runner deployment
##
volumes: []
  # Example:
  # - name: my-volume
  #   persistentVolumeClaim:
  #     claimName: my-pvc

ᐅ helm install -f gitlab-runner-chart-values.yaml gitlab-runner gitlab/gitlab-runner
NAME: gitlab-runner
LAST DEPLOYED: Thu Aug 24 16:00:26 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Your GitLab Runner should now be registered against the GitLab instance reachable at: "http://gitlab.local/"

Runner namespace "default" was found in runners.config template.

Create label and milestone.

Create a project label.

| Infra > add-forge-now-requests > Labels

Create a project milestone.

| Infra > add-forge-now-requests > Milestones

Create a pipeline trigger token and a project access token.

Create a trigger pipeline token.

| Infra > add-forge-now-requests > CI/CD Settings > Pipeline trigger tokens

Create a project access token for issues processing.

| Infra > add-forge-now-requests > Token Access

Register access token in a masked and protected variable.

| Infra > add-forge-now-requests > CI/CD Settings > Variables

Build container image and push it in minikube registry.

Minikube with a registry.

ᐅ sudo cat /etc/docker/daemon.json
{
  "insecure-registries" : ["http://192.168.49.2:5000"]
}

ᐅ minikube start --insecure-registry "192.168.49.0/24"
ᐅ minikube addons enable ingress
ᐅ minikube addons enable registry

ᐅ curl -s http://192.168.49.2:5000/v2/_catalog
{"repositories":[]}

Build scheduler image.

Dockerfile

FROM python:3.10-bookworm

RUN apt-get -y update && \
    apt-get -y upgrade && \
    apt-get install -y libcmph-dev netcat-openbsd jq && \
    apt clean && \
    addgroup --gid 1000 swh && \
    useradd --gid 1000 --uid 1000 -m -d /opt/swh swh
USER swh
WORKDIR /opt/swh

COPY --chown=swh:swh scheduler.yml /opt/swh

ENV PATH=/opt/swh/.local/bin:$PATH

RUN /usr/local/bin/python -m pip install --upgrade pip && \
    pip install --no-cache-dir swh-scheduler

ENV SWH_CONFIG_FILENAME=/opt/swh/scheduler.yml

CMD [ "/bin/bash", "-i" ]

ᐅ docker build --no-cache -t $(minikube ip):5000/add-forge-now:$(date +%F|sed 's/-//g').1 .
ᐅ docker push $(minikube ip):5000/add-forge-now:$(date +%F|sed 's/-//g').1

ᐅ curl -s http://192.168.49.2:5000/v2/_catalog
{"repositories":["add-forge-now"]}
ᐅ curl -s http://192.168.49.2:5000/v2/add-forge-now/tags/list | jq
{
  "name": "add-forge-now",
  "tags": [
    "20230817.1"
  ]
}

Clone project repository and copy project files.

ᐅ git clone http://gitlab.local/infra/add-forge-now-requests.git
ᐅ cp 
-rp test_amqp.py test_pg.py scheduler.yml Dockerfile .gitlab-ci.yml pictures README.md add-forge-now-requests/

Commit and push...

Pipeline.

.gitlab-ci.yml

image:
  name: 192.168.49.2:5000/add-forge-now:20230817.1

default:
  before_script:
    - |
      if [ -f build.env ] ; then
      export SCHEDULER_URL=$(awk -F '=' '/SCHEDULER/{print $2}' build.env)
      export ENV=$(awk -F '=' '/ENV/{print $2}' build.env)
      export ISSUE_ID=$(awk -F '=' '/ISSUE_ID/{print $2}' build.env)
      fi
  tags:
    - AddForgeNowRequest

variables:
  STAGING_SCHEDULER_URL: http://scheduler.local:5008
  PRODUCTION_SCHEDULER_URL: http://saatchi.local:5008
  LISTING_DELAY: 20
  MILESTONE_ID: 1

# STAGES

stages:
  - staging_list-origins
  - staging_ingest-origins
  - staging_set_status
  #- production_list-origins
  #- production_ingest-origins
  - production_set_status

# TEMPLATES

.register-lister_template:
  script:
    - echo $LISTER_TYPE $INSTANCE_NAME
    - swh scheduler --config-file $SWH_CONFIG_FILENAME
      --url $SCHEDULER_URL
      add-forge-now --preset $ENV
      register-lister $LISTER_TYPE
      instance=$INSTANCE_NAME
    - sleep $LISTING_DELAY
  rules:
    - if: $CI_PIPELINE_SOURCE == "trigger"
      when: on_success

.check-listed-origins:
  script:
    - echo $LISTER_TYPE $INSTANCE_NAME $ENV $SCHEDULER_URL
    - swh scheduler --url $SCHEDULER_URL
      origin check-listed-origins -l
      $LISTER_TYPE
      $INSTANCE_NAME
  rules:
    - if: $CI_PIPELINE_SOURCE == "trigger"
      when: on_success

.schedule-first-visits:
  script:
    - echo $LISTER_TYPE $INSTANCE_NAME $ENV $SCHEDULER_URL
    - swh scheduler --config-file $SWH_CONFIG_FILENAME
      --url $SCHEDULER_URL
      add-forge-now --preset $ENV
      schedule-first-visits
        --type-name git
        --lister-name $LISTER_TYPE
        --lister-instance-name $INSTANCE_NAME
  rules:
    - if: $CI_PIPELINE_SOURCE == "trigger"
      when: on_success

.check-ingested-origins:
  script:
    - echo $LISTER_TYPE $INSTANCE_NAME $ENV $SCHEDULER_URL
    - swh scheduler --url $SCHEDULER_URL
      origin check-ingested-origins -l -w
      $LISTER_TYPE
      $INSTANCE_NAME
  rules:
    - if: $CI_PIPELINE_SOURCE == "trigger"
      when: on_success

# PIPELINE

check_and_prepare:
  stage: .pre
  script:
    #- env
    - env | grep "SWH_CONFIG_FILENAME"
    - if [ -n "$SWH_CONFIG_FILENAME" ] ; then cat "$SWH_CONFIG_FILENAME"; fi
    - swh scheduler -h
    - for url in $STAGING_SCHEDULER_URL $PRODUCTION_SCHEDULER_URL; do
      echo -e "\n$url"; curl -sI $url; done
    - ./test_amqp.py
    - ./test_pg.py
    - curl -I http://amqp.local:15672
    - env | grep "LISTER_TYPE\|INSTANCE_NAME"
    - curl -sI --connect-timeout 5 https://${INSTANCE_NAME}. | grep '^HTTP.*\(2..\|3..\)'
    - |
      ISSUE_ID=$(curl -s -X POST -H "PRIVATE-TOKEN: ${ADD_FORGE_NOW_ISSUE_TOKEN}" \
      "http://gitlab.local/api/v4/projects/1/issues?\
      title=%5BAdd%20Forge%20Now%5D%20process%20https%3A%2F%2F${INSTANCE_NAME}&\
      labels=AddForgeNow&description=Type%3A%20${LISTER_TYPE}&\
      milestone_id=${MILESTONE_ID}" \
      | jq '.iid')
    - echo "ENV=staging" > build.env
    - echo "SCHEDULER_URL=$STAGING_SCHEDULER_URL" >> build.env
    - echo "ISSUE_ID=$ISSUE_ID" >> build.env
  artifacts:
    untracked: false
    expire_in: "7 days"
    paths:
      - "build.env"
  rules:
    - if: $CI_PIPELINE_SOURCE == "trigger"

1s_register-lister:
  stage: staging_list-origins
  extends: .register-lister_template
  dependencies:
    - "check_and_prepare"

2s_check-listed-origins:
  stage: staging_list-origins
  extends: .check-listed-origins
  needs:
    - job: check_and_prepare
      artifacts: true
    - job: 1s_register-lister

3s_schedule-first-visits:
  stage: staging_ingest-origins
  extends: .schedule-first-visits

4s_check-ingested-origins:
  stage: staging_ingest-origins
  extends: .check-ingested-origins
  needs:
    - job: check_and_prepare
      artifacts: true
    - job: 3s_schedule-first-visits

staging_status_successful:
  stage: staging_set_status
  script:
    - echo "set webapp request status to scheduled"
    - |
      curl -s -X POST -H "PRIVATE-TOKEN: ${ADD_FORGE_NOW_ISSUE_TOKEN}" \
      "http://gitlab.local/api/v4/projects/1/issues/${ISSUE_ID}/notes?\
      body=Request%20successfully%20processed%20in%20${ENV}%20environment%2E%20%20%0D%0A\
      See%20pipeline%20details%3A%20${CI_PIPELINE_URL}%2E"
    - echo "ENV=production" > build.env
    - echo "SCHEDULER_URL=$PRODUCTION_SCHEDULER_URL" >> build.env
    - echo "ISSUE_ID=$ISSUE_ID" >> build.env
  artifacts:
    untracked: false
    expire_in: "7 days"
    paths:
      - "build.env"
  rules:
    - if: $CI_PIPELINE_SOURCE == "trigger"
      when: on_success
  dependencies:
    - "check_and_prepare"

staging_status_unsuccessful:
  stage: staging_set_status
  script:
    - |
      curl -s -X POST -H "PRIVATE-TOKEN: ${ADD_FORGE_NOW_ISSUE_TOKEN}" \
      "http://gitlab.local/api/v4/projects/1/issues/${ISSUE_ID}/notes?\
      body=Request%20completly%20messed%20up%20in%20${ENV}%20environment%2E%20%20%0D%0A\
      See%20pipeline%20details%3A%20${CI_PIPELINE_URL}%2E"
    - echo "ENV=production" > build.env
    - echo "SCHEDULER_URL=$PRODUCTION_SCHEDULER_URL" >> build.env
    - echo "ISSUE_ID=$ISSUE_ID" >> build.env
  artifacts:
    untracked: false
    expire_in: "7 days"
    paths:
      - "build.env"
  rules:
    - if: $CI_PIPELINE_SOURCE == "trigger"
      when: on_failure
  dependencies:
    - "check_and_prepare"

#1p_register-lister:
#  stage: production_list-origins
#  extends: .register-lister_template
#  dependencies:
#    - "staging_status_successful"
#
#2p_check-listed-origins:
#  stage: production_list-origins
#  extends: .check-listed-origins
#  needs:
#    - job: staging_status_successful
#      artifacts: true
#    - job: 1p_register-lister
#
#3p_schedule-first-visits:
#  stage: production_ingest-origins
#  extends: .schedule-first-visits
#
#4p_check-ingested-origins:
#  stage: production_ingest-origins
#  extends: .check-ingested-origins
#  needs:
#    - job: staging_status_successful
#      artifacts: true
#    - job: 3p_schedule-first-visits
#
production_status_successful:
  stage: production_set_status
  script:
    - echo "set webapp status to first-origin-loaded"
    - cat build.env
    - echo $ADD_FORGE_NOW_ISSUE_TOKEN
    - |
      curl -s -X POST -H "PRIVATE-TOKEN: ${ADD_FORGE_NOW_ISSUE_TOKEN}" \
      "http://gitlab.local/api/v4/projects/1/issues/${ISSUE_ID}/notes?\
      body=Request%20successfully%20processed%20in%20${ENV}%20environment%2E%20%20%0D%0A\
      See%20pipeline%20details%3A%20${CI_PIPELINE_URL}%2E"
    - |
      curl -s -X PUT -H "PRIVATE-TOKEN: ${ADD_FORGE_NOW_ISSUE_TOKEN}" \
      "http://gitlab.local/api/v4/projects/1/issues/${ISSUE_ID}?state_event=close"
  rules:
    - if: $CI_PIPELINE_SOURCE == "trigger"
      when: on_success
  dependencies:
    - "staging_status_successful"

production_status_unsuccessful:
  stage: production_set_status
  script:
    - echo "set webapp status to unsuccessful"
    - cat build.env
    - echo $ADD_FORGE_NOW_ISSUE_TOKEN
    - |
      curl -s -X POST -H "PRIVATE-TOKEN: ${ADD_FORGE_NOW_ISSUE_TOKEN}" \
      "http://gitlab.local/api/v4/projects/1/issues/${ISSUE_ID}/notes?\
      body=Request%20completly%20messed%20up%20in%20${ENV}%20environment%2E%20%20%0D%0A\
      See%20pipeline%20details%3A%20${CI_PIPELINE_URL}%2E"
  rules:
    - if: $CI_PIPELINE_SOURCE == "trigger"
      when: on_failure
  dependencies:
    - "staging_status_unsuccessful"

Finally trigger a pipeline.

ᐅ TOKEN=glptt-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
ᐅ REF_NAME=main
ᐅ curl -X POST \                                                   
     --fail \       
     -F "token=$TOKEN" \                                       
     -F "ref=$REF_NAME" \        
     -F "variables[LISTER_TYPE]=cgit" \                                                                                        
     -F "variables[INSTANCE_NAME]=git.spwhitton.name" \                                                                        
http://gitlab.local/api/v4/projects/1/trigger/pipeline                                                                         
{"id":175,"iid":106,"project_id":1,"sha":"0ff811871248aa51ae816e3f09fc89511dbdc5d9","ref":"main","status":"created","source":"trigger","created_at":"2023-08-25T13:21:32.887Z","updated_at":"2023-08-25T13:21:32.887Z","web_url":"http://gitlab.local/infra/ad
d-forge-now-requests/-/pipelines/175","before_sha":"0000000000000000000000000000000000000000","tag":false,"yaml_errors":null,"user":{"id":2,"username":"gsamson","name":"gsamson","state":"active","avatar_url":"https://www.gravatar.com/avatar/b95e8133f9a93
19c0b9f975c5ab5b7b8?s=80\u0026d=identicon","web_url":"http://gitlab.local/gsamson"},"started_at":null,"finished_at":null,"committed_at":null,"duration":null,"queued_duration":null,"coverage":null,"detailed_status":{"icon":"status_created","text":"created
","label":"created","group":"created","tooltip":"created","has_details":true,"details_path":"/infra/add-forge-now-requests/-/pipelines/175","illustration":null,"favicon":"/assets/ci_favicons/favicon_status_created-4b975aa976d24e5a3ea7cd9a5713e6ce2cd9afd0
8b910415e96675de35f64955.png"}}

reset swh-enviromnent

ᐅ dck down
ᐅ docker volume ls | awk '{print $2}' | grep -v 'minikube\|VOLUME\|gitlab\|runner' | xargs docker volume rm

mentioned in commit swh-apps@aeb71fb4

mentioned in commit add-forge-now-requests@d393e15b

mentioned in commit add-forge-now-requests@372d8466

mentioned in commit add-forge-now-requests@cb6bae09

mentioned in commit add-forge-now-requests@583406e2

mentioned in commit add-forge-now-requests@66bd62f2

mentioned in commit add-forge-now-requests@e346c332

mentioned in commit add-forge-now-requests@2a724d40

mentioned in commit add-forge-now-requests@5506017a

mentioned in commit add-forge-now-requests@0c3584d7

mentioned in commit add-forge-now-requests@0f3f68b7

mentioned in commit add-forge-now-requests@406351bd

mentioned in commit add-forge-now-requests@1d7bd0d4

mentioned in commit add-forge-now-requests@2cf551bf

mentioned in commit add-forge-now-requests@256ac8ad

mentioned in commit add-forge-now-requests@ab30a710

mentioned in commit add-forge-now-requests@4a445944

mentioned in commit add-forge-now-requests@38b06883

mentioned in commit add-forge-now-requests@13ec4b61

mentioned in commit add-forge-now-requests@bd51a0ad

mentioned in commit add-forge-now-requests@4c29c184

closed