Skip to content

swh/production: Bulk ingest svn opencores.org

Guillaume Samson requested to merge bulk_ingest_opencores into production

Related to swh/infra/sysadm-environment#5427 (closed)

These modifications will deploy a job to bulk ingest the SVN repositories.
To ensure the repositories list will be processed one at a time, the maximum replica of the loader oneshot has been set at 1.

helm-diff 
[swh] Comparing changes between branches production and bulk_ingest_opencores (per environment)...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment staging, namespace swh...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra-next-version...
[swh] Generate config in bulk_ingest_opencores branch for environment staging...
[swh] Generate config in bulk_ingest_opencores branch for environment staging...
[swh] Generate config in bulk_ingest_opencores branch for environment staging...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment production, namespace swh...
[swh] Generate config in production branch for environment production, namespace swh-cassandra...
[swh] Generate config in production branch for environment production, namespace swh-cassandra-next-version...
[swh] Generate config in bulk_ingest_opencores branch for environment production...
[swh] Generate config in bulk_ingest_opencores branch for environment production...
[swh] Generate config in bulk_ingest_opencores branch for environment production...


------------- diff for environment staging namespace swh -------------

     _        __  __
   _| |_   _ / _|/ _|  between /tmp/swh-chart.swh.CWcKSygE/staging-swh.before, 139 documents
 / _' | | | | |_| |_       and /tmp/swh-chart.swh.CWcKSygE/staging-swh.after, 139 documents
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned no differences
        |___/



------------- diff for environment staging namespace swh-cassandra -------------

     _        __  __
   _| |_   _ / _|/ _|  between /tmp/swh-chart.swh.CWcKSygE/staging-swh-cassandra.before, 438 documents
 / _' | | | | |_| |_       and /tmp/swh-chart.swh.CWcKSygE/staging-swh-cassandra.after, 438 documents
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned no differences
        |___/



------------- diff for environment staging namespace swh-cassandra-next-version -------------

     _        __  __
   _| |_   _ / _|/ _|  between /tmp/swh-chart.swh.CWcKSygE/staging-swh-cassandra-next-version.before, 345 documents
 / _' | | | | |_| |_       and /tmp/swh-chart.swh.CWcKSygE/staging-swh-cassandra-next-version.after, 345 documents
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned no differences
        |___/



------------- diff for environment production namespace swh -------------

     _        __  __
   _| |_   _ / _|/ _|  between /tmp/swh-chart.swh.CWcKSygE/production-swh.before, 449 documents
 / _' | | | | |_| |_       and /tmp/swh-chart.swh.CWcKSygE/production-swh.after, 450 documents
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned one difference
        |___/

(file level)
    ---
    # Source: swh/templates/toolbox/bulk-load-job.yaml
    apiVersion: batch/v1
    kind: Job
    metadata:
      name: toolbox-bulk-load-oneshot-opencores
      namespace: swh
      labels:
        app: toolbox-bulk-load-oneshot-opencores
    spec:
      backoffLimit: 0
      template:
        metadata:
          labels:
            app: toolbox-bulk-load-oneshot-opencores
        spec:
          restartPolicy: Never
          initContainers:
          - name: prepare-configuration-scheduler
            image: "container-registry.softwareheritage.org/swh/infra/swh-apps/utils:20231211.1"
            imagePullPolicy: IfNotPresent
            command:
            - /entrypoints/prepare-configuration.sh
            volumeMounts:
            - name: config-utils
              mountPath: /entrypoints
              readOnly: true
            - name: configuration
              mountPath: /etc/swh
            - name: configuration-template
              mountPath: /etc/swh/configuration-template
            env:
            - name: AMQP_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: amqp-secrets
                  key: swhproducer-password
                  optional: false
            - name: POSTGRESQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: swh-scheduler-postgresql-common-secret
                  key: postgres-swh-scheduler-password
                  optional: false
          containers:
          - name: bulk-load
            image: "container-registry.softwareheritage.org/swh/infra/swh-apps/toolbox:20240910.1"
            imagePullPolicy: IfNotPresent
            command:
            - /opt/swh/bin/bulk_load.sh
            resources:
              requests:
                memory: 128Mi
                cpu: 100m
            env:
            - name: SWH_CONFIG_FILENAME
              value: /etc/swh/config.yml
            - name: ORIGINS
              value: opencores.lst
            - name: ORIGINS_URL
              value: "https://gitlab.softwareheritage.org/-/project/80/uploads/a4a8f1ca3c57dd2a994f982b9b4fa227/opencores.org-projects-opencores.org-ocsvn-all-svn-repos.txt"
            - name: TASK_TYPE
              value: load-svn
            - name: QUEUE_PREFIX
              value: oneshot
            - name: MAX_TASKS
              value: 250
            volumeMounts:
            - name: configuration
              mountPath: /etc/swh
            - name: toolbox-script-utils
              mountPath: /opt/swh/bin
              readOnly: true
          volumes:
          - name: configuration
            emptyDir: {}
          - name: configuration-template
            configMap:
              name: toolbox-scheduler-template
              items:
              - key: config.yml.template
                path: config.yml.template
          - name: config-utils
            configMap:
              name: config-utils
              defaultMode: 0555
          - name: toolbox-script-utils
            configMap:
              name: toolbox-script-utils
              defaultMode: 0555
    
  
------------- diff for environment production namespace swh-cassandra -------------

     _        __  __
   _| |_   _ / _|/ _|  between /tmp/swh-chart.swh.CWcKSygE/production-swh-cassandra.before, 219 documents
 / _' | | | | |_| |_       and /tmp/swh-chart.swh.CWcKSygE/production-swh-cassandra.after, 219 documents
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned no differences
        |___/

Merge request reports