sysadmin: Document the origin bulk update process

Related to swh/infra/sysadm-environment#5301

docs/sysadm/deployment/howto-bulk-ingest.rst

+.. _howto-bulk-ingest:
+
+How to bulk ingest a list of origins
+====================================
+
+.. admonition:: Intended audience
+   :class: important
+
+   sysadm staff members
+
+
+The scheduler provides a cli to send a list of origins directly in a rabbitmq queue. If a loading stack
+is configured to listen to this queue, these origins will be loaded by the loaders the classical way.
+
+.. warning:: Only a one-shot loading will be performed, a recurring task is not created.
+
+The following example explains how to launch an ingestion from a raw list of origins.
+
+The toolbox deployed in kubernetes contains all the configuration pre-installed to simplify the
+interaction with the scheduler. The example is based on this. You must have the `kubectl`
+command installed on your local environment and the configuration to access the staging and production
+clusters.
+
+- Deploy the loader stack with a queue configuration <prefix>:<usual queue names>
+
+  - for example `Activate oneshot loaders <https://gitlab.softwareheritage.org/swh/infra/ci-cd/swh-charts/-/commit/f478419e4f350f3710ad4d32b3c8875bcf0db812>`_
+
+- If not provided, sort the origins per loader type (git/svn/hg/cvs/...)
+
+  - A quick and dirty helper script could help: `snippets bulk import directory <https://gitlab.softwareheritage.org/swh/devel/snippets/-/tree/master/sysadmin/bulk_import>`_
+
+- Prepare your local environment, the next commands are for staging, adapt according to your needs
+
+.. code::
+
+  CONTEXT=archive-staging-rke2
+  NAMESPACE=swh-cassandra
+  TOOLBOX=toolbox-oneshot-loading
+  ORIGINS=git_origins.lst
+
+- Create a dedicated toolbox pod:
+
+.. code::
+
+  kubectl debug --context $CONTEXT -n $NAMESPACE \
+  $(kubectl --context $CONTEXT -n $NAMESPACE get pods -l app=swh-toolbox -o name | head -1) \
+     --container=swh-toolbox --copy-to=$TOOLBOX -- sleep infinity
+
+Cloning the pod allows to not stop the loading if an deployment happens before the end of the
+loading
+
+- Copy the file containing the list of origins in the pod
+
+.. code::
+
+  kubectl --context $CONTEXT cp $ORIGINS $NAMESPACE/$TOOLBOX:$ORIGINS -c swh-toolbox
+
+- Connect to the pod via kubectl or k9s
+
+.. code::
+
+  kubectl --context $CONTEXT exec --namespace $NAMESPACE -ti $TOOLBOX -c swh-toolbox -- bash
+
+- Populate the celery queue
+
+.. code::
+
+  export SWH_CONFIG_FILENAME=/etc/swh/config-scheduler.yml
+  ORIGINS=git_origins.lst
+  TASK_TYPE=load-git
+  MAX_TASKS=10000
+
+  nohup bash -c "cat $ORIGINS | swh scheduler -C $SWH_CONFIG_FILENAME origin \
+    send-origins-from-file-to-celery $TASK_TYPE --threshold=$MAX_TASKS \
+    --queue-name-prefix oneshot " | tee -a $ORIGINS.output &
+
+The process is detached from the terminal so you can exit the pod without stopping the process.
+It will run until the end unless the pod is restarted by a maintenance or crash.
+
+- Check the output in the log file and the queue in rabbitmq
+
+  - rabbitmq urls are available in the :ref:`services page <service-url>`
+
+- When the loading is done, remove the temporary toolbox pod
+
+.. code::
+
+  kubectl --context $CONTEXT --namespace $NAMESPACE delete pods $TOOLBOX
+

docs/sysadm/deployment/index.rst

@@ -14,3 +14,4 @@ SWH Software Deployment
    howto-add-forge-now-pipeline
    howto-add-forge-now-process-requests
    howto-add-journal-user-credential
+   howto-bulk-ingest