Skip to content

Automate the bulk loading of origins in a kubernetes job

The bulk loading of origins can be quite long.

The commands to launch [1] are in the toolbox so it could be possible to automate the process in a kubernetes job not have to launch everything manually.

The idea is to have a part in the helm chart that can handle that.

The information to have are:

  • an url of a file containing the origins to load
  • the task type
  • possibly the origin batch size

Positive side effect:

  • Job configuration is committed
  • Logs are ingested in kibana

[1] https://docs.softwareheritage.org/sysadm/deployment/howto-bulk-ingest.html

Tasks:

  • Update the chart
  • Update the documentation
Edited by Vincent Sellier