Skip to content

Deploy RPM lister and loader on staging

The implementation of the RPM lister and loader is done so we can deploy and test it on the staging infra.

The remaining point to handle before deploying to production is about which origin URL to use but this is not a blocker for staging deployment.

Below are the commands to register listing of RPM source packages according to the distribution:

  • CentOS:
$ swh scheduler task add list-rpm-incremental url=https://www.centos.org instance=CentOS rpm_src_data='[{"base_url": "https://vault.centos.org/", "releases": ["3.7", "3.8", "3.9", "4.0", "4.1", "4.2", "4.3", "4.4", "4.5", "4.6", "4.7", "4.8", "4.9", "5.0", "5.1", "5.2", "5.3", "5.4", "5.5", "5.6", "5.7", "5.8", "5.9", "5.10", "5.11", "6.0", "6.1", "6.2", "6.3", "6.4", "6.5", "6.6", "6.7", "6.8", "6.9", "6.10", "7.0.1406", "7.1.1503", "7.2.1511", "7.3.1611", "7.4.1708", "7.5.1804", "7.6.1810", "7.7.1908", "7.8.2003", "7.9.2009", "8-stream", "8.0.1905", "8.1.1911", "8.2.2004", "8.3.2011", "8.4.2105", "8.5.2111"], "components": ["AppStream", "BaseOS", "HighAvailability", "PowerTools", "SCL", "addons", "centosplus", "contrib", "cr", "csgfs", "dotnet", "extras", "fasttrack", "opstools", "os", "rt", "testing", "updates", "xen4"], "index_url_templates": ["$base_url/$release/$component/Source/", "$base_url/$release/$component/SRPMS/", "$base_url/$release/$component/x86_64/"]}, {"base_url": "https://mirror.stream.centos.org", "releases": ["9-stream"], "components": ["AppStream", "BaseOS", "CRB", "HighAvailability", "NFV", "RT", "ResilientStorage"], "index_url_templates": ["$base_url/$release/$component/source/tree/"]}]'
  • Fedora:
$ swh scheduler task add list-rpm-incremental url=https://fedoraproject.org instance=Fedora rpm_src_data='[{"base_url": "https://archives.fedoraproject.org/pub/archive/fedora/linux/", "releases": ["2", "3", "4", "5", "6"], "components": ["core", "extras"], "index_url_templates": ["$base_url/$component/$release/SRPMS", "$base_url/$component/$release/source/SRPMS", "$base_url/$component/$release/x86_64/os/"]}, {"base_url": "https://archives.fedoraproject.org/pub/archive/fedora/linux/", "releases": ["7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "30", "31", "32", "33", "34", "35"], "components": ["Everything", "Server", "Workstation", "Modular", "Fedora"], "index_url_templates": ["$base_url/releases/$release/$component/source/tree/", "$base_url/updates/$release/$component/source/tree/", "$base_url/releases/$release/$component/source/SRPMS/", "$base_url/updates/$release/SRPMS/"]}, {"base_url": "https://dl.fedoraproject.org/pub/fedora/linux/", "releases": ["36", "37", "38"], "components": ["Everything", "Server", "Workstation", "Modular", "Fedora"], "index_url_templates": ["$base_url/releases/$release/$component/source/tree/", "$base_url/updates/$release/$component/source/tree/"]}]'
  • openSUSE:
$ swh scheduler task add list-rpm-incremental url=http://opensuse.org instance=openSUSE rpm_src_data='[{"base_url": "http://download.opensuse.org/source/", "releases": ["tumbleweed", "jump/15.2", "leap/15.0-Current", "leap/15.0", "leap/15.1", "leap/15.2", "leap/15.3", "leap/15.4", "leap/15.5", "leap/42.2", "leap/42.3-Current", "leap/42.3"], "components": ["oss", "non-oss"], "index_url_templates": ["$base_url/distribution/$release/repo/$component/", "$base_url/distribution/$release/repo/$component/suse/", "$base_url/$release/repo/$component/"]}]'
  • OracleLinux:
$ swh scheduler task add list-rpm-incremental url=https://www.oracle.com/linux instance=OracleLinux rpm_src_data='[{"base_url": "https://yum.oracle.com/repo/EnterpriseLinux/", "releases": ["EL5"], "components": ["addons", "oracle_addons", "unsupported", "0/base", "1/base", "2/base", "3/base", "4/base", "5/base"], "index_url_templates": ["$base_url/$release/$component/x86_64"]}, {"base_url": "https://yum.oracle.com/repo/OracleLinux/", "releases": ["OL5", "OL6", "OL7", "OL8", "OL9"], "components": ["0/base", "0/baseos/base", "1/base", "1/baseos/base", "10/base", "11/base", "2/base", "2/baseos/base", "3/base", "3/baseos/base", "4/base", "4/baseos/base", "4/security/validation", "5/base", "5/baseos/base", "6/base", "6/baseos/base", "7/base", "7/baseos/base", "8/base", "8/baseos/base", "8/security/validation", "9/base", "MODRHCK", "MySQL", "MySQL56", "MySQL57_community", "MySQL80/community", "MySQL80/connectors/community", "MySQL80/tools/community", "MySQL80_community", "RDMA", "SoftwareCollections", "UEK/latest", "UEKR3", "UEKR3/latest", "UEKR3_OFED20", "UEKR4", "UEKR4/OFED", "UEKR4/archive", "UEKR5", "UEKR5/RDMA", "UEKR5/archive", "UEKR6", "UEKR6/RDMA", "UEKR7", "UEKR7/RDMA", "addons", "appstream", "appstream/developer", "automation2", "baseos/developer", "baseos/latest", "beta", "ceph", "ceph30", "codeready/builder", "codeready/builder/developer", "developer", "developer/EPEL", "developer/EPEL/modular", "developer/UEKR5", "developer/UEKR6", "developer/UEKR7", "developer/golang117", "developer/golang118", "developer/golang119", "developer/kvm/utils", "developer/nodejs12", "developer/olcne", "developer/php74", "developer_EPEL", "developer_gluster310", "developer_gluster312", "distro/builder", "gluster/appstream", "gluster312", "gluster41", "gluster5", "gluster6", "gluster8", "kvm/appstream", "kvm/utils", "latest", "latest/archive", "leapp", "ofed_UEK", "olcne", "olcne11", "olcne12", "olcne13", "olcne14", "olcne15", "olcne16", "openstack10", "openstack21", "openstack30", "openstack40", "openstack40_extras", "openstack50", "openstack50_extras", "optional", "optional/archive", "optional/beta", "oracle/instantclient", "oracle/instantclient21", "oraclelinuxmanager210/client", "oraclelinuxmanager210/server", "ovirt42", "ovirt42/extras", "ovirt43", "ovirt43/extras", "ovirt44", "ovirt44/extras", "security/validation", "spacewalk210/client", "spacewalk210/server", "spacewalk24/client", "spacewalk24/server", "spacewalk26/client", "spacewalk26/server", "spacewalk27/client", "spacewalk27/server"], "index_url_templates": ["$base_url/$release/$component/x86_64"]}]'
  • RockyLinux:
$ swh scheduler task add list-rpm-incremental url=https://rockylinux.org instance=RockyLinux rpm_src_data='[{"base_url": "https://download.rockylinux.org/", "releases": ["8.3", "8.4", "8.4-RC1", "8.5", "8.6", "8.7", "8.8", "9.0", "9.1", "9.2"], "components": ["AppStream", "BaseOS", "Devel", "HighAvailability", "Minimal", "PowerTools", "ResilientStorage", "CRB", "NFV", "RT", "SAP", "SAPHANA", "devel", "extras", "plus", "nfv", "rockyrpi"], "index_url_templates": ["$base_url/vault/rocky/$release/$component/source/tree/", "$base_url/pub/rocky/$release/$component/source/tree/"]}]'

plan:

  • Packaging (docker image) - already built
    • swh.loader.core v5.10
    • swh.lister v5.9.8
  • swh/infra/ci-cd/swh-charts!127 (merged): swh-charts: Reference new lister and loader
  • Deploy ^
  • swh-apps!25 (merged): Create toolbox (no longer depend on debian package)
  • swh-toolbox (instead of scheduler node) : Reference new task types
  • swh-toolbox (instead of scheduler node): Reference instance to list (as per the cli above, @anlambert thx ;)
  • scheduler node (scheduler0): Upgrade package & Restart swh-scheduler-schedule-recurrent.service (so listing can be scheduled)
  • Trigger listing
  • Analyse issue with the listing
  • swh/devel/swh-lister!500 (merged): Fix lister so listing can happen
  • swh/infra/ci-cd/swh-charts!129 (merged): Release fix, deploy again
  • Trigger listing
  • Unstuck debian package for python3-swh.loader.core
  • scheduler node: Upgrade package & restart swh-scheduler-scheduler-recurrent.service (so loading tasks output of the lister can be scheduled too)
  • Checks
    • listing
    • loading
Edited by Antoine R. Dumont