end-to-end checks are no longer parametrized properly per environment

For example, here is the deposit end-to-end check expanded for the staging environment. It's hitting the deposit.softwareheritage.org and not the deposit.staging.swh.network.

The settings are properly set. As pergamon (on which this runs) is a production node, puppet is installing production configuration.

It used to be installed through puppet collected resources so that may have been broken at the time of the rewrite without being seen.

[1]

/opt/swh-e2e-checks/bin/swh icinga_plugins \
  --prometheus-exporter \
  --prometheus-exporter-directory \
  /var/lib/prometheus/node-exporter \
  --environment staging \
  --warning 600  --critical 3600 \
  check-deposit \
  --server https://deposit.softwareheritage.org/1 \  <---- should be https://deposit.staging.swh.network/1
  --username $user \
  --password $token \
  --collection swh \
  --provider-url https://www.softwareheritage.org \
  --swh-web-url https://archive.softwareheritage.org \
  --poll-interval 1 \
  single \
  --archive /opt/swh-e2e-checks/lib/python3.11/site-packages/swh/icinga_plugins/data/deposit/jesuisgpl.tgz \
  --metadata /opt/swh-e2e-checks/lib/python3.11/site-packages/swh/icinga_plugins/data/deposit/jesuisgpl.tgz.xml

Note: It's probably the same for the other check end-to-end vault, webapp, etc...

changed the description

changed the description

mentioned in issue #5589 (closed)

changed the description

mentioned in commit swh/infra/puppet/puppet-swh-site@6705164f

mentioned in commit swh/infra/puppet/puppet-swh-site@778ee930

mentioned in merge request swh/infra/puppet/puppet-swh-site!730 (merged)

mentioned in commit swh/infra/puppet/puppet-swh-site@d2f2e5ce

mentioned in commit swh/infra/puppet/puppet-swh-site@3e504898

mentioned in commit swh/infra/puppet/puppet-swh-site@8284522f

mentioned in commit swh/infra/puppet/puppet-swh-site@c90465af

Note: It's probably the same for the other check end-to-end vault, webapp, etc...

Indeed for the webapp too. But the vault is fine because it already did not depend on the 'lookup' call.

---

The fixes are at the mr [1]

[1] swh/infra/puppet/puppet-swh-site!730 (merged)

Deployed:

root@pergamon:~# puppet agent -t
Info: Using environment 'production'                                                                                                                                                                                Info: Retrieving pluginfacts                                                                                                                                                                                        Info: Retrieving plugin                                                                                                                                                                                             Info: Loading facts
Error: Facter: Error while resolving custom fact fact='puppetdb_version', resolution='<anonymous>': undefined method `strip' for nil:NilClass

    output.split(':').last.strip                                                                                                                                                                                                              ^^^^^^
Info: Caching catalog for pergamon.softwareheritage.org
Info: Applying configuration version '1741791996'
Notice: /Stage[main]/Profile::Icinga2::Objects::E2e_checks_base/Exec[pip install e2e checks]/returns: executed successfully (corrective)
Notice: /Stage[main]/Profile::Icinga2::Objects::Static_checks/Profile::Swh::Deploy::Webapp::Icinga_checks[archive.softwareheritage.org]/Profile::Icinga2::Objects::E2e_checks_savecodenow[End-to-end SaveCodeNow Check - git with type git in production]/Icinga2::Object::Checkcommand[production-check-savecodenow-cmd-git-git]/Icinga2::Object[icinga2::object::CheckCommand::production-check-savecodenow-cmd-git-git]/Concat[/etc/icinga2/conf.d/e2e-checks.conf]/File[/etc/icinga2/conf.d/e2e-checks.conf]/content:
--- /etc/icinga2/conf.d/e2e-checks.conf 2025-01-14 17:30:57.550554416 +0000                                                                                                                                         +++ /tmp/puppet-file20250312-1600790-xk9ao7     2025-03-12 15:08:06.112838006 +0000                                                                                                                                 @@ -38,28 +38,28 @@
 object CheckCommand "staging-check-deposit-cmd" {                                                                                                                                                                     import "plugin-check-command"                                                                                                                                                                                                                                                                                                                                                                                                        -  command = [ "/opt/swh-e2e-checks/bin/swh", "icinga_plugins", "--prometheus-exporter", "--prometheus-exporter-directory", "/var/lib/prometheus/node-exporter", "--environment", "staging", "--warning", 600, "--critical", 3600, "check-deposit", "--server", "https://deposit.softwareheritage.org/1", "--username", "swh", "--password", "girdle-tumble-spiral-valid-robin", "--collection", "swh", "--provider-url", "https://www.softwareheritage.org", "--swh-web-url", "https://archive.softwareheritage.org", "--poll-interval", 1, "single", "--archive", "/opt/swh-e2e-checks/lib/python3.11/site-packages/swh/icinga_plugins/data/deposit/jesuisgpl.tgz", "--metadata", "/opt/swh-e2e-checks/lib/python3.11/site-packages/swh/icinga_plugins/data/deposit/jesuisgpl.tgz.xml", ]
+  command = [ "/opt/swh-e2e-checks/bin/swh", "icinga_plugins", "--prometheus-exporter", "--prometheus-exporter-directory", "/var/lib/prometheus/node-exporter", "--environment", "staging", "--warning", 600, "--critical", 3600, "check-deposit", "--server", "https://deposit.staging.swh.network/1", "--username", "swh", "--password", "girdle-tumble-spiral-valid-robin", "--collection", "swh", "--provider-url", "https://www.softwareheritage.org", "--swh-web-url", "https://archive.softwareheritage.org", "--poll-interval", 1, "single", "--archive", "/opt/swh-e2e-checks/lib/python3.11/site-packages/swh/icinga_plugins/data/deposit/jesuisgpl.tgz", "--metadata", "/opt/swh-e2e-checks/lib/python3.11/site-packages/swh/icinga_plugins/data/deposit/jesuisgpl.tgz.xml", ]
   timeout = 4800
 }

 object CheckCommand "staging-check-savecodenow-cmd-git-git" {
   import "plugin-check-command"

-  command = [ "/opt/swh-e2e-checks/bin/swh", "icinga_plugins", "--prometheus-exporter", "--prometheus-exporter-directory", "/var/lib/prometheus/node-exporter", "--environment", "staging", "--warning", 300, "--critical", 600, "check-savecodenow", "--swh-web-url", "https://archive.softwareheritage.org", "origin", "https://github.com/rdicosmo/parmap", "https://github.com/SoftwareHeritage/swh-core", "https://github.com/SoftwareHeritage/swh-lister", "https://github.com/SoftwareHeritage/swh-loader-bzr", "https://github.com/SoftwareHeritage/swh-loader-core", "https://github.com/SoftwareHeritage/swh-loader-git", "https://github.com/SoftwareHeritage/swh-loader-mercurial", "https://github.com/SoftwareHeritage/swh-loader-svn", "https://github.com/SoftwareHeritage/swh-model", "https://github.com/SoftwareHeritage/swh-web", "--visit-type", "git", ]
+  command = [ "/opt/swh-e2e-checks/bin/swh", "icinga_plugins", "--prometheus-exporter", "--prometheus-exporter-directory", "/var/lib/prometheus/node-exporter", "--environment", "staging", "--warning", 300, "--critical", 600, "check-savecodenow", "--swh-web-url", "https://webapp.staging.swh.network", "origin", "https://github.com/rdicosmo/parmap", "https://github.com/SoftwareHeritage/swh-core", "https://github.com/SoftwareHeritage/swh-lister", "https://github.com/SoftwareHeritage/swh-loader-bzr", "https://github.com/SoftwareHeritage/swh-loader-core", "https://github.com/SoftwareHeritage/swh-loader-git", "https://github.com/SoftwareHeritage/swh-loader-mercurial", "https://github.com/SoftwareHeritage/swh-loader-svn", "https://github.com/SoftwareHeritage/swh-model", "https://github.com/SoftwareHeritage/swh-web", "--visit-type", "git", ]
   timeout = 900
 }

 object CheckCommand "staging-check-savecodenow-cmd-mercurial-hg" {
   import "plugin-check-command"

-  command = [ "/opt/swh-e2e-checks/bin/swh", "icinga_plugins", "--prometheus-exporter", "--prometheus-exporter-directory", "/var/lib/prometheus/node-exporter", "--environment", "staging", "--warning", 300, "--critical", 600, "check-savecodenow", "--swh-web-url", "https://archive.softwareheritage.org", "origin", "https://foss.heptapod.net/mercurial/hgview", "https://foss.heptapod.net/mercurial/tortoisehg", "https://foss.heptapod.net/mercurial/evolve", "https://foss.heptapod.net/mercurial/hg-git", "https://foss.heptapod.net/mercurial/hg-docgraph", "https://foss.heptapod.net/mercurial/mercurial-devel", "https://foss.heptapod.net/mercurial/pytest", "https://foss.heptapod.net/mercurial/scmperf", "--visit-type", "hg", ]
+  command = [ "/opt/swh-e2e-checks/bin/swh", "icinga_plugins", "--prometheus-exporter", "--prometheus-exporter-directory", "/var/lib/prometheus/node-exporter", "--environment", "staging", "--warning", 300, "--critical", 600, "check-savecodenow", "--swh-web-url", "https://webapp.staging.swh.network", "origin", "https://foss.heptapod.net/mercurial/hgview", "https://foss.heptapod.net/mercurial/tortoisehg", "https://foss.heptapod.net/mercurial/evolve", "https://foss.heptapod.net/mercurial/hg-git", "https://foss.heptapod.net/mercurial/hg-docgraph", "https://foss.heptapod.net/mercurial/mercurial-devel", "https://foss.heptapod.net/mercurial/pytest", "https://foss.heptapod.net/mercurial/scmperf", "--visit-type", "hg", ]
   timeout = 900
 }

 object CheckCommand "staging-check-savecodenow-cmd-subversion-svn" {
   import "plugin-check-command"

-  command = [ "/opt/swh-e2e-checks/bin/swh", "icinga_plugins", "--prometheus-exporter", "--prometheus-exporter-directory", "/var/lib/prometheus/node-exporter", "--environment", "staging", "--warning", 300, "--critical", 600, "check-savecodenow", "--swh-web-url", "https://archive.softwareheritage.org", "origin", "https://subversion.renater.fr/anonscm/svn/panda", "https://svn.code.sf.net/p/freeorion/code", "https://svn.code.sf.net/p/swig/code", "https://svn.code.sf.net/p/zookeeper/code", "https://svn.code.sf.net/p/esitools/code", "https://svn.code.sf.net/p/sauerbraten/code/", "https://svn.code.sf.net/p/civ4mods/code", "https://svn.code.sf.net/p/ldap-sdk/code", "--visit-type", "svn", ]
+  command = [ "/opt/swh-e2e-checks/bin/swh", "icinga_plugins", "--prometheus-exporter", "--prometheus-exporter-directory", "/var/lib/prometheus/node-exporter", "--environment", "staging", "--warning", 300, "--critical", 600, "check-savecodenow", "--swh-web-url", "https://webapp.staging.swh.network", "origin", "https://subversion.renater.fr/anonscm/svn/panda", "https://svn.code.sf.net/p/freeorion/code", "https://svn.code.sf.net/p/swig/code", "https://svn.code.sf.net/p/zookeeper/code", "https://svn.code.sf.net/p/esitools/code", "https://svn.code.sf.net/p/sauerbraten/code/", "https://svn.code.sf.net/p/civ4mods/code", "https://svn.code.sf.net/p/ldap-sdk/code", "--visit-type", "svn", ]
   timeout = 900
 }


Info: Computing checksum on file /etc/icinga2/conf.d/e2e-checks.conf
Info: /Stage[main]/Profile::Icinga2::Objects::Static_checks/Profile::Swh::Deploy::Webapp::Icinga_checks[archive.softwareheritage.org]/Profile::Icinga2::Objects::E2e_checks_savecodenow[End-to-end SaveCodeNow Check - git with type git in production]/Icinga2::Object::Checkcommand[production-check-savecodenow-cmd-git-git]/Icinga2::Object[icinga2::object::CheckCommand::production-check-savecodenow-cmd-git-git]/Concat[/etc/icinga2/conf.d/e2e-checks.conf]/File[/etc/icinga2/conf.d/e2e-checks.conf]: Filebucketed /etc/icinga2/conf.d/e2e-checks.conf to puppet with sum 2a6fec5f896c0735c0b26ebb0039a52e2abd901f167ab42ac1de9f2dceb3349b
Notice: /Stage[main]/Profile::Icinga2::Objects::Static_checks/Profile::Swh::Deploy::Webapp::Icinga_checks[archive.softwareheritage.org]/Profile::Icinga2::Objects::E2e_checks_savecodenow[End-to-end SaveCodeNow Check - git with type git in production]/Icinga2::Object::Checkcommand[production-check-savecodenow-cmd-git-git]/Icinga2::Object[icinga2::object::CheckCommand::production-check-savecodenow-cmd-git-git]/Concat[/etc/icinga2/conf.d/e2e-checks.conf]/File[/etc/icinga2/conf.d/e2e-checks.conf]/content: content changed '{sha256}2a6fec5f896c0735c0b26ebb0039a52e2abd901f167ab42ac1de9f2dceb3349b' to '{sha256}5a683990ce466bc301b5f9d745715b9a4e60cd6c9307c87100d30b2f4122cc72'
Info: Concat[/etc/icinga2/conf.d/e2e-checks.conf]: Scheduling refresh of Class[Icinga2::Service]
Info: Class[Icinga2::Service]: Scheduling refresh of Service[icinga2]
Notice: /Stage[main]/Icinga2::Service/Service[icinga2]: Triggered 'refresh' from 1 event
Notice: Applied catalog in 61.03 seconds

closed