Investigate end-to-end monitoring which no longer reports issues

I no longer see any icinga alerts about those.

I noticed an issue on latest deposit which got not properly reloaded/restarted after latest deployment (my bad probably). In any case, it should have been brought up by those icinga notifications (timeout on the deposit end-to-end or some such).

Checking the icinga node (pergamon), the following logs can be seen:

ardumont@pergamon:/var/log/icinga2% tail -f icinga2.log icinga2.log.1 error.log error.log.1 | grep -i deposit
[2020-10-21 21:28:54 +0000] information/Checkable: Checkable 'deposit.internal.staging.swh.network!apt' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-10-21 21:28:54 +0000] information/Notification: Sending 'Problem' notification 'deposit.internal.staging.swh.network!apt!irc-notify-all-services' for user 'root'
[2020-10-21 21:28:54 +0000] information/Notification: Completed sending 'Problem' notification 'deposit.internal.staging.swh.network!apt!irc-notify-all-services' for checkable 'deposit.internal.staging.swh.network!apt' and user 'root' using command 'irc-service-notification'.
[2020-10-21 22:58:01 +0000] information/Notification: Sending reminder 'Problem' notification 'pergamon.softwareheritage.org!Check deposit end-to-end!irc-notify-all-services' for user 'root'
[2020-10-21 22:58:01 +0000] information/Notification: Completed sending 'Problem' notification 'pergamon.softwareheritage.org!Check deposit end-to-end!irc-notify-all-services' for checkable 'pergamon.softwareheritage.org!Check deposit end-to-end' and user 'root' using command 'irc-service-notification'.

---

Migrated from T2727 (view on Phabricator)