sentry is regularly failing to post new issues

root@riverside:/var/lib/sentry-onpremise# docker compose logs -f | grep ERROR | head
sentry-self-hosted-sentry-cleanup-1                                | 00:00:20 [ERROR] sentry_sdk.errors: Unexpected status code: 503 (body: b'{"detail":"failed to queue envelope","causes":["envelope buffer capacity exceeded"]}')
sentry-self-hosted-relay-1                                         | 2025-02-16T12:24:01.585450Z ERROR relay_server::endpoints::common: error handling request error=failed to queue envelope error.sources=[envelope buffer capacity exceeded]
sentry-self-hosted-relay-1                                         | 2025-02-16T12:24:01.587658Z ERROR relay_server::utils::managed_envelope: dropped envelope: internal error tags.project_key="a43d441a6b88449ab408557fcc16be32" tags.has_attachments=false tags.has_sessions=false tags.has_profiles=false tags.has_transactions=false tags.has_replays=false tags.has_checkins=false tags.event_category=None cached_summary=EnvelopeSummary { event_category: None, attachment_quantity: 0, session_quantity: 0, profile_quantity: 0, replay_quantity: 0, checkin_quantity: 0, secondary_transaction_quantity: 0, has_plain_attachments: false, event_metrics_extracted: false, payload_size: 366 } recomputed_summary=EnvelopeSummary { event_category: None, attachment_quantity: 0, session_quantity: 0, profile_quantity: 0, replay_quantity: 0, checkin_quantity: 0, secondary_transaction_quantity: 0, has_plain_attachments: false, event_metrics_extracted: false, payload_size: 0 }
sentry-self-hosted-relay-1                                         | 2025-02-16T12:24:01.587687Z ERROR relay_server::endpoints::common: error handling request error=failed to queue envelope error.sources=[envelope buffer capacity exceeded]
sentry-self-hosted-relay-1                                         | 2025-02-16T12:24:01.852490Z ERROR relay_server::utils::managed_envelope: dropped envelope: internal error tags.project_key="41a0821993d0473bbb1cbfca21389e17" tags.has_attachments=false tags.has_sessions=false tags.has_profiles=false tags.has_transactions=false tags.has_replays=false tags.has_checkins=false tags.event_category=None cached_summary=EnvelopeSummary { event_category: None, attachment_quantity: 0, session_quantity: 0, profile_quantity: 0, replay_quantity: 0, checkin_quantity: 0, secondary_transaction_quantity: 0, has_plain_attachments: false, event_metrics_extracted: false, payload_size: 373 } recomputed_summary=EnvelopeSummary { event_category: None, attachment_quantity: 0, session_quantity: 0, profile_quantity: 0, replay_quantity: 0, checkin_quantity: 0, secondary_transaction_quantity: 0, has_plain_attachments: false, event_metrics_extracted: false, payload_size: 0 }
sentry-self-hosted-relay-1                                         | 2025-02-16T12:24:01.852564Z ERROR relay_server::endpoints::common: error handling request error=failed to queue envelope error.sources=[envelope buffer capacity exceeded]
sentry-self-hosted-relay-1                                         | 2025-02-16T12:24:01.918255Z ERROR relay_server::utils::managed_envelope: dropped envelope: internal error tags.project_key="55ca41cc92b04ee5be454f31c5af5699" tags.has_attachments=false tags.has_sessions=false tags.has_profiles=false tags.has_transactions=false tags.has_replays=false tags.has_checkins=false tags.event_category=None cached_summary=EnvelopeSummary { event_category: None, attachment_quantity: 0, session_quantity: 0, profile_quantity: 0, replay_quantity: 0, checkin_quantity: 0, secondary_transaction_quantity: 0, has_plain_attachments: false, event_metrics_extracted: false, payload_size: 308 } recomputed_summary=EnvelopeSummary { event_category: None, attachment_quantity: 0, session_quantity: 0, profile_quantity: 0, replay_quantity: 0, checkin_quantity: 0, secondary_transaction_quantity: 0, has_plain_attachments: false, event_metrics_extracted: false, payload_size: 0 }
sentry-self-hosted-relay-1                                         | 2025-02-16T12:24:01.918325Z ERROR relay_server::endpoints::common: error handling request error=failed to queue envelope error.sources=[envelope buffer capacity exceeded]
sentry-self-hosted-relay-1                                         | 2025-02-16T12:24:01.942657Z ERROR relay_server::utils::managed_envelope: dropped envelope: internal error tags.project_key="41a0821993d0473bbb1cbfca21389e17" tags.has_attachments=false tags.has_sessions=false tags.has_profiles=false tags.has_transactions=false tags.has_replays=false tags.has_checkins=false tags.event_category=None cached_summary=EnvelopeSummary { event_category: None, attachment_quantity: 0, session_quantity: 0, profile_quantity: 0, replay_quantity: 0, checkin_quantity: 0, secondary_transaction_quantity: 0, has_plain_attachments: false, event_metrics_extracted: false, payload_size: 318 } recomputed_summary=EnvelopeSummary { event_category: None, attachment_quantity: 0, session_quantity: 0, profile_quantity: 0, replay_quantity: 0, checkin_quantity: 0, secondary_transaction_quantity: 0, has_plain_attachments: false, event_metrics_extracted: false, payload_size: 0 }
sentry-self-hosted-relay-1                                         | 2025-02-16T12:24:01.942708Z ERROR relay_server::endpoints::common: error handling request error=failed to queue envelope error.sources=[envelope buffer capacity exceeded]```

Some related issues on GitHub:

• https://github.com/getsentry/self-hosted/issues/1929
• https://github.com/getsentry/self-hosted/issues/2836

Following through the 2nd link [1], disabling the default option 'proxy_buffering off' and stopping and starting up sentry [2]. The errors seems to have subsided. [3]

TIL about the organization stats [4] which confirms it too (there are new errors since the last hour).

[1] https://github.com/getsentry/self-hosted/issues/2836

[2]

root@riverside:/var/lib/sentry-onpremise# grep -B1 proxy_buffering nginx/nginx.conf
        # https://github.com/getsentry/self-hosted/issues/2836
        # proxy_buffering off;

root@riverside:/var/lib/sentry-onpremise# docker compose stop && docker compose up -d

[3]

sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:57 +0000] "POST /api/4/envelope/ HTTP/1.1" 200 2 "-" "sentry.python/2.20.0" "192.168.100.109"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:57 +0000] "POST /api/9/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.20.0" "192.168.100.132"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:57 +0000] "POST /api/9/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.20.0" "192.168.100.132"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:57 +0000] "POST /api/3/envelope/ HTTP/1.1" 200 2 "-" "sentry.python/2.20.0" "192.168.130.141"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:57 +0000] "POST /api/9/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.20.0" "192.168.100.132"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:57 +0000] "POST /api/3/envelope/ HTTP/1.1" 200 2 "-" "sentry.python/2.20.0" "192.168.130.141"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:57 +0000] "POST /api/9/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.20.0" "192.168.100.132"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:57 +0000] "POST /api/9/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.20.0" "192.168.100.132"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:57 +0000] "POST /api/3/envelope/ HTTP/1.1" 200 2 "-" "sentry.python/2.20.0" "192.168.100.132"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:57 +0000] "POST /api/9/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.20.0" "192.168.100.132"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:58 +0000] "POST /api/9/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.20.0" "192.168.100.132"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:58 +0000] "POST /api/4/envelope/ HTTP/1.1" 200 2 "-" "sentry.python/2.20.0" "192.168.100.131"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:58 +0000] "POST /api/9/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.20.0" "192.168.100.132"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:58 +0000] "POST /api/9/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.20.0" "192.168.100.132"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:58 +0000] "POST /api/25/envelope/ HTTP/1.1" 200 2 "-" "sentry.python/2.20.0" "192.168.130.151"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:58 +0000] "POST /api/9/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.20.0" "192.168.100.132"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:58 +0000] "POST /api/4/envelope/ HTTP/1.1" 200 2 "-" "sentry.python/2.20.0" "192.168.130.151"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:58 +0000] "POST /api/3/envelope/ HTTP/1.1" 200 2 "-" "sentry.python/2.16.0" "192.168.130.151"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:58 +0000] "POST /api/9/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.20.0" "192.168.100.132"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:58 +0000] "POST /api/9/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.20.0" "192.168.100.132"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:58 +0000] "POST /api/3/envelope/ HTTP/1.1" 200 2 "-" "sentry.python/2.20.0" "192.168.100.132"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:58 +0000] "POST /api/9/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.20.0" "192.168.100.132"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:58 +0000] "POST /api/9/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.20.0" "192.168.100.132"
sentry-self-hosted-nginx-1                                         | 192.168.50.20 - - [17/Feb/2025:11:45:58 +0000] "POST /api/9/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/2.20.0" "192.168.100.132"

[4] https://sentry.softwareheritage.org/organizations/swh/stats/?statsPeriod=1h

The error message makes me think it's more an internal queue overloading (mostly related to https://github.com/getsentry/self-hosted/issues/1929#issuecomment-1404785112) due to a slow down or a peak after the migration of the VM to chaillot

The restart date match the beginning of the errors:

(from https://sentry.softwareheritage.org/organizations/swh/stats/?project=9&statsPeriod=14d)

The issue was probably solved because with the docker compose down/up, the full stack was restarted.

Yes, it makes more sense. Maybe if that happens again, we could increase the queue size as suggested in the referenced link too.

yep, I checked https://italux.github.io/sentry-prometheus-exporter/ to monitor some sentry metrics.

I will create a dedicated issue to test it.

#5573 (closed)

marked this issue as related to #5530 (closed)

marked this issue as related to #5573 (closed)

closed

reopened

changed title from sentry is failing to post new issues to sentry is regularly failing to post new issues

mentioned in commit swh/infra/puppet/puppet-swh-site@d8d55815

mentioned in commit swh/infra/puppet/puppet-swh-site@d8d55815

Reopening this as this is a recurrent issue. When it happens, restarting the relay container seems to unstick things.

I've upgraded sentry to the latest onpremise version (we were only one year late...), which has support for statsd metrics. And we now have /lots/ of metrics:

$ curl http://riverside.internal.admin.swh.network:9102/metrics 2>/dev/null| grep ^sentry_ | wc -l
5123

https://docs.sentry.io/product/relay/monitoring/collected-metrics/ (get mapped to prometheus sentry_relay_* metrics)