Clean sentry reports
We currently have a lot of open Sentry issues, but this is very raw data that isn't very usable or visible.
We should either:
- explicitly ignore them / stop raising them
- assign them a task in the forge (manually! otherwise it would just be more noise)
- fix them and/or close them
comments:
- it is a process
- better managing the elements that are in Sentry
Migrated from T3957 (view on Phabricator)
- Show closed items
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- David Douard added Roadmap 2022 meta-task priority:Normal labels
added Roadmap 2022 meta-task priority:Normal labels
- Maintainer
Sentry Sprint #1 report :
Planning
- April 13th
- 10:00 - 12:00
- Kickoff
- Definition of the sprint goals
- Identification of the scoped sentry issues
- Team organization
- 14:00 - 17:00
- First iteration
- 10:00 - 12:00
- April 14th
- 10:00 - 10:30
- Synchronization
- 10:30 - 12:00
- Second iteration...
- 14:00 - 16:30
- ...second iteration
- 16:30 - 17:00
- Retrospective
- 10:00 - 10:30
Sprint Goals
- We currently have a lot of open Sentry issues, but this is very raw data that isn’t very usable or visible. They should be cleaned up so that under normal conditions, the number of reported issues stays “minimal”.
- Clean-up options:
- Refine issues data
- Eliminate redundant/useless issues
- Identify sources of issues >> potential easy hacks ?
Tasks
Task Pair Diffs turn off sentry error reporting logging integration douardda / bchauvet swh/devel/swh-core!266 Use explicit sentry exception capture in RPC back-end anlambert / olasd swh/devel/swh-core!263 swh/devel/swh-core!264 RetryError Handling anlambert / olasd swh/devel/swh-storage!783 UniqueViolation and HashCollision server-side only vsellier / vlorentz Vault ardumont/jayesh swh/devel/swh-core!263 swh/devel/swh-vault!146 Define statsd metrics for sentry-removed issues anlambert/olasd swh/devel/swh-core!265 Add missing sentry captures team #3957 Pairs
vsellier - vlorentz olasd - anlambert ardumont - jayesh douardda - bchauvet
Tasks details
-
Review the places in the code where we log exceptions to check there's an excplicit sentry capture
-
Decide to turn off the sentry error reporting in the logging integration https://docs.sentry.io/platforms/python/guides/logging/
-
Identify and Remove expected issues:
- UniqueViolation (swh-storage) backend + client-side RetryError : ignore the backend issue, keep the client as the actual underlying error
- Throttled (web app) - easy hack -> https://forge.softwareheritage.org/source/swh-web/browse/master/swh/web/common/exc.py$132-142
- Not Found (silented) - easy hack -> https://forge.softwareheritage.org/source/swh-loader-core/browse/master/swh/loader/core/loader.py$349-363
- QueryCanceled: drop the server-side error, keep the client one only
- OSError
-
Define statsd metrics for sentry-removed issues
- swh/devel/swh-core!265 to add an error_type tag to already existing (backend function) error metrics
-
UniqueViolation and HashCollision: not to be reported in sentry if the error can be encoded and sent to the client >> create a task for it
-
Loaders
- ValueError: Convert to NotFound exception
- OSError: to be removed
- MaybeEncodingError: to be fixed (celery)
- RetryError: report the underlying error instead (when the last try fails)
- ConnectionError: base package loader - list of known exceptions that should be logged instead of being raised to sentry
-
Storage
- HashCollision
- 1CK
- 1BW
- -> many are false positives that are hard to resolve: https://forge.softwareheritage.org/swh/devel/swh-storage!288
- UniqueViolation
- HashCollision
-
Web App
- list of errors not sent to Sentry: https://forge.softwareheritage.org/source/swh-web/browse/master/swh/web/common/exc.py$132-142
- ?
- QueryCancelled: not to be logger to sentry storage-side (it's the client to report)
-
Vault
- NotFoundExc should not be reported to sentry https://sentry.softwareheritage.org/organizations/swh/issues/9151/
- April 13th
- Author Maintainer
seems we still have a lot of NotFound error from the swh-loader-git: