Stabilize the provenance service in production

After a clarification meeting, here are the description / observations about current status of the service:

swh-provenance rpc server:

The service is based on 2 endpoints (doc : https://gitlab.softwareheritage.org/product-management/core-platform/-/work_items/31#note_165622)

• whereis : 1 content -> 1 qualified swhid (or none)
• whereare : n contents -> n qualified swhids, one or none per each content

scanner

from a directory on local

• recursively found the existing objects in the archive (content, directory) via the api/1/known api endpoint
• lot of small queries
• use the remaining quota to rate-limit the queries

If the scanner is started with the --provenance option

• Send a batch of 1000 swhids per requests to the provenance api
• Launch several requests in parallel
  • 5 initially -> timeouts
  • 3 now works better but still timeout from time to time

Several action were identified, some out of scope of this issue centered on the production environment:

• [out-of-scope] Add metrics on swh-graph
• [out-of-scope] Add logs on the scanner to identify problematic batchs
• [out-of-scope] Add debug logs on swh-provenance to trace the durations of the queries to swh-graph with the swhid / global duration (swh/devel/swh-provenance!179 (merged))
• Deploy the last version of swh-provenance (the one with swh/devel/swh-provenance!178 (merged) and swh/devel/swh-provenance!179 (merged))
• Increase the swh-provenance gunicorn worker timeouts to 30s (timeout is already 60s)
• To avoid any side effect related to the increase of timeout, limit the maximum number of parallel queries on the /api/1/provenance endpoint to 10.

One point to diagnose is the duration of the swh-provenance requests, it seems the swh-graph queries take globally around 50 pico-seconds, with a couple taking several milli seconds. It's not clear which ones yet.

There is also a couple of issues in sentry that need to be investigate

assigned to @vsellier

changed the description

mentioned in merge request swh/devel/swh-provenance!179 (merged)

mentioned in merge request swh/devel/swh-provenance!178 (merged)

I misread the grafana graphs when I checked the timeouts.

The effective deployed timeout on the swh-provenance gunicorn workers is 60s. regarding that I'm not going to change anything in the configuration.

This is some logs when scanner (from master branch commit b2065ae91a3a75749affa9c8fd9ac053a6efea0c) is launched on swh-counters:

• webapp and provenance (easier to check the timeline)

82.xx.xx.xxx - - [11/Sep/2024:06:55:38 +0000] "POST /api/1/known/ HTTP/1.1" 200 545 "-" "python-requests/2.32.3" 1656 0.130 [swh-web-archive-5004] [] 10.43.34.133:5004 545 0.129 200 d43aa59b26f77c7bcee14e806c5735c1
10.42.204.27 - - [11/Sep/2024:06:55:38 +0000] "POST /whereare HTTP/1.1" 200 2 "-" "python-requests/2.32.3" 535 0.086 [swh-provenance-graph-granet-5014] [] 10.43.5.101:5014 2 0.086 200 2d792d14ba7b76e05d1cbf91f4fb0fee
82.xx.xx.xxx - - [11/Sep/2024:06:55:38 +0000] "POST /api/1/provenance/whereare/ HTTP/1.1" 200 6 "-" "python-requests/2.32.3" 1291 0.176 [swh-web-archive-5004] [] 10.43.34.133:5004 6 0.176 200 8b7e40c664c93ec6b68510dcd63c5f0f
10.42.131.35 - - [11/Sep/2024:06:56:17 +0000] "POST /whereare HTTP/1.1" 200 3130 "-" "python-requests/2.32.3" 1983 38.756 [swh-provenance-graph-granet-5014] [] 10.43.5.101:5014 3130 38.755 200 334f739204e61cd8244216c3189689a9
82.xx.xx.xxx - - [11/Sep/2024:06:56:17 +0000] "POST /api/1/provenance/whereare/ HTTP/1.1" 200 2728 "-" "python-requests/2.32.3" 2319 38.827 [swh-web-archive-5004] [] 10.43.34.133:5004 2728 38.827 200 03fbff07b4373cca2cdde55767553621
82.xx.xx.xxx - - [11/Sep/2024:06:56:17 +0000] "POST /whereare HTTP/1.1" 200 2 "-" "python-requests/2.32.3" 535 0.010 [swh-provenance-graph-granet-5014] [] 10.43.5.101:5014 2 0.010 200 dbaf25cad7fa24d1aa54c3dd13086962
82.xx.xx.xxx - - [11/Sep/2024:06:56:17 +0000] "POST /api/1/provenance/whereare/ HTTP/1.1" 200 6 "-" "python-requests/2.32.3" 1291 0.137 [swh-web-archive-5004] [] 10.43.34.133:5004 6 0.137 200 9df176bbef3390a06fdb016e7eb7822c
82.xx.xx.xxx - - [11/Sep/2024:06:56:17 +0000] "POST /api/1/provenance/whereare/ HTTP/1.1" 200 1284 "-" "python-requests/2.32.3" 1832 0.123 [swh-web-archive-5004] [] 10.43.34.133:5004 1284 0.124 200 448cfe90e3cdb3bec34cd212335b0f12    <--- Where did you go? is there some kind of cache in the webapp or a log line was lost somewhere?
10.42.204.61 - - [11/Sep/2024:06:56:54 +0000] "POST /whereare HTTP/1.1" 200 1655 "-" "python-requests/2.32.3" 1220 36.365 [swh-provenance-graph-granet-5014] [] 10.43.5.101:5014 1655 36.365 200 99bfed70c841e938f685c8ac06675004
82.xx.xx.xxx - - [11/Sep/2024:06:56:54 +0000] "POST /api/1/provenance/whereare/ HTTP/1.1" 200 1439 "-" "python-requests/2.32.3" 1778 36.431 [swh-web-archive-5004] [] 10.43.34.133:5004 1439 36.430 200 c2c2c9a4d450bb1187002f488b681eff

The duration is the field before the 200

• graph

sept. 11 06:55:38 granet swh[486308]: 2024-09-11T06:55:38+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 95.779µs
sept. 11 06:55:38 granet swh[486308]: 2024-09-11T06:55:38+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 53.05µs
sept. 11 06:55:38 granet swh[486308]: 2024-09-11T06:55:38+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 1.350666ms
sept. 11 06:55:38 granet swh[486308]: 2024-09-11T06:55:38+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 44.162µs
sept. 11 06:55:38 granet swh[486308]: 2024-09-11T06:55:38+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 54.074µs
sept. 11 06:55:38 granet swh[486308]: 2024-09-11T06:55:38+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 55.117µs

sept. 11 06:55:56 granet swh[486308]: 2024-09-11T06:55:56+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 70.772µs <--- what is this request ? one from 06:56:17 or totally something else

sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 69.679µs <--- 06:56:17 request
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 41.025µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 43.442µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 53.317µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 48.039µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 55.162µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 41.629µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 50.291µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 49.707µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 54.119µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 39.906µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 48.816µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 38.417µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 53.253µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 38.351µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 49.869µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 30.653µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 37.094µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 47.318µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 52.281µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 32.494µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 36.825µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 51.982µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 30.365µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 36.906µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 50.541µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 36.968µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 48.711µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 36.707µs
sept. 11 06:56:14 granet swh[486308]: 2024-09-11T06:56:14+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 50.797µs
sept. 11 06:56:16 granet swh[486308]: 2024-09-11T06:56:16+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 54.554µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 69.792µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 43.65µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 53.607µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 42.152µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 38.823µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 63.752µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 51.561µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 50.331µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 40.416µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 48.913µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 62.376µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 53.493µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 53.628µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 41.694µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 51.311µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 32.313µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 65.395µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 52.054µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 52.097µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 38.606µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 52.876µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 38.692µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 52.035µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 38.463µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 53.576µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 44.907µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 54.351µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 42.356µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 42.727µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 50.661µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 67.917µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 52.209µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 65.299µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 53.979µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 51.91µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 51.503µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 55.703µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 42.812µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 52.836µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 39.053µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 51.824µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 38.582µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 55.159µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 39.499µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 52.343µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 32.427µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 45.426µs
sept. 11 06:56:17 granet swh[486308]: 2024-09-11T06:56:17+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 35.781µs <--- end of 06:56:17

sept. 11 06:56:36 granet swh[486308]: 2024-09-11T06:56:36+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 62.487µs <--- Same pattern, one request and nothing before 30s

sept. 11 06:56:54 granet swh[486308]: 2024-09-11T06:56:54+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 106.881µs
sept. 11 06:56:54 granet swh[486308]: 2024-09-11T06:56:54+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 44.666µs
sept. 11 06:56:54 granet swh[486308]: 2024-09-11T06:56:54+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 65.576µs
sept. 11 06:56:54 granet swh[486308]: 2024-09-11T06:56:54+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 50.926µs
sept. 11 06:56:54 granet swh[486308]: 2024-09-11T06:56:54+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 52.058µs
sept. 11 06:56:54 granet swh[486308]: 2024-09-11T06:56:54+00:00 - INFO - 200 OK - http://graph.internal.softwareheritage.org:50091/swh.graph.TraversalService/Traverse - 50.571µs

If the swh-graph logs can be trusted, what is seen is there is ~30s between each return of provenance request. These logs are no enough to diagnose where they are spend.

Is it a delay between 2 scanner request? Is it a delay in the webapp ? Is it a delay in swh-provenance ? or something else in the infra ?

The debug logs in provenance will help to diagnose that.

One more information: there is 2 provenance instances running with each one 4 gunicorn workers configured with 1 thread. It theoretically allows to handle 8 parallel requests but in the reality there is no guaranty a new request will not be sent to a busy instance and will be put in the waiting queue.

Is it a delay between 2 scanner requests? Is it a delay in the webapp ? Is it a delay in swh-provenance ? or something else in the infra ?

given the request duration is >30s in swh-provenance it seems the swh-provenance is a good suspect to start the investigations.

the weird pattern of the requests made on the graph is probably something to dig

I will stop the investigation here. the debug logs will tell more about that

changed the description

mentioned in issue #5413 (closed)

Testing using a local instance of swh-provenance (running with swh provenance rpc-serve) using granet's grpc as backend, executing 1 whereare query with 353 swhids, I have the following response time histogram for traversal query times (these are in ms):

(array([324, 218,   6,   5,   1,   0,   2,   0,   0,   2]),
 array([   10,    20,    50,   100,   200,   500,  1000,  2000,  5000, 10000, 20000]))

The same with max_edges=100000 (instead of the default 10000):

(array([ 90, 221, 152,  73,   2,   0,   2,   0,   0,   2]),
 array([   10,    20,    50,   100,   200,   500,  1000,  2000,  5000, 10000, 20000]))

So without too much of a surprise, most queries are ok, the empty file is a perf killer.

Now, with the idea of filtering these edge case swhids (swh/devel/swh-provenance!177 (merged)), here are the overall whereis response time for each of the (with max_edges=100000) in second:

[('swh:1:cnt:e69de29bb2d1d6434b8b29ae775ad8c2e48c5391', 36.285656899999594),
 ('swh:1:dir:4b825dc642cb6eb9a060e54bf8d69288fbee4904', 13.257851957008825),
 ('swh:1:dir:29a422c19251aeaeb907175e9b3219a9bed6c616', 5.861998074993608),
 ('swh:1:cnt:8b137891791fe96927ad78e64b0aad7bded08bdc', 4.983283403009409),
 ('swh:1:cnt:654d0bfe943437d43242325b1fbcff5f400d84ee', 4.611207509005908),
 ('swh:1:dir:d564d0bc3dd917926892c55e3706cc116d5b165e', 4.348841104001622),
 ('swh:1:cnt:4d29575de80483b005c29bfcac5061cd2f45313e', 4.257414676001645),
 ('swh:1:cnt:261eeb9e9f8b2b4b0d119366dda99c6fd7d35c64', 3.724961934000021),
 ('swh:1:cnt:dfe0770424b2a19faf507a501ebfc23be8f54e7b', 3.675598482004716),
 ('swh:1:cnt:e7af2f77107d73046421ef56c4684cbfdd3c1e89', 3.5961616200074786),
 ('swh:1:cnt:f288702d2fa16d3cdf0035b15a9fcbc552cd88e7', 2.7842661239992594),
 ('swh:1:cnt:1ff0c423042b46cb1d617b81efb715defbe8054d', 2.6283129960065708),
 ('swh:1:cnt:94a9ed024d3859793618152ea559a168bbcbb5e2', 2.6105939529952593),
 ('swh:1:cnt:19129e315fe593965a2fdd50ec0d1253bcbd2ece', 2.3507767579867505),
 ('swh:1:cnt:d645695673349e3947e8e5ae42332d0ac3164cd7', 2.0868855049920967),
 ('swh:1:cnt:8dada3edaf50dbc082c9a125058f25def75e625a', 1.7257062680000672),
 ('swh:1:cnt:ee27ba4b4412b0e4a05af5e3d8a005bc6681fdf3', 1.6642030039947713),
 ('swh:1:cnt:9cecc1d4669ee8af2ca727a5d8cde10cd8b2d7cc', 1.4901558950077742),
 ('swh:1:cnt:6b60c1042f58d9fabb75485aa3624dddcf633b5c', 1.3454428410041146),
 ('swh:1:cnt:d159169d1050894d3ea3b98e1c965c4058208fe1', 1.2696577399910893),
 ('swh:1:dir:9d1dcfdaf1a6857c5f83dc27019c7600e1ffaff8', 0.9425193259958178),
 ('swh:1:dir:82e3a754b6a0fcb238b03c0e47d05219fbf9cf89', 0.9353011259954656),
 ('swh:1:cnt:2e65efe2a145dda7ee51d1741299f848e5bf752e', 0.8951035130012315),
 ('swh:1:cnt:d511905c1647a1e311e8b20d5930a37a9c2531cd', 0.7386718549969373),
 ('swh:1:cnt:dba13ed2ddf783ee8118c6a581dbf75305f816a3', 0.6846408260025783),
 ('swh:1:cnt:d60c31a97a544b53039088d14fe9114583c0efc3', 0.5757868950022385),
 ('swh:1:cnt:65c5ca88a67c30becee01c5a8816d964b03862f9', 0.5657865829998627),
 ('swh:1:cnt:f76dd238ade08917e6712764a16a22005a50573d', 0.4654381750005996),
 ('swh:1:cnt:0519ecba6ea913e21689ec692e81e9e4973fbf73', 0.41043540299870074),
 ('swh:1:cnt:56a6051ca2b02b04ef92d5150c9ef600403cb1de', 0.40196315199136734),
 ('swh:1:cnt:25cb955ba23571f6fefd57cecf5f67454210bbc8', 0.3394993770052679),
 ('swh:1:cnt:62f9457511f879886bb7728c986fe10b0ece6bcb', 0.3331791139935376),
 ('swh:1:cnt:6b2aaa7640726588bcd3d57e1de4b1315b7f315e', 0.3201613900018856),
 ('swh:1:cnt:c227083464fb9af8955c90d2924774ee50abb547', 0.27784057399549056),
 ('swh:1:cnt:bf0d87ab1b2b0ec1a11a3973d2845b42413d9767', 0.2775194690038916),
 ('swh:1:cnt:d8263ee9860594d2806b0dfd1bfd17528b0ba2a4', 0.2621963300043717),
 ('swh:1:cnt:6bf0c97a7f84620a0bb4cf6380ec307748e043bd', 0.2433579899952747),
 ('swh:1:cnt:945c9b46d684f08ec84cb316e1dc0061e361f794', 0.2411985010112403),
 ('swh:1:cnt:8c7e5a667f1b771847fe88c01c3de34413a1b220', 0.23372483100683894),
 ('swh:1:cnt:f59ec20aabf5842d237244ece8c81ab184faeac1', 0.21547396900132298),
 ('swh:1:cnt:3818ded670e4fc852dd922ff3299908102068e42', 0.21118225500686094),
 ('swh:1:cnt:f7a8cadeb57c7e92cda09b565e88c21487119d8e', 0.2094835379975848),
 ('swh:1:cnt:63d8dbd40c23542e740659a7168a0ce3138ea748', 0.20489607700437773),
 ('swh:1:cnt:eea1bf0c31f3d493509fa1ddd9d25c734eb35277', 0.20122505999461282),
 ('swh:1:cnt:22ded55aa2c14af167996451c06c190f270e78a9', 0.19538175000343472),
 ('swh:1:cnt:301160a93062df23030a69f4b5e4d9bf71866ee9', 0.18937872498645447),
 ('swh:1:cnt:8214d0ee079917c29e57d16e764fc46de8fb50bf', 0.18728968499635812),
 ('swh:1:cnt:ea9ed43d2b2445bf0ef9bc4e146f93f329d90109', 0.1857915430009598),
 ('swh:1:cnt:050ac90ecbd9ce5a88212058fad711b4231d104d', 0.18484142399393022),
 ('swh:1:cnt:7813681f5b41c028345ca62a2be376bae70b7f61', 0.18475935500464402),
 ('swh:1:cnt:5416677bc7dab0c8bec3f5bf44d7d28b4ff73b13', 0.18474558500747662),
 ('swh:1:cnt:c9cdc63b07017d57efbf45789ceaecc069064a8b', 0.18455564399482682),
 ('swh:1:cnt:4287ca8617970fa8fc025b75cb319c7032706910', 0.1804604439967079),
 ('swh:1:cnt:ce542efaa5124a0437f0c4db329d7ec4b7ba70a7', 0.1795915170077933),
 ('swh:1:cnt:009080e8e0bf834027bd7a55094c2e09e868e67b', 0.17704322800273076),
 ('swh:1:cnt:f11c82a4cb6cc2e8f3bdf52b5cdeaad4d5bb214e', 0.1764608550001867),
 ('swh:1:cnt:e440e5c842586965a7fb77deda2eca68612b1f53', 0.17543939800816588),
 ('swh:1:cnt:7937c68fbcf7c484f2d5ce7801944416eedf0d2c', 0.17449528699216899),
 ('swh:1:cnt:c59d9b6344f1af00e504ba698129f07a34bbed8d', 0.17245302400260698),
 ('swh:1:cnt:4f6c4ee9d928270b4304e3abcd8d81df3e740d12', 0.16957332199672237),
 ('swh:1:cnt:d50394efdf1c93a17379ff5799d1922b2d436026', 0.16944666299968958),
 ('swh:1:cnt:8ac2eb508956928faf94cbca788daa7b2aeb7735', 0.1676800139975967),
 ('swh:1:cnt:45c30f49c0daf7d28dedf8e6caf6196e5caf4a92', 0.1665965900028823),
 ('swh:1:cnt:d133603777a89a23bce43ec32130942a3c93ba5c', 0.16466190699429717),
 ('swh:1:cnt:3cf20d57b0b8258463711cedd592007b0b5cdfe8', 0.16346620599506423),
 ('swh:1:cnt:c7930257dfef505fd996e1d6f22f2f35149990d0', 0.16118666200782172),
 ('swh:1:cnt:35ec3b9d7586b46c0fd3450ba21e30ef666cfcd6', 0.16060386200842913),
 ('swh:1:cnt:02358d2358658574ba0767140caa4216ee7ea5bf', 0.16031745600048453),
 ('swh:1:cnt:675f43ab433f9bacd8574d633afe18364af6a107', 0.1596656440087827),
 ('swh:1:cnt:1c8a0e7976207fb9f03ed7e260950b62b8b9d396', 0.15749601899005938),
 ('swh:1:cnt:c32b65a02039b3e986f3b49bdd19f90ffd6e23ff', 0.15647462400374934),
 ('swh:1:cnt:9c95a6ba6af7377402d44dc60243769a4c604364', 0.15411921599297784),
 ('swh:1:cnt:f4c82dc4e83cb89b72d1713652f6cdc19ccb2aa4', 0.15246416500303894),
 ('swh:1:cnt:3410062ba67c5ed59b854387a8bc0ec012479368', 0.15152129999478348),
 ('swh:1:cnt:c1b0730e0133447badcfd47fd144e254807b06e1', 0.15148620000400115),
 ('swh:1:cnt:00b15c0a321af2492988194cb33b5f55b2ae8332', 0.15074375200492796),
 ('swh:1:cnt:e7754cae5adecf1f21102527fbdeae39280f8e24', 0.15019936999306083),
 ('swh:1:cnt:4be2460889810589fe96270d93248b528079cc7b', 0.14942939100728836),
 ('swh:1:cnt:0022a3ee843f9c3d983aaf4f255cefeafe2859a0', 0.14865684800315648),
 ('swh:1:cnt:be54354a9433a1e798cf17a5cddffbf581e3afa2', 0.14843373600160703),
 ('swh:1:cnt:b7d5379f9e3bb741760e46cf87c89b081f8f44d9', 0.1478635690000374),
 ('swh:1:cnt:a3871d4508259dc2e3eae0be6fc69d1d3daf0a35', 0.14779326801362913),
 ('swh:1:cnt:083b700b23248d19fdf63d7501b802a9556dcd90', 0.14761492000252474),
 ('swh:1:cnt:6f4f765ed6998a016a32bf9b4c48b77219ef7f0a', 0.14696511600050144),
 ('swh:1:cnt:195ee38114e6c3445dd9b88f3dd21cc184662067', 0.14639892800187226),
 ('swh:1:cnt:9fb75b8d4f4c7faa7ba59d138746231ada07c7b0', 0.14601674900040962),
 ('swh:1:cnt:c5fa78456dbde5c0ffa05377925131501c0c794d', 0.1458279729995411),
 ('swh:1:cnt:147efaa6d93d25dca26529dc3940f2079c330880', 0.1457386249967385),
 ('swh:1:cnt:41622b472098a6142f0225b50140189cfbd51779', 0.1456061359931482),
 ('swh:1:cnt:ea0c8a85cb7293feae2c9e151d1d395be59b61fa', 0.1455773979978403),
 ('swh:1:cnt:1d2f01491f783c8c7f0917cc68526c6307d80e39', 0.14541474600264337),
 ('swh:1:cnt:c137216fe167556782049b618443432e9409fa53', 0.1449505560012767),
 ('swh:1:cnt:8b1296cad20a66969613992eb9e0eb493a1b21fa', 0.14479652799491305),
 ('swh:1:cnt:2f94675b7cc5772325e379fc48538156958bab62', 0.14464635599870235),
 ('swh:1:cnt:24de910c13bb1e60fc5ec37a1058d356b1f2fa4d', 0.14444536399969365),
 ('swh:1:cnt:96583aabf5a13fdc518899efd0c92c41da28eb2f', 0.14431070799764711),
 ('swh:1:cnt:b4158c40d2962b0ec90bde1019290ae984adf2a3', 0.14361653200467117),
 ('swh:1:cnt:64845fb7679efbd24296bf0499f7386cde449f0a', 0.14343796600587666),
 ('swh:1:cnt:5639b6ddcf62a692b4cbd6d97d3138829c83b9e7', 0.14333332500245888),
 ('swh:1:cnt:02691e3522cd51ad600902451eace5b54cce5018', 0.1430939860001672),
 ('swh:1:cnt:9e99dd5ec98fdb081b58e76344a05122e25dcb94', 0.14267513799131848),
 ('swh:1:cnt:4977bc62c0e5eedda3a63e1250139237f32fb671', 0.1415989140077727),
 ('swh:1:cnt:ec5fd4221a9dbb645852c59f2a455671b4f2bfd0', 0.14137887800461613),
 ('swh:1:cnt:d426ba8a32b9ab2d8f994b193c0302ea46d5e5c3', 0.1410460159968352),
 ('swh:1:cnt:54caf60b13678be3b856d9d9d5b1aa16b6cbe8fe', 0.14093361899722368),
 ('swh:1:cnt:45a8ca02bfc82ca6f053c14f35bc90a6382b2612', 0.1404536750051193),
 ('swh:1:cnt:0d758c9c7bc06c1e307f05d92d896aaf0a8a6d2c', 0.1404287490004208),
 ('swh:1:cnt:8e2f0bef135ba8e52e4110b6a5b0ebf19a528ca4', 0.1398789829981979),
 ('swh:1:cnt:08e5b85ed7d87899de10f8cda7f7d94c8541a8fd', 0.13933080100105144),
 ('swh:1:cnt:449e49efc2a6de36206ab0e4f38ecfce9ec15632', 0.139222580997739),
 ('swh:1:cnt:baf72b1da3ee845c0543fe0acf4e02e1a031f397', 0.13826104599866085),
 ('swh:1:cnt:22aac29bb31be88e6db5e2f285243c3ab0c921dc', 0.13819912100734655),
 ('swh:1:cnt:a4a063a1573238b224841eb6556061a7ec1f822e', 0.13772802700987086),
 ('swh:1:cnt:b66e5558e5a67ea386d6f6fea028555b8009d0f9', 0.1375916539982427),
 ('swh:1:cnt:e25f1814e51579d5f55c0f1fe0135ddb28a47f4a', 0.1364948799891863),
 ('swh:1:cnt:e8a0f87653d8b78789cb183ba19f357c636ad33f', 0.136356565009919),
 ('swh:1:cnt:08b9811c98f0d90dbacc006ddcd80c5945b9ea55', 0.13532431800558697),
 ('swh:1:cnt:ad2823b48f78a0667817300ddcac54c2f6c385e9', 0.13528257299913093),
 ('swh:1:cnt:851c75cc5e74cf97d145360755d116d2409881f2', 0.13496985100209713),
 ('swh:1:cnt:4fc3fe1ce587c1fb2acc24b2074dac4438d1b30b', 0.13488715499988757),
 ('swh:1:cnt:dd67376917e5edd850e1db02049d5c6115462d8d', 0.13461925899900962),
 ('swh:1:cnt:6d0b7ebde95e1fbd7cc77a60d46a3468f61038cc', 0.13397039999836124),
 ('swh:1:cnt:2f259b79aa7e263f5829bb6e98096e7ec976d998', 0.133883173999493),
 ('swh:1:cnt:52d0524746f70b293da3def801a5823b8b9ce3a1', 0.1321609300066484),
 ('swh:1:cnt:ae9780bc629ea3cb16fadbba91e393318af71465', 0.13205293599457946),
 ('swh:1:cnt:12183abc9e3e34700451537045dd9c0b6db609f3', 0.13119369999913033),
 ('swh:1:cnt:6b10f9584314d9e7304b2c13c480fc3d01acabe9', 0.1306781029998092),
 ('swh:1:cnt:ac044e5e4649cd149e3d0cf9d23720d299288a1e', 0.13016800899640657),
 ('swh:1:cnt:e7a5832a436197d3bdc6c6d879a732c627497b0d', 0.12988247400789987),
 ('swh:1:cnt:0901ffdd0da48f1648b19e1833a4e2d9b5a24fbe', 0.1292870510078501),
 ('swh:1:cnt:bb79ec2de59197fe11eeb60d312673a87c1b8932', 0.12882517100661062),
 ('swh:1:cnt:597a6db294cb721d184d6f12560dfa9e8a67de33', 0.128138025989756),
 ('swh:1:cnt:c96ab3cc70e72f99ab51e3bdab9465fe177c7923', 0.12761187199794222),
 ('swh:1:cnt:ff30235f076b3fde71fba6ceafa0f510e0006261', 0.12676391001150478),
 ('swh:1:cnt:3f1695f161395210e9ab0e22d48014e179514e21', 0.12673786999948788),
 ('swh:1:cnt:8a908eccd273b758a246975d2eaf96968ceb330d', 0.12598939800227527),
 ('swh:1:cnt:0817502b11d3d776cac53346c03336e236cfa665', 0.12572567899769638),
 ('swh:1:cnt:265ee26ceb272945f7c08a8f37c79bc6c55bb4d8', 0.12535347799712326),
 ('swh:1:cnt:07e2dac48ff656acc86895efc33941223195b371', 0.12511731700215023),
 ('swh:1:cnt:67c3297611451bb352bbda488b0cd7cb2528a0d3', 0.12454920700110961),
 ('swh:1:cnt:680fc5115355e433bf39e51c98f66ddb8e3dce22', 0.12445053900592029),
 ('swh:1:cnt:16e45d390712388410556b522c74a11637716844', 0.12437790801050141),
 ('swh:1:cnt:f3c6c3c68af484bd95bc7b7a38276c6c014d6aa3', 0.12378032201377209),
 ('swh:1:cnt:1db515f9a1604d126df3ed98ffa4cc5a844b96a2', 0.12293052799941506),
 ('swh:1:cnt:ef073cc45ccc66c921adc7ccc6221e38aa54ae17', 0.12280515799648128),
 ('swh:1:cnt:3a6e607aa5ab4cb455d463049b8c5d7ed0d6e4b8', 0.12247212900547311),
 ('swh:1:cnt:4d1ae35ba2c8ec712fa2a379db44ad639ca277bd', 0.122047034004936),
 ('swh:1:cnt:e49435269d332947aed299a315268e347fb29e7e', 0.12190942499728408),
 ('swh:1:cnt:23fa7d31a729cb3b60694cf15e906aee5823b96f', 0.12151172399171628),
 ('swh:1:cnt:f46d387bf94c89b81aaec9cff2ee52d1f9ab4187', 0.12067821700475179),
 ('swh:1:cnt:8cbf1afc66d01c03210c74c6c3837cf879484525', 0.12047362700104713),
 ('swh:1:cnt:b8635bcf0d86ea51ce65da503c779e72262c0c9e', 0.12002620300336275),
 ('swh:1:cnt:74e0f12e3246e5d0b556558359a30e0991092cdc', 0.11952875400311314),
 ('swh:1:cnt:b54b8e79bb8d039944e1adf71c97c3c74676ab73', 0.11898280201421585),
 ('swh:1:cnt:34164076ecce82a01452f1c2979bac48c45e3afb', 0.1185239300102694),
 ('swh:1:cnt:fa7af8bf5fdd704f73beb3adc5612682a98e1af5', 0.1180789240024751),
 ('swh:1:cnt:866ad4749eb52020cf6b9791205db4af53d337e1', 0.11649419099558145),
 ('swh:1:cnt:b1d81e79354c904c6cd8bf5bf91f4c9c9502f381', 0.11645552099798806),
 ('swh:1:cnt:7371f47a6f8bd23a8fa1a8b2a9479cdd76380e54', 0.11635677400045097),
 ('swh:1:cnt:afb09b4e6d0e92bd13951b4f9d3ded90c8c4e89a', 0.11556883899902459),
 ('swh:1:cnt:05fdf94240ddb18dae9aa42f4d35d6099b93bc7e', 0.11518792899732944),
 ('swh:1:cnt:b516b2c489f1fefd53b4a1905bf37480a8eb056d', 0.11514432700641919),
 ('swh:1:cnt:ddf3b06abfe0500f89028d1b1304c57169da8c87', 0.11471358600829262),
 ('swh:1:cnt:84f8fe2f54c9b5bf32ddbea402b6ae4af382fa76', 0.11457952800265048),
 ('swh:1:cnt:7136b4c89bbca73c68a2ee0eaca246942d6fa2bb', 0.11456015700241551),
 ('swh:1:cnt:cd571f464ed47d4549f59166841c6a041921342b', 0.11393886899168137),
 ('swh:1:cnt:8c53a7f5f36f2172d12e3142124b3da1ee9321b6', 0.11393855900678318),
 ('swh:1:cnt:96d80cd6c4e7158dbebd0849f4fb7ce513e5828c', 0.11330178899515886),
 ('swh:1:cnt:501a6bbaf1e2cec78ba2c39f1bd551a43638094b', 0.11302238899224903),
 ('swh:1:cnt:9d68933c44f13985b9eb19159da6eb3ff0e574bf', 0.112861232002615),
 ('swh:1:cnt:013d565bb4055b02b0853816a9f6c69d1478f3f3', 0.11262685600377154),
 ('swh:1:cnt:6cc8370785d2e6c51b9de5ff36cda89a1a9aa2f9', 0.11122256600356195),
 ('swh:1:cnt:04f7b5be6987d303c24a0aa2761207922f6bbf66', 0.10857875800866168),
 ('swh:1:cnt:0fe2fa50e8e79f56b23267be86bde68d6f86195b', 0.10745159100042656),
 ('swh:1:cnt:32f64f4d836716819dc5fa9a1e09a29b428881df', 0.10711265300051309),
 ('swh:1:cnt:bb7d13c5e9acc98633147acb5e2e5cd2dda09978', 0.10648624999157619),
 ('swh:1:cnt:883ad6e8ef9a7392b45f6fc9e7d53c88f502388b', 0.10603477699623909),
 ('swh:1:cnt:14ceb3ee12a1980c71ee530968609e528159164e', 0.10549176299537066),
 ('swh:1:cnt:9cbe6ea56f225388ae614c419249bfc6d734cc30', 0.10522356299043167),
 ('swh:1:cnt:3ea63c2ccdc40ee588966429fe688570ccb02c7c', 0.10439902399957646),
 ('swh:1:cnt:60a89ed235449c0bc49f2280ca81ccc88125fff6', 0.10239979698962998),
 ('swh:1:cnt:b59c5945f9f68d5e239c35ed33100dd24b27b800', 0.1020320000097854),
 ('swh:1:cnt:500c0709ca24338426091ca19777e13a1920ebdf', 0.10093228600453585),
 ('swh:1:cnt:9b26e9b102ab2917db3dc1f6ced91065a38205a8', 0.10081301801255904),
 ('swh:1:cnt:16e0e90df089debac429ebcae3cd8ea4b886aa62', 0.09986729599768296),
 ('swh:1:cnt:ef6bce1d1d15c6721aa1c5cce64b10378dfcc844', 0.09947287999966647),
 ('swh:1:cnt:81750b96f9d83b395f285233d54ec0c9df9ab93d', 0.09753718100546394),
 ('swh:1:cnt:835a5816354544643aeb095a3de6ae8c43136c6d', 0.09495558899652679),
 ('swh:1:cnt:303e398c82e88040b372475614e4fe2888bcbe56', 0.09298244600358885),
 ('swh:1:cnt:0f137124110a2cc080e70794f98f7c8e5fd87e75', 0.09087489299417939),
 ('swh:1:cnt:1d79949788956cdcd635f20bc67797e63beb8df9', 0.08956174699414987),
 ('swh:1:cnt:080c324fec6abbac88b03ca1298b71b6d69f9de1', 0.0866937870014226),
 ('swh:1:cnt:59cd9bddc043471be2a00f84e2a122af85fe8a08', 0.08621002399013378),
 ('swh:1:cnt:eda5949cbef3c59679d61a78aa0654d43c3f06fa', 0.08347308699740097),
 ('swh:1:cnt:5a77f05831a38ab834f5f49342e29f931e073f5a', 0.07940481000696309),
 ('swh:1:cnt:52771883833a931bf221105e2eb19fdc30a1631a', 0.07753480700193904),
 ('swh:1:cnt:92a39f398b80310c58aa2574dd97a3e35527b73b', 0.07648618699749932),
 ('swh:1:cnt:2b8e128b91d3f449a15bf506aa87126c40bc68d8', 0.07635874999687076),
 ('swh:1:cnt:4f0734cbe3ae9d87287203f06e908edec7c8ed4d', 0.07633817399619147),
 ('swh:1:cnt:5cd813e5c5f312673ce9cf39fb832fb2d55116cc', 0.0751863699988462),
 ('swh:1:cnt:e5db411da00fcba56fe620bad5192fc6a7b1325b', 0.07496902298589703),
 ('swh:1:cnt:aa91d5c03e2002a1057f8075f67a189aeaae3f2b', 0.0749124839931028),
 ('swh:1:cnt:2725bca0006db42c8ee38c15dfa290bbe57f9a94', 0.07430420198943466),
 ('swh:1:cnt:395831125cab5cd356b62c6d4883454a8a511a28', 0.07388512299803551),
 ('swh:1:cnt:ef6080906700f3f3cdac7d60341a5de7b5da5581', 0.0730236910021631),
 ('swh:1:cnt:c2fb4f3370c4e65a8a889f93628d0dd0d0d02726', 0.07297272598952986),
 ('swh:1:cnt:5bd7dea14b4c665b222ff69cbb7a0496dc3b71b1', 0.07270906699704938),
 ('swh:1:cnt:825026bf2f91b30f020cdba8326485cc600d8a4d', 0.07195889099966735),
 ('swh:1:cnt:e2cd95fdbf767aa5e9a1f1bd67167db4a0725d5d', 0.07153263100190088),
 ('swh:1:cnt:fe1990c57a840e97c1075904f9e96daed2e65030', 0.07113850599853322),
 ('swh:1:cnt:c30d0581bfa3e1d73eb6a5efbe638c72911dadb7', 0.07103491699672304),
 ('swh:1:cnt:ff7dbb100623d2c1f447485012e8de229821cc84', 0.07072352999239229),
 ('swh:1:cnt:a5a3498915908e0e03f66cd28e52b9fc87d1c3d5', 0.07065946099464782),
 ('swh:1:cnt:8b43ca9ac41e863b1817ba9c2631c33ed0b95265', 0.07064405000710394),
 ('swh:1:cnt:bd0fd35942234660f0c30ba251abaa81eda68fd1', 0.07011526499991305),
 ('swh:1:cnt:9d3cd6889873b86c6796ac62f33b7c8d57d44bc0', 0.06949705599981826),
 ('swh:1:cnt:4489a65003548ac3a08ee5d03b9535c7f7f7dead', 0.06886444200063124),
 ('swh:1:cnt:f8fa5a235408d5c6f2c2844e44cd98c1fa666ac8', 0.06867764701019041),
 ('swh:1:cnt:8b0e2fb7fa3d187b31a6970199816db30d79e8df', 0.06800826400285587),
 ('swh:1:cnt:c471733217fd3bdca8cae8a713b434a3d16ea303', 0.06730564999452326),
 ('swh:1:cnt:0015308348bf55e30efa51f842ae48361a8d0bdb', 0.06713406200287864),
 ('swh:1:cnt:4e3bc519eef7d5a35fff67687eaee65616832e45', 0.06676060300378595),
 ('swh:1:cnt:caad5337178380d33d72638694c823b03963b7da', 0.0661511249927571),
 ('swh:1:cnt:301792a3a0f42258c745f18acbbcc5dcf0286be0', 0.06595788100094069),
 ('swh:1:cnt:39e8d66025179ed66f49e6d776fe9c4468d14eff', 0.06585914600873366),
 ('swh:1:cnt:152f9ed5aa2d840c4a115d34ac0271262ca416ef', 0.06557884599897079),
 ('swh:1:cnt:b9798c2427e725d88f22a1fe75cd165046f614f7', 0.06556966899370309),
 ('swh:1:cnt:57814dd0025fca1ddd1e1058fa08ca9e56651f04', 0.06552854900655802),
 ('swh:1:cnt:fc2b5693e00bdbea3f74ac60459b1df43c295356', 0.06552601399016567),
 ('swh:1:cnt:9beacf8e6f5cd023ac57d0b97432c7cc250c6b96', 0.06538455000554677),
 ('swh:1:cnt:0f0f3e3b1ff2bc6722afc3e3812e6b782683896f', 0.06512390199350193),
 ('swh:1:cnt:6d7ce64fb8c0411b6a006095186464c3a70de1be', 0.06415866299357731),
 ('swh:1:cnt:b15531f5dd7f917929f0331f7f65018f9c7c25cc', 0.06382859099539928),
 ('swh:1:cnt:79b7e4bad2a1676d0db76503825455527b89d932', 0.06352026600507088),
 ('swh:1:cnt:d6c11f4cd0a101d19c54e9c378526bdf21988732', 0.06330892100231722),
 ('swh:1:cnt:50c8be35f7782588f82895b7f1890f5fca711866', 0.06326884900045116),
 ('swh:1:cnt:148a7b00c1946b6655f525fcfbd1dade6cf0dd2e', 0.0630232609983068),
 ('swh:1:cnt:e0aa8a9ce7243aa018b70b3839d567b14fcf01e0', 0.0629849459946854),
 ('swh:1:cnt:40a2dbb37b14d86d8da15ba799af13f013722c2a', 0.062373363005463034),
 ('swh:1:cnt:2a2a526d8c3f95e6b1c615a3e056806cbf8cc7a3', 0.06229305699525867),
 ('swh:1:cnt:7c42345690273ab86713294ad761386871c7b708', 0.0622266669961391),
 ('swh:1:cnt:f7cb26f07be14f68a35431ff8ab12b69b61424a3', 0.06218647900095675),
 ('swh:1:cnt:a2344cdfa5b899d869dde30985646273655014d0', 0.06190200099081267),
 ('swh:1:cnt:eec1a4233f3f92568aa3ede33cb1f4ddb6dcc251', 0.0616499869938707),
 ('swh:1:cnt:566d900abb9af2df549e573a5c0b8c5d7c3204fc', 0.06106720400566701),
 ('swh:1:cnt:2882b1818ef42e381f47427af8f2cf8dc2337e10', 0.06093267801043112),
 ('swh:1:cnt:f327b9acc7ffd3c3008a5363dad25358ca32f643', 0.06080726400250569),
 ('swh:1:cnt:1d8c7673ef786c5ff5e3b3defd0040af843bbc95', 0.06058294599642977),
 ('swh:1:cnt:df3d21358f562c835826070e8ef31e0de97e13a4', 0.05955267799436115),
 ('swh:1:cnt:384f448f22e6a76511824c5395336eb84e13959f', 0.05937159100722056),
 ('swh:1:cnt:bd97d75340e460bdddedbc17115518f47aa5ba93', 0.059195893991272897),
 ('swh:1:cnt:52e60b4495af680809173f50b2918757eb176e76', 0.05870940499880817),
 ('swh:1:cnt:1987bbd3f002a19fb61286144bb143274285f89d', 0.05845894600497559),
 ('swh:1:cnt:e0074173a81d3825f14fe83d3aefc03d51476f00', 0.05758264599717222),
 ('swh:1:cnt:7b71c6e679738d980d5d03559d35fd88a66ee9e4', 0.057297752995509654),
 ('swh:1:cnt:5d33882543d5ade31a988fd3b8aef02faf12d742', 0.05705296999076381),
 ('swh:1:cnt:9fc8861321f106d71a197c0e06655b10242f82a1', 0.05689964100020006),
 ('swh:1:cnt:8f7462996850b690a4a2b2d5c368475744ec29fb', 0.05633648300135974),
 ('swh:1:cnt:a4ceb359eba9f667d23d07784d7208f8188fcbbe', 0.0561668829905102),
 ('swh:1:cnt:9de294132dec5e172c3245f7453b48ddfd1b2006', 0.055872215001727454),
 ('swh:1:cnt:31f442a2f86c480aa94912d5c2ead0b70426e2fb', 0.05565502599347383),
 ('swh:1:cnt:e515dc1cf2373da83cc4b248c392c241e4ea2afc', 0.055492690997198224),
 ('swh:1:cnt:f9825866e00f2b8cd33a9dabe1a23a9ee7b3c984', 0.05543490000127349),
 ('swh:1:cnt:b0b2b1c8ddb2dd9e025526d2ea88d82854de6be8', 0.0551386029983405),
 ('swh:1:cnt:1bd8bf6057312efa3c5e9d7e11fdacd688640b6c', 0.05460915500589181),
 ('swh:1:cnt:9280c0d31d5a7c1fd9abaaa7ffd6160759eca320', 0.053715239992015995),
 ('swh:1:cnt:1bbf238cfb853e6bc5f6f5c85118d6b850b581ff', 0.05342404200928286),
 ('swh:1:cnt:f1b38290cdf3564ec0d9a35bea05c5cb5cd5f503', 0.05275039000844117),
 ('swh:1:cnt:54a81dcac6cfc078fbf4b74de360a4a2d9c762f5', 0.052574317989638075),
 ('swh:1:cnt:127620c78a33b65b9b1b3f40843e7236e50e8c6c', 0.0523124210012611),
 ('swh:1:cnt:a8e73b1b7c227edf8c112fc273e9929b88301b13', 0.051834504993166775),
 ('swh:1:cnt:13c603768561c8efce4659491508b21865b635f8', 0.050538962008431554),
 ('swh:1:cnt:05de4782e137f9a25e63fa2cac3c94ba218dc69d', 0.04904750798596069),
 ('swh:1:cnt:d77740a55759db09434d496f10527e0560ee590d', 0.04897115800122265),
 ('swh:1:cnt:7de85f863158583176767468e52c7cbb92996542', 0.047341456011054106),
 ('swh:1:cnt:2105af1f86eb44a91698fd33caccf5e105b36f9e', 0.04585653000685852),
 ('swh:1:cnt:46df076df476712741056c452b634482efcf74d8', 0.044585834999452345),
 ('swh:1:cnt:1de00ecd4fc96b12f0281d2a802ff1ecdf76256a', 0.04199193700333126),
 ('swh:1:cnt:1578e7b311acfb74c9dffb8a38edfa9a4171fdcc', 0.039945630996953696),
 ('swh:1:cnt:27f1d22a655c42c5a5fc9ffd583789950da11fc8', 0.03604459000052884),
 ('swh:1:cnt:4238428a9eacce28e3248da18e7847c756668e6b', 0.03555884098750539)]

so winners are the empty file and the empty directory:

 ('swh:1:cnt:e69de29bb2d1d6434b8b29ae775ad8c2e48c5391', 36.285656899999594),
 ('swh:1:dir:4b825dc642cb6eb9a060e54bf8d69288fbee4904', 13.257851957008825),

Then (swhids with a response time > 1s):

swh:1:dir:29a422c19251aeaeb907175e9b3219a9bed6c616 5.86  Directory with empty .keep
swh:1:cnt:8b137891791fe96927ad78e64b0aad7bded08bdc 4.98  1-byte contents
swh:1:cnt:654d0bfe943437d43242325b1fbcff5f400d84ee 4.61  MIT License
swh:1:dir:d564d0bc3dd917926892c55e3706cc116d5b165e 4.34  Directory with empty .gitkeep
swh:1:cnt:4d29575de80483b005c29bfcac5061cd2f45313e 4.25  Create React App .gitignore
swh:1:cnt:261eeb9e9f8b2b4b0d119366dda99c6fd7d35c64 3.72  Apache License 2.0
swh:1:cnt:dfe0770424b2a19faf507a501ebfc23be8f54e7b 3.67  Very common .gitattributes
swh:1:cnt:e7af2f77107d73046421ef56c4684cbfdd3c1e89 3.59  MIT License
swh:1:cnt:f288702d2fa16d3cdf0035b15a9fcbc552cd88e7 2.78  GPLv3
swh:1:cnt:1ff0c423042b46cb1d617b81efb715defbe8054d 2.62  VisualStudio default .gitattributes
swh:1:cnt:94a9ed024d3859793618152ea559a168bbcbb5e2 2.61  GPLv3
swh:1:cnt:19129e315fe593965a2fdd50ec0d1253bcbd2ece 2.35  ISB License
swh:1:cnt:d645695673349e3947e8e5ae42332d0ac3164cd7 2.08  Apache License 2.0
swh:1:cnt:8dada3edaf50dbc082c9a125058f25def75e625a 1.72  Apache License 2.0
swh:1:cnt:ee27ba4b4412b0e4a05af5e3d8a005bc6681fdf3 1.66  MIT license
swh:1:cnt:9cecc1d4669ee8af2ca727a5d8cde10cd8b2d7cc 1.49  GPLv3
swh:1:cnt:6b60c1042f58d9fabb75485aa3624dddcf633b5c 1.34  React logo
swh:1:cnt:d159169d1050894d3ea3b98e1c965c4058208fe1 1.26  GPLv2

so the v0.2.0 is the response, isn't it ?

(deleted comment, read too fast).

v0.2 is a workaround (for now) I believe yes

I made another MR to make it possible to disable filter license files using a config option.

Not sure whether we want to set it true or false for the prod deployment...

swh/devel/swh-provenance!180 (merged)

marked this issue as related to #5413 (closed)

marked the checklist item Deploy the last version of swh-provenance (the one with swh/devel/swh-provenance!178 (merged) and swh/devel/swh-provenance!179 (merged)) as completed

marked the checklist item [out-of-scope] Add debug logs on swh-provenance to trace the durations of the queries to swh-graph with the swhid / global duration (swh/devel/swh-provenance!179 (merged)) as completed

mentioned in commit swh/infra/ci-cd/swh-charts@8ebc0ab5

The chain as now 16 more workers to consume, just to ensure the demo goes well.

The rollback will be done next week

The rollback to 2 replicas of provenance pods (4 workers each) was made during the migration to the cassandra namespace.

closed