Counter refresh history exit in error
Sentry Issue: SWH-COUNTERS-1D
SystemExit: 1
(17 additional frame(s) were not displayed)
...
File "http/client.py", line 1375, in getresponse
response.begin()
File "http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "socket.py", line 705, in readinto
return self._sock.recv_into(b)
File "gunicorn/workers/base.py", line 204, in handle_abort
sys.exit(1)Activity
added Counters environment: production labels
It seems it's a common error when thanos does not manage to reply in time to the history query.
The cronjob retry the query each second in case of error which doesn't help and trigger a log of error in sentry.
The cron job will be updated to use the retry command to correctly ensure the rpc service is running and fail the job if an error occurs. The retry will be automatically done one hour later in case of an error which is better for thanos
assigned to @vsellier
mentioned in commit swh-apps@714e7d06
Not sure it's related, if the historical data was not fetched from thanos, the graph should stop in 2021. As it's end in 2024, at least some data is retrieved, it needs more investigations.
It's more probably a timeout of thanos-query that failed to retrieve the data stored in the azure bucket but the live data in the local prometheus is correctly retrieved, which explain the 2024 values.
A refresh should solve issue.
Edited by Vincent Sellierindeed:
août 26 15:42:28 thanos thanos[3174165]: level=warn ts=2024-08-26T15:42:28.113372213Z caller=proxy.go:571 component=proxy request="min_time:1618414927000 max_time:1724686944000 matchers:<name:\"environment\" value:\"production\" > matchers:<name:\"object_type\" value:\"content\" > matchers:<name:\"__name__\" value:\"swh_archive_object_total\" > aggregates:MAX step:43200000 " err="receive series from Addr: thanos.internal.admin.swh.network:19093 LabelSets: {replica=\"0\", tenant=\"historical-data\"} Mint: 1625378400000 Maxt: 1724680800000: rpc error: code = Unavailable desc = error reading from server: EOF" msg="returning partial response" août 26 15:42:28 thanos thanos[3174165]: level=warn ts=2024-08-26T15:42:28.119404864Z caller=endpointset.go:416 component=endpointset msg="update of endpoint failed" err="getting metadata: fallback fetching info from thanos.internal.admin.swh.network:19093: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 192.168.50.90:19093: connect: connection refused\"" address=thanos.internal.admin.swh.network:19093août 26 15:44:51 thanos kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/system-thanos\x2dstore.slice/thanos-store@historical.service,task=thanos,pid=3693247,uid=113 août 26 15:44:51 thanos kernel: Out of memory: Killed process 3693247 (thanos) total-vm:8923316kB, anon-rss:2761028kB, file-rss:0kB, shmem-rss:0kB, UID:113 pgtables:16056kB oom_score_adj:0
Nice to have the full counters history back ! This made me notice a small issue in the histogram tooltip code, fixed in swh/devel/swh-web!1309 (merged).
mentioned in commit swh/infra/ci-cd/swh-charts@454956ba
mentioned in merge request swh/infra/ci-cd/swh-charts!463 (merged)
mentioned in commit swh-sysadmin-provisioning@f3bac121
mentioned in commit swh/infra/ci-cd/swh-charts@16d29ee4
It seems the jobs is failing due to the 10s gunicorn timeout
❯ kubectl --context archive-production-rke2 get pods -n swh | grep refresh 17:05:15 counters-refresh-counters-cache-cronjob-28746060-b4js8 0/1 Error 0 125m counters-refresh-counters-cache-cronjob-28746060-bh6hh 0/1 Error 0 124m counters-refresh-counters-cache-cronjob-28746060-gdzll 0/1 Completed 0 124m counters-refresh-counters-cache-cronjob-28746060-p78dl 0/1 Error 0 125m counters-refresh-counters-cache-cronjob-28746120-5vz66 0/1 Completed 0 65m counters-refresh-counters-cache-cronjob-28746120-pf4lg 0/1 Error 0 65m counters-refresh-counters-cache-cronjob-28746180-8bfc5 0/1 Error 0 5m49s counters-refresh-counters-cache-cronjob-28746180-cj9cx 0/1 Completed 0 4m56s counters-refresh-counters-cache-cronjob-28746180-q2j67 0/1 Error 0 5m28s ❯ kubectl --context archive-production-rke2 logs -n swh counters-refresh-counters-cache-cronjob-28746180-q2j67 ... + curl -f -v -XPOST -H 'Content-Type: application/json' http://counters-rpc-ingress/refresh_history -d @/tmp/tmp.jbYgzQuSWr Note: Unnecessary use of -X or --request, POST is already inferred. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 10.43.59.157:80... * Connected to counters-rpc-ingress (10.43.59.157) port 80 (#0) > POST /refresh_history HTTP/1.1 > Host: counters-rpc-ingress > User-Agent: curl/7.74.0 > Accept: */* > Content-Type: application/json > Content-Length: 125 > } [125 bytes data] * upload completely sent off: 125 out of 125 bytes 100 125 0 0 100 125 0 12 0:00:10 0:00:10 --:--:-- 0* Mark bundle as not supporting multiuse * The requested URL returned error: 500 Internal Server Error 100 125 0 0 100 125 0 11 0:00:11 0:00:10 0:00:01 0 <------ it fails after 10s * Closing connection 0 curl: (22) The requested URL returned error: 500 Internal Server Error + rm -f /tmp/tmp.jbYgzQuSWrThe timeout is increased to 30s to check if it's more stable
mentioned in commit swh/infra/ci-cd/swh-charts@92fdb49f
mentioned in commit swh/infra/ci-cd/swh-charts@3f785164
mentioned in commit swh/infra/ci-cd/swh-charts@335d78c1
