replay: overhaul worker thread reporting

• Strong typing for report queue messages
• Add idle and "object in progress" reporting
• Add the ability to request a restart
• Restart worker threads when they raise an issue
• Print warnings when a thread is detected as stalled (including which object is being processed).

assigned to @olasd

added 1 commit

• 99c70d65 - replay: Clarify some variable names

Compare with previous version

Jenkins job DOBJSRPL/gitlab-builds #69 succeeded in 3 min 22 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

Jenkins job DOBJSRPL/gitlab-builds #70 succeeded in 3 min 12 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

mentioned in commit swh/infra/swh-apps@1152acce

mentioned in commit swh/infra/ci-cd/swh-charts@792d4c11

added 1 commit

• 4d9b6ebe - Report worker stalls to a status file

Compare with previous version

Jenkins job DOBJSRPL/gitlab-builds #71 succeeded in 3 min 32 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

marked this merge request as ready

This has been running the production winery replayer for a while and seems fairly stable.

I'm not a huge fan of the thread stall reporting going through a temporary file that gets monitored by the container runtime, but there's no way to kill a thread, and I'm not sure moving to a multiprocess architecture is really worth it? Combined with a static consumer group, the restarting behavior is fairly transparent.

I'm not sure what kind of approach to take for tests either.

added 4 commits

• b8384a5c - 1 commit from branch swh/devel:master
• 9b475737 - replay: overhaul worker thread reporting
• 11893268 - replay: Clarify some variable names
• bbd1cc25 - Report worker stalls to a status file

Compare with previous version

Jenkins job DOBJSRPL/gitlab-builds #76 succeeded in 2 min 32 sec.
See Console Output, Blue Ocean and Coverage Report for more details.