failing worker consumes remaining tasks without processing them
What is implicit in the parent task swh-loader-mercurial#964 (closed):
- out of ram worker is killed (it cannot clean up since it's killed)
- the node running the worker is then mostly idle for that particular work (in regards to the other sister nodes)
- so it starts consuming the queue faster than the other workers (since they do actual work)
- and fails faster
- resulting in an empty queue in the end
That is what i was trying to solve in swh-loader-mercurial#964 (closed) (well finding proper solution to implement for the moment).
As I realized it was not explicitly mentioned, opening a dedicated issue for it.
Migrated from T982 (view on Phabricator)