Skip to content

failing worker consumes remaining tasks without processing them

What is implicit in the parent task swh-loader-mercurial#964 (closed):

  • out of ram worker is killed (it cannot clean up since it's killed)
  • the node running the worker is then mostly idle for that particular work (in regards to the other sister nodes)
  • so it starts consuming the queue faster than the other workers (since they do actual work)
  • and fails faster
  • resulting in an empty queue in the end

That is what i was trying to solve in swh-loader-mercurial#964 (closed) (well finding proper solution to implement for the moment).

As I realized it was not explicitly mentioned, opening a dedicated issue for it.

Migrated from T982 (view on Phabricator)