Make long running task stop fast when warm shutdown is triggered

This would currently help to stop/restart fast swh-worker services (as currently deployed in our stack). That would also help with elastic workers to downscale fast (if need be) or to time frame some loaders. [1]

As it is not currently the case, a stop or restart is as fast as the current ingestion ongoing will take. For some hg or git repositories, that can take time (hours, well there is now a timeout but still).

This could be implemented for example with a signal handler to react to the warm shutdown signal. For loaders, the signal handler would do a partial snapshot with what it had ingested so far. For listers, it can flush its visited state for the ongoing forge (on incremental).

[1] Related to #3592 (closed) (but as described not only that)

Migrated from T3640 (view on Phabricator)

Edited Oct 18, 2022 by Antoine R. Dumont