Skip to content
Snippets Groups Projects
Forked from Platform / Development / swh-loader-mercurial
Source project has a limited visibility.
  • Raphaël Gomès's avatar
    23260277
    Use billiard instead of stdlib multiprocessing · 23260277
    Raphaël Gomès authored
    This circumvents a few celery-related issues, and is consistent with
    what the rest of the codebase does.
    
    stdlib multiprocessing is not able to spawn children from daemonic
    processes, and even says so plainly if you try:
    
    `AssertionError: daemonic processes are not allowed to have children`
    
    This is incompatible with the SWH infrastructure which needs to do this
    exactly. Fortunately, we're already using billiard and celery. I'm
    assuming that there could be other blocking or annoying differences
    between stdlib and billiard, but we will save ourselves the trouble of
    finding out.
    23260277
    History
    Use billiard instead of stdlib multiprocessing
    Raphaël Gomès authored
    This circumvents a few celery-related issues, and is consistent with
    what the rest of the codebase does.
    
    stdlib multiprocessing is not able to spawn children from daemonic
    processes, and even says so plainly if you try:
    
    `AssertionError: daemonic processes are not allowed to have children`
    
    This is incompatible with the SWH infrastructure which needs to do this
    exactly. Fortunately, we're already using billiard and celery. I'm
    assuming that there could be other blocking or annoying differences
    between stdlib and billiard, but we will save ourselves the trouble of
    finding out.