Skip to content
Snippets Groups Projects
Forked from Platform / Development / swh-lister
Source project has a limited visibility.
  • Antoine Lambert's avatar
    35871896
    pattern: Improve handling of max_origins_per_page parameter · 35871896
    Antoine Lambert authored
    Instead of fully consuming the get_origins_from_page generator into
    a list and truncate it, prefer to consume the generator origin per
    origin and abort the process when the max number of origin per page
    is reached.
    
    Indeed some non trivial listers like the cgit one can perform costly
    processing, HTTP request for instance, for each origin in a page.
    So better not consuming the full generator in a row to avoid such
    side effects.
    35871896
    History
    pattern: Improve handling of max_origins_per_page parameter
    Antoine Lambert authored
    Instead of fully consuming the get_origins_from_page generator into
    a list and truncate it, prefer to consume the generator origin per
    origin and abort the process when the max number of origin per page
    is reached.
    
    Indeed some non trivial listers like the cgit one can perform costly
    processing, HTTP request for instance, for each origin in a page.
    So better not consuming the full generator in a row to avoid such
    side effects.