Skip to content

lister: Add utility module to ease HTTP requests rate limit handling

Antoine Lambert requested to merge generated-differential-D4869-source into generated-differential-D4869-target

Add swh.lister.utils.throttling_retry decorator enabling to retry a function that performs an HTTP request who can return a 429 status code.

The implementation is based on the tenacity module and it is assumed that the requests library is used when querying an URL.

The default wait strategy is based on exponential backoff.

The default max number of attempts is set to 5, HTTPError exception will then be reraised.

All tenacity.retry parameters can also be overridden in client code.

I also ensured that introduced code can be executed and tested with the tenacity version packaged in debian buster (4.12).

Example of use:


import requests

from swh.lister.utils import throttling_retry

URL = "http://example.org/api/repositories"

@throttling_retry()
def make_request():
    response = requests.get(URL)
    response.raise_for_status()
    return response

Migrated from D4869 (view on Phabricator)

Merge request reports