Skip to content

Make HTTP requests more resilient

Pierre-Yves David requested to merge generated-differential-D8541-source into generated-differential-D8541-target

Series on change to clean the http client code a bit and add a first layer of retry and rate limiting capabilities

This have no reason to be a closure, so we extract it as a proper function.

client: reorder some conditionnal

Let us put the simple case first and the longer one after it.

client: move make_request to be a method

This has not reason to be a closure, so let it not be a closure.

client: transforme the swhid to string earlier

This simplify the function we call in the loop, cannot hurt.

client: compute the client endpoin once and ealier

This make the user function simpler.

fixup! output: pick "summary" as the default

client: prepare for retry

client: add some basic error detection and rate limiting

We retry on error and we adjust to the X-RateLimit header the server send us. This is quite basic, and has to struggle against the async implementation of the request.

So further works should probably refactor the concurrency of the request implementation before going further.

See inline documentation for details.

Better handling of ClientConnectionError are also needed.

Test Plan

ran tox and successully did not got banned from sending http request to SWH

Migrated from D8541 (view on Phabricator)

Merge request reports