Skip to content

WebAPIClient: add the option to issue some request concurrently

Pierre-Yves David requested to merge marmoute/swh-web-client:parallel into master

Some large call might be sliced into multiple http requests. Since we known in advance we will send all these request, issuing then in parallel will run faster.

The new rate limiting machinery will ensure that we do not issue requests faster than the server intend us to do.

The "known" method is using such chunking and use this new approach when the parameter is set.

The default concurrency of 20 have been picked arbitrarily, it seem large enough to provide a significant speedup and small enough to not hammer the server too much.

Merge request reports

Loading