save_bulk: Speedup listing process with multi-threading

Check multiple origins in parallel using the concurrent.futures module to greatly speedup the whole listing process.

Related to #4709 (closed)

Jenkins job DLS/gitlab-builds #413 succeeded in 5 min 8 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

approved this merge request

added 1 commit

• 33029c90 - save_bulk: Speedup listing process with multi-threading

Compare with previous version

resolved all threads

^ Update:

• remove not needed futures variable
• set pool max workers to 8
• ensure HTTP requests can be retried as we can be easily hit by rate limit with parallel requests

Jenkins job DLS/gitlab-builds #414 succeeded in 4 min 1 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

added 1 commit

• 4a9c0dbd - save_bulk: Speedup listing process with multi-threading

Compare with previous version

^ using 4 workers is enough, rate limit (on bitbucket and github for instance) is reached too quickly otherwise.

Jenkins job DLS/gitlab-builds #416 succeeded in 5 min 11 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

added 3 commits

• 4a9c0dbd...88a715d0 - 2 commits from branch swh/devel:master
• 1a038fa4 - save_bulk: Speedup listing process with multi-threading

Compare with previous version

resolved all threads

Jenkins job DLS/gitlab-builds #421 succeeded in 3 min 57 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

added 1 commit

• 05fef4e2 - save_bulk: Speedup listing process with multi-threading

Compare with previous version

resolved all threads

added 1 commit

• 227cd056 - save_bulk: Speedup listing process with multi-threading

Compare with previous version

Jenkins job DLS/gitlab-builds #422 succeeded in 5 min 6 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

Jenkins job DLS/gitlab-builds #423 succeeded in 3 min 57 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

added 1 commit

• a8871bd4 - save_bulk: Speedup listing process with multi-threading

Compare with previous version

resolved all threads

Jenkins job DLS/gitlab-builds #424 succeeded in 3 min 54 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

approved this merge request

resolved all threads

merged