Skip to content

package/utils: Add retry policy to download in case of throttling

Antoine Lambert requested to merge generated-differential-D8390-source into generated-differential-D8390-target

Some HTTP download requests might be throttled by remote servers so add retry mechanism with exponential backoff to fix tarball downloads in some loaders.

I observed that kind of issues with the AUR loader when testing it in docker, see below:

docker-swh-loader-1  | [2022-09-05 13:38:26,727: ERROR/ForkPoolWorker-74] Failed to load branch releases/1.79-1/nvi.tar.gz for https://aur.archlinux.org/packages/nvi
docker-swh-loader-1  | Traceback (most recent call last):
docker-swh-loader-1  |   File "/src/swh-loader-core/swh/loader/package/loader.py", line 649, in load
docker-swh-loader-1  |     res = self._load_release(p_info, origin)
docker-swh-loader-1  |   File "/src/swh-loader-core/swh/loader/package/loader.py", line 822, in _load_release
docker-swh-loader-1  |     dl_artifacts = self.download_package(p_info, tmpdir)
docker-swh-loader-1  |   File "/src/swh-loader-core/swh/loader/package/loader.py", line 399, in download_package
docker-swh-loader-1  |     return [download(p_info.url, dest=tmpdir, filename=p_info.filename)]
docker-swh-loader-1  |   File "/src/swh-loader-core/swh/loader/package/utils.py", line 109, in download
docker-swh-loader-1  | ValueError: Fail to query 'https://aur.archlinux.org/cgit/aur.git/snapshot/nvi.tar.gz'. Reason: 429
docker-swh-loader-1  | [2022-09-05 13:38:26,728: DEBUG/ForkPoolWorker-74] default version: 1.79-1
docker-swh-loader-1  | [2022-09-05 13:38:26,728: DEBUG/ForkPoolWorker-74] extra branches: {}
docker-swh-loader-1  | [2022-09-05 13:38:26,728: DEBUG/ForkPoolWorker-74] releases: {'1.79-1': []}
docker-swh-loader-1  | [2022-09-05 13:38:26,728: DEBUG/ForkPoolWorker-74] snapshot: {'branches': {}}
docker-swh-loader-1  | [2022-09-05 13:38:26,729: DEBUG/ForkPoolWorker-74] snapshot: Snapshot(branches=ImmutableDict({}), id=hash_to_bytes('1a8893e6a86f444e8be8e7bda6cb34fb1735a00e'))
docker-swh-loader-1  | [2022-09-05 13:38:26,729: DEBUG/ForkPoolWorker-74] Flushing 1 objects of type snapshot
docker-swh-loader-1  | [2022-09-05 13:38:26,823: WARNING/ForkPoolWorker-74] 1 failed branches
docker-swh-loader-1  | [2022-09-05 13:38:26,823: WARNING/ForkPoolWorker-74] Failed branches: releases/1.79-1/nvi.tar.gz
docker-swh-loader-1  | [2022-09-05 13:38:26,827: INFO/ForkPoolWorker-74] Task swh.loader.package.aur.tasks.LoadAur[8fafddd4-9005-4609-a385-c02e546ce52e] succeeded in 0.3548696079997171s: {'status': 'uneventful', 'snapshot_id': '1a8893e6a86f444e8be8e7bda6cb34fb1735a00e'}
docker-swh-loader-1  | [2022-09-05 13:38:26,847: DEBUG/ForkPoolWorker-74] Loading config file /loader.yml
docker-swh-loader-1  | [2022-09-05 13:38:26,852: INFO/MainProcess] Task swh.loader.git.tasks.UpdateGitRepository[1b7cae47-9cc1-42e8-b5e8-84231efce0e8] received
docker-swh-loader-1  | [2022-09-05 13:38:26,915: DEBUG/ForkPoolWorker-74] last snapshot: None
docker-swh-loader-1  | [2022-09-05 13:38:26,916: DEBUG/ForkPoolWorker-74] package_info: AurPackageInfo(url='https://aur.archlinux.org/cgit/aur.git/snapshot/libomxil-component-xvideo.tar.gz', filename='libomxil-component-xvideo.tar.gz', directory_extrinsic_metadata=[], name='libomxil-component-xvideo', version='0.1-1', last_modified='2015-11-27T07:01:23+00:00')
docker-swh-loader-1  | [2022-09-05 13:38:27,068: ERROR/ForkPoolWorker-74] Failed to load branch releases/0.1-1/libomxil-component-xvideo.tar.gz for https://aur.archlinux.org/packages/libomxil-component-xvideo
docker-swh-loader-1  | Traceback (most recent call last):
docker-swh-loader-1  |   File "/src/swh-loader-core/swh/loader/package/loader.py", line 649, in load
docker-swh-loader-1  |     res = self._load_release(p_info, origin)
docker-swh-loader-1  |   File "/src/swh-loader-core/swh/loader/package/loader.py", line 822, in _load_release
docker-swh-loader-1  |     dl_artifacts = self.download_package(p_info, tmpdir)
docker-swh-loader-1  |   File "/src/swh-loader-core/swh/loader/package/loader.py", line 399, in download_package
docker-swh-loader-1  |     return [download(p_info.url, dest=tmpdir, filename=p_info.filename)]
docker-swh-loader-1  |   File "/src/swh-loader-core/swh/loader/package/utils.py", line 109, in download
docker-swh-loader-1  | ValueError: Fail to query 'https://aur.archlinux.org/cgit/aur.git/snapshot/libomxil-component-xvideo.tar.gz'. Reason: 429
docker-swh-loader-1  | [2022-09-05 13:38:27,069: DEBUG/ForkPoolWorker-74] default version: 0.1-1
docker-swh-loader-1  | [2022-09-05 13:38:27,069: DEBUG/ForkPoolWorker-74] extra branches: {}
docker-swh-loader-1  | [2022-09-05 13:38:27,069: DEBUG/ForkPoolWorker-74] releases: {'0.1-1': []}
docker-swh-loader-1  | [2022-09-05 13:38:27,069: DEBUG/ForkPoolWorker-74] snapshot: {'branches': {}}
docker-swh-loader-1  | [2022-09-05 13:38:27,069: DEBUG/ForkPoolWorker-74] snapshot: Snapshot(branches=ImmutableDict({}), id=hash_to_bytes('1a8893e6a86f444e8be8e7bda6cb34fb1735a00e'))
docker-swh-loader-1  | [2022-09-05 13:38:27,069: DEBUG/ForkPoolWorker-74] Flushing 1 objects of type snapshot
docker-swh-loader-1  | [2022-09-05 13:38:27,116: WARNING/ForkPoolWorker-74] 1 failed branches
docker-swh-loader-1  | [2022-09-05 13:38:27,116: WARNING/ForkPoolWorker-74] Failed branches: releases/0.1-1/libomxil-component-xvideo.tar.gz
docker-swh-loader-1  | [2022-09-05 13:38:27,118: INFO/ForkPoolWorker-74] Task swh.loader.package.aur.tasks.LoadAur[5ce421cd-1610-4554-b0a4-26b4791145d3] succeeded in 0.2711032420011179s: {'status': 'uneventful', 'snapshot_id': '1a8893e6a86f444e8be8e7bda6cb34fb1735a00e'}

Migrated from D8390 (view on Phabricator)

Merge request reports