Skip to content

lauchpad: Manage unhandled exception when reading page of result

Antoine R. Dumont requested to merge generated-differential-D7195-source into generated-differential-D7195-target

Prior to this commit, the listing could fail when reading the page result in lauchpad. This now traps the exception and let the listing continue. In effect, this could mean we lose some part of data when listing incrementally.

We do need the full listing to happen regularly anyway. So that sounds like a fair trade-off.

This also allows the listing to finish in case of those issues happening.

Related to swh-loader-bzr#3945 (closed) Depends on !412 (closed)

Test Plan

tox (existing tests are happy, i don't see clearly how to simulate that error in tests though so as is sounds good to me)

docker

Without this commit, this can happen:

swh-lister_1                        | [2022-02-17 09:39:28,685: INFO/MainProcess] Task swh.lister.launchpad.tasks.IncrementalLaunchpadLister[51954eb9-5c55-467e-9091-729361ab4ef8] received
swh-lister_1                        | [2022-02-17 09:45:31,974: ERROR/ForkPoolWorker-1] Task swh.lister.launchpad.tasks.IncrementalLaunchpadLister[51954eb9-5c55-467e-9091-729361ab4ef8] raised unexpected: RestfulError()
swh-lister_1                        | Traceback (most recent call last):
swh-lister_1                        |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/celery/app/trace.py", line 450, in trace_task
swh-lister_1                        |     R = retval = fun(*args, **kwargs)
swh-lister_1                        |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/scheduler/task.py", line 55, in __call__
swh-lister_1                        |     result = super().__call__(*args, **kwargs)
swh-lister_1                        |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/celery/app/trace.py", line 731, in __protected_call__
swh-lister_1                        |     return self.run(*args, **kwargs)
swh-lister_1                        |   File "/src/swh-lister/swh/lister/launchpad/tasks.py", line 27, in list_launchpad_incremental
swh-lister_1                        |     return lister.run().dict()
swh-lister_1                        |   File "/src/swh-lister/swh/lister/pattern.py", line 130, in run
swh-lister_1                        |     full_stats.origins += self.send_origins(origins)
swh-lister_1                        |   File "/src/swh-lister/swh/lister/pattern.py", line 233, in send_origins
swh-lister_1                        |     for batch_origins in grouper(origins, n=1000):
swh-lister_1                        |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/utils.py", line 47, in grouper
swh-lister_1                        |     for _data in itertools.zip_longest(*args, fillvalue=stop_value):
swh-lister_1                        |   File "/src/swh-lister/swh/lister/launchpad/lister.py", line 149, in get_origins_from_page
swh-lister_1                        |     for repo in repos:
swh-lister_1                        |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/lazr/restfulclient/resource.py", line 819, in __iter__
swh-lister_1                        |     next_get = self._root._browser.get(URI(next_link))
swh-lister_1                        |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/lazr/restfulclient/_browser.py", line 439, in get
swh-lister_1                        |     response, content = self._request(url, extra_headers=headers)
swh-lister_1                        |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/lazr/restfulclient/_browser.py", line 429, in _request
swh-lister_1                        |     raise error
swh-lister_1                        | lazr.restfulclient.errors.RestfulError

Migrated from D7195 (view on Phabricator)

Merge request reports