launchpad: Ignore erratic page and continue listing

In case of all retry tryouts failed, finally ignore that page and continue listing the next pages.

Related to #3948 (closed)

Test Plan

tox

Applied on staging and triggered the listing. That actually allowed the listing to finish properly.

Feb 17 17:40:37 worker1 python3[2168786]: [2022-02-17 17:40:37,571: INFO/MainProcess] Received task: swh.lister.gitlab.tasks.IncrementalGitLabLister[496b82a5-6e7a-4e39-841c-e9158334fc1e]
Feb 17 17:40:37 worker1 python3[2168796]: [2022-02-17 17:40:37,796: INFO/ForkPoolWorker-4] Task swh.lister.gitlab.tasks.IncrementalGitLabLister[496b82a5-6e7a-4e39-841c-e9158334fc1e] succeeded in 0.19229740649461746s: {'pages': 1, 'origins': 0}

It finished:

Feb 17 20:36:17 worker1 python3[2168793]: [2022-02-17 20:36:17,861: INFO/ForkPoolWorker-1] Task swh.lister.launchpad.tasks.FullLaunchpadLister[d8bb1c71-4c2c-406c-9431-f358cdca5ec5] succeeded in 10058.709554322995s: {'pages': 2, 'origins':
216789}

---

Migrated from D7198 (view on Phabricator)

swh/lister/launchpad/lister.py

@@ -12,6 +12,7 @@ import iso8601
 from launchpadlib.launchpad import Launchpad
 from lazr.restfulclient.errors import RestfulError
 from lazr.restfulclient.resource import Collection
+from tenacity.before_sleep import before_sleep_log
 
 from swh.lister.utils import retry_if_exception, throttling_retry
 from swh.scheduler.interface import SchedulerInterface
@@ -99,7 +100,10 @@ class LaunchpadLister(Lister[LaunchpadListerState, LaunchpadPageType]):
                     d[attribute_name] = date_last_modified.isoformat()
         return d
 
-    @throttling_retry(retry=retry_if_restful_error)
+    @throttling_retry(
+        retry=retry_if_restful_error,
+        before_sleep=before_sleep_log(logger, logging.WARNING),
+    )
     def _page_request(
         self, launchpad, vcs_type: str, date_last_modified: Optional[datetime]
     ) -> Optional[Collection]:
@@ -141,7 +145,6 @@ class LaunchpadLister(Lister[LaunchpadListerState, LaunchpadPageType]):
                 continue
             yield vcs_type, result
 
-    @throttling_retry(retry=retry_if_restful_error)
     def get_origins_from_page(self, page: LaunchpadPageType) -> Iterator[ListedOrigin]:
         """
         Iterate on all git repositories and yield ListedOrigin instances.
@@ -150,30 +153,33 @@ class LaunchpadLister(Lister[LaunchpadListerState, LaunchpadPageType]):
 
         vcs_type, repos = page
 
-        for repo in repos:
-            origin_url = origin(vcs_type, repo)
+        try:
+            for repo in repos:
+                origin_url = origin(vcs_type, repo)
 
-            # filter out origins with invalid URL
-            if not origin_url.startswith("https://"):
-                continue
+                # filter out origins with invalid URL
+                if not origin_url.startswith("https://"):
+                    continue
 
-            last_update = repo.date_last_modified
+                last_update = repo.date_last_modified
 
-            self.date_last_modified[vcs_type] = last_update
+                self.date_last_modified[vcs_type] = last_update
 
-            logger.debug(
-                "Found origin %s with type %s last updated on %s",
-                origin_url,
-                vcs_type,
-                last_update,
-            )
+                logger.debug(
+                    "Found origin %s with type %s last updated on %s",
+                    origin_url,
+                    vcs_type,
+                    last_update,
+                )
 
-            yield ListedOrigin(
-                lister_id=self.lister_obj.id,
-                visit_type=vcs_type,
-                url=origin_url,
-                last_update=last_update,
-            )
+                yield ListedOrigin(
+                    lister_id=self.lister_obj.id,
+                    visit_type=vcs_type,
+                    url=origin_url,
+                    last_update=last_update,
+                )
+        except RestfulError as e:
+            logger.warning("Listing %s origins raised %s", vcs_type, e)
 
     def finalize(self) -> None:
         git_date_last_modified = self.date_last_modified["git"]

[Antoine R. Dumont]

I don't know how to make tests go through that path.