packagist: Make lister more resilient & actually list origins

4 commits:

• 1. packagist: Randomize the packages list

That avoids listing always the same origins when some issues happened in the previous listing

• 2. packagist: Ensure to continue listing even if github hangs up on us (when requiring canonical urls [1])

Currently the packagist listing takes a lot of time and at some point, github just hangs up and that fails the listing

• 3. packagist: Allow to override the record_batch_size from constructor

This is for cli testing, to not wait the hard-coded 1000 origins which can take some time with packagist

• 4. packagist: Skip package if unable to parse the last update date

Some date are apparently not parseable which break the process. They are not caught and the package is skipped (same behavior than currently when no last date is provided).

• 5. packaging: Yield pages of origins to regularly record origins

This is the most important one for packagist. That allows to flush regularly origins to the scheduler db during listing. So now, instead of one huge page of origins, this yields pages of
records that are flushed along the way (we currently lose everything at each failed attempt and we only got failed attempts so far, hence the bunch of mrs since the beginning of the week).

nth docker run check ongoing (so far so good).

$ swh-doco exec swh-lister swh lister run -l packagist record_batch_size=10
+ cd /home/tony/work/inria/repo/swh/swh-environment/docker
+ docker compose -f docker-compose.yml -f docker-compose.override.yml exec swh-lister swh lister run -l packagist record_batch_size=10
+ ...
 'record_batch_size': 10,
 'scheduler': {'cls': 'remote', 'url': 'http://swh-scheduler:5008/'}}
INFO:swh.lister.pattern:record-batch-size: 10
INFO:swh.lister.pattern:Record valid 10 origins in the scheduler
INFO:swh.lister.pattern:Record valid 10 origins in the scheduler
WARNING:swh.lister.pattern:Skipping invalid origin: git@gitlab.com:lighty/installers.git
WARNING:swh.lister.pattern:Skipping invalid origin:
INFO:swh.lister.pattern:Record valid 8 origins in the scheduler
INFO:swh.lister.pattern:Record valid 10 origins in the scheduler
INFO:swh.lister.pattern:Record valid 10 origins in the scheduler
INFO:swh.lister.pattern:Record valid 10 origins in the scheduler
INFO:swh.lister.pattern:Record valid 10 origins in the scheduler
INFO:swh.lister.pattern:Record valid 10 origins in the scheduler
INFO:swh.lister.pattern:Record valid 10 origins in the scheduler
INFO:swh.lister.pattern:Record valid 10 origins in the scheduler
INFO:swh.lister.pattern:Record valid 10 origins in the scheduler
...

2023-08-03 16:18:08 swh-scheduler@localhost:5433 λ select now(), instance_name, visit_type, count(*) from listed_origins lo inner join listers l on l.id=lo.lister_id where lister_id in (select id from listers where name='Packagist') group by instance_name, visit_type order by count asc;
+-------------------------------+---------------+------------+-------+
|              now              | instance_name | visit_type | count |
+-------------------------------+---------------+------------+-------+
| 2023-08-03 14:18:13.235783+00 | packagist     | hg         |     1 |
| 2023-08-03 14:18:13.235783+00 | packagist     | svn        |     2 |
| 2023-08-03 14:18:13.235783+00 | packagist     | git        | 23516 |
+-------------------------------+---------------+------------+-------+
(3 rows)

Time: 28.171 ms
2023-08-03 16:18:13 swh-scheduler@localhost:5433 λ \watch 60
             Thu 03 Aug 2023 04:18:19 PM CEST (every 60s)

+-------------------------------+---------------+------------+-------+
|              now              | instance_name | visit_type | count |
+-------------------------------+---------------+------------+-------+
| 2023-08-03 14:18:19.755392+00 | packagist     | hg         |     1 |
| 2023-08-03 14:18:19.755392+00 | packagist     | svn        |     2 |
| 2023-08-03 14:18:19.755392+00 | packagist     | git        | 23533 |
+-------------------------------+---------------+------------+-------+
(3 rows)

Time: 17.521 ms

Refs. swh/meta#5001 (closed)

[1] http://kibana0.internal.softwareheritage.org:5601/goto/84b652d3f37e6c1e6b08607557e9ecf1

swh/lister/packagist/lister.py

@@ -6,11 +6,14 @@
 from dataclasses import dataclass
 from datetime import datetime, timezone
 import logging
+from random import shuffle
 from typing import Any, Dict, Iterator, List, Optional
 
 import iso8601
 import requests
+from tenacity import RetryError
 
+from swh.core.utils import grouper
 from swh.scheduler.interface import SchedulerInterface
 from swh.scheduler.model import ListedOrigin
 
@@ -73,6 +76,7 @@ class PackagistLister(Lister[PackagistListerState, PackagistPageType]):
         max_origins_per_page: Optional[int] = None,
         max_pages: Optional[int] = None,
         enable_origins: bool = True,
+        record_batch_size: int = 1000,
     ):
         super().__init__(
             scheduler=scheduler,
@@ -83,6 +87,7 @@ class PackagistLister(Lister[PackagistListerState, PackagistPageType]):
             max_origins_per_page=max_origins_per_page,
             max_pages=max_pages,
             enable_origins=enable_origins,
+            record_batch_size=record_batch_size,
         )
 
         self.session.headers.update({"Accept": "application/json"})
@@ -122,10 +127,11 @@ class PackagistLister(Lister[PackagistListerState, PackagistPageType]):
             return {}
 
     def get_pages(self) -> Iterator[PackagistPageType]:
-        """
-        Yield a single page listing all Packagist projects.
-        """
-        yield self.api_request(self.url)["packageNames"]
+        """Retrieve & randomize unique list of packages into pages of packages."""
+        package_names = self.api_request(self.url)["packageNames"]
+        shuffle(package_names)
+        for page_packages in grouper(package_names, n=self.record_batch_size):
+            yield page_packages
 
     def _get_metadata_from_page(
         self, package_url_format: str, package_name: str
@@ -233,7 +239,10 @@ class PackagistLister(Lister[PackagistListerState, PackagistPageType]):
                 dist_time_str = version_info.get("time", "")
                 if not dist_time_str:
                     continue
-                dist_time = iso8601.parse_date(dist_time_str)
+                try:
+                    dist_time = iso8601.parse_date(dist_time_str)
+                except iso8601.iso8601.ParseError:
+                    continue
                 if last_update is None or dist_time > last_update:
                     last_update = dist_time
 
@@ -245,9 +254,14 @@ class PackagistLister(Lister[PackagistListerState, PackagistPageType]):
                 # Non-github urls will be returned as is, github ones will be canonical
                 # ones
                 assert self.github_session is not None
-                origin_url = (
-                    self.github_session.get_canonical_url(origin_url) or origin_url
-                )
+                try:
+                    origin_url = (
+                        self.github_session.get_canonical_url(origin_url) or origin_url
+                    )
+                except (requests.exceptions.ConnectionError, RetryError):
+                    # server hangs up, let's ignore it for now
+                    # that might not happen later on
+                    continue
 
             # bitbucket closed its mercurial hosting service, those origins can not be
             # loaded into the archive anymore