Ingest Packagist

• Develop
• ...
• Deploy in staging
• swh/infra/sysadm-environment#4995 (closed): Deploy packagist in production
• loop: check, fix, deploy
  • swh/devel/swh-lister!492 (merged): Restore regular batch origins record to scheduler in the pattern lister
  • swh/devel/swh-lister!493 (merged): swh.lister.pattern: Make batch size parametric & ensure non-regression
  • swh/infra/ci-cd/swh-charts!106 (merged): Deploy lister v5.9.2
  • swh/devel/swh-lister!494 (merged): Adapt listing to use metadata urls (with fallback)
  • swh/infra/ci-cd/swh-charts!107 (merged): Deploy lister v5.9.3
  • swh/devel/swh-lister!495 (merged): Fix format json parsing (for new on page parsed)
  • swh/infra/ci-cd/swh-charts!108 (merged): Deploy lister v5.9.4
  • swh/devel/swh-lister!496 (merged): Make lister resilient & write origins during listing
  • swh/infra/ci-cd/swh-charts!109 (merged): Deploy lister v5.9.5
  • swh/infra/ci-cd/swh-charts!110 (merged): Deploy swh.lister v5.9.6
  • swh/devel/swh-core!362 (merged): github.utils: Add max attempts & retry requests on connection errors
  • swh/infra/ci-cd/swh-charts!111 (merged): Deploy swh.core v2.24
  • Listing ok

changed milestone to %Extend archive coverage [Roadmap - Collect]

marked this issue as related to #4670 (closed)

added #5004 (closed) as child task

added #5005 (closed) as child task

added #5006 (closed) as child task

added ll-status::in-progress label

added ll-status::staging label and removed ll-status::in-progress label

added ll-status::ready-to-deploy label and removed ll-status::staging label

mentioned in issue swh/infra/sysadm-environment#4995 (closed)

changed the description

added ll::status::deployed label and removed ll-status::ready-to-deploy label

marked the checklist item swh/infra/sysadm-environment#4995 (closed): Deploy packagist in production as completed

changed the description

changed the description

mentioned in merge request swh/devel/swh-lister!494 (merged)

mentioned in merge request swh/devel/swh-lister!493 (merged)

mentioned in merge request swh/devel/swh-lister!492 (merged)

mentioned in merge request swh/devel/swh-lister!495 (merged)

mentioned in merge request swh/devel/swh-lister!496 (merged)

changed the description

marked the checklist item (review) swh/devel/swh-lister!496 (merged): Make lister resilient & write origins during listing as completed

changed the description

marked the checklist item ... as completed

marked the checklist item swh/infra/ci-cd/swh-charts!109 (merged): Deploy v5.9.5 as completed

After fighting it for a while it finally accepted to run fully [1] (roughly the same number of origins we had in the first staging listing a while back).

[1]

2023-08-08 09:36:56 softwareheritage-scheduler@belvedere:5432 λ select now(), instance_name, visit_type, count(*) from listed_origins lo inner join listers l on l.id=lo.lister_id where lister_id in (select id from listers where name='Packagist') group by instance_name, visit_type order by count asc;
+-------------------------------+---------------+------------+--------+
|              now              | instance_name | visit_type | count  |
+-------------------------------+---------------+------------+--------+
| 2023-08-08 07:36:57.493844+00 | packagist     | hg         |      9 |
| 2023-08-08 07:36:57.493844+00 | packagist     | svn        |     31 |
| 2023-08-08 07:36:57.493844+00 | packagist     | git        | 213612 |
+-------------------------------+---------------+------------+--------+
(3 rows)

Time: 2954.415 ms (00:02.954)

changed the description

The coverage widget shows steady progress ~100k origins ingested so far [1]

[1] https://archive.softwareheritage.org

Run log records [1]

[1] http://kibana0.internal.softwareheritage.org:5601/app/discover#/?_g=h@0e494b1&_a=h@6c8059a

closed

removed ll::status::deployed label