staging: Deploy latest nixguix related loaders and lister

Includes:

• swh/devel/swh-loader-core!459 (closed): Nar computations is now done in python (no longer need for the nix binary now)
• swh/devel/swh-loader-core!463 (closed): Content|Directory loaders now store the nar checksums as extids
• swh/devel/swh-lister!466 (merged): checksum_layout field renaming
• swh/devel/swh-loader-core!464 (merged): Impacted checksum_layout field renaming ^ from lister in the loaders

Plan:

• Depends on swh/devel/swh-lister!466 (merged) + swh/devel/swh-loader-core!464 (merged) being merged first
• Release swh.lister v5.2.0
• (ongoing) Release swh.loader.core v5.3.0 [1]
• swh-apps: Drop nix binary in the dockerfile [2]
• swh-apps: Build images ^
• swh-charts: Update values.yaml
• Deploy
• Trigger listing
• Checks
• swh/devel/swh-lister!467 (merged): Fix lister
• Redeploy lister
• Trigger listing again
• Listing
• (ongoing) Checks
  • (fail) swh-scheduler: listed origins with "checksum_layout" of type "nar" exist [3]
  • the lister works as before but it does not ingest any "nar" yet due to changes that need to happen
  • Discussion to clarify what needs to happen (starting point [4])
  • Clarified
• Ingestion
• Checks
  • swh: extid of type "nar-%" got written in the archive

[1] unstable debian build is ko but the pypi release got done

[1'] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DLDBASE/job/gbp-buildpackage/298/console

[2] It's nowhere to be found so noop

[3]

select * from listed_origins where lister_id=(select id from listers where name='nixguix' and instance_name='guix.gnu.org') and visit_type in ('content', 'directory') and (extra_loader_arguments#>>'{"checksums_computation"}'<>'standard' and extra_loader_arguments#>>'{"checksum_layout"}'<>'standard');

[4] #4868 (comment 137800)

Refs. swh/meta#4979 (closed)

changed the description

marked the checklist item Depends on swh/devel/swh-lister!466 (merged) + swh/devel/swh-loader-core!464 (merged) being merged first as completed

changed the description

changed the description

changed milestone to %Extend archive coverage [Roadmap - Collect]

added activity::Deployment label

marked the checklist item swh-apps: Build images ^ as completed

changed the description

mentioned in commit swh/infra/ci-cd/swh-charts@14c34fe0

mentioned in commit swh/infra/ci-cd/swh-charts@594c41f7

marked the checklist item swh-charts: Update values.yaml as completed

marked the checklist item Deploy as completed

changed the description

marked the checklist item Trigger listing as completed

Scheduling content and directory [1] listed from the nixguix listing [0]. And now content loader is happy [2]. So is the directory loader [3]

[0]

┌────────────────────────────────────────────────────────────────────────────────────────── Pods(all)[5] </lister|loader-content|loader-directory> ───────────────────────────────────────────────────────────────────────────────────────────┐
│ NAMESPACE     NAME                                   PF     READY         RESTARTS STATUS             CPU      MEM      %CPU/R      %CPU/L      %MEM/R      %MEM/L↓ IP                  NODE                                    AGE         │
│ swh           loader-directory-67f6765fbc-9w64z      ●      1/1                  0 Running             91      129          45         n/a          50          n/a 10.42.240.97        rancher-node-staging-rke2-worker4       3m8s        │
│ swh           loader-content-bdfc9d575-fs5rs         ●      1/1                  0 Running              1      106           0         n/a          41          n/a 10.42.233.187       rancher-node-staging-rke2-worker2       7m8s        │
│ swh           lister-nixguix-7cd8476564-924cc        ●      1/1                  0 Running             25      151          10         n/a          59          n/a 10.42.233.147       rancher-node-staging-rke2-worker2       27m         │

[1]

swhscheduler@scheduler0:~$ swh scheduler -C /etc/softwareheritage/scheduler/listener-runner.yml origin send-to-celery content
10000 slots available in celery queue
2334 visits to send to celery
swhscheduler@scheduler0:~$ swh scheduler -C /etc/softwareheritage/scheduler/listener-runner.yml origin send-to-celery directory
10000 slots available in celery queue
10000 visits to send to celery

content loader is happy [2]

[2]

loaders [2023-04-27 08:53:56,121: INFO/ForkPoolWorker-22] Task swh.loader.core.tasks.LoadContent[5b22173f-910e-457a-b15f-187f786a0dfc] succeeded in 0.44345839880406857s: {'status': 'eventful'}
loaders [2023-04-27 08:53:56,124: INFO/MainProcess] Task swh.loader.core.tasks.LoadContent[06162689-3062-47f8-974f-8d3e4b98b02d] received
loaders [2023-04-27 08:53:56,224: INFO/ForkPoolWorker-22] Load origin 'https://unpkg.com/bokehjs@0.12.15/build/js/bokeh.js' with type 'content'
loaders [2023-04-27 08:53:56,489: INFO/ForkPoolWorker-22] Task swh.loader.core.tasks.LoadContent[c82e99f0-660c-41bf-a09d-e5328fb813f5] succeeded in 0.3651189748197794s: {'status': 'eventful'}
loaders [2023-04-27 08:53:56,492: INFO/MainProcess] Task swh.loader.core.tasks.LoadContent[6bbc3f2c-473f-4066-83dd-93b36fdb8e86] received
loaders [2023-04-27 08:53:56,585: INFO/ForkPoolWorker-22] Load origin 'https://letsencrypt.org/certs/lets-encrypt-r4.pem' with type 'content'
loaders [2023-04-27 08:53:56,963: INFO/ForkPoolWorker-22] Task swh.loader.core.tasks.LoadContent[06162689-3062-47f8-974f-8d3e4b98b02d] succeeded in 0.4718460626900196s: {'status': 'eventful'}
loaders [2023-04-27 08:53:56,966: INFO/MainProcess] Task swh.loader.core.tasks.LoadContent[8ab72ae4-7177-4124-8b7c-780a51daedaf] received
loaders [2023-04-27 08:53:57,099: INFO/ForkPoolWorker-22] Load origin 'https://raw.githubusercontent.com/jspreadsheet/ce/8af1960f76e6803bebc5750013d2ebe95354e88a/dist/jexcel.js' with type 'content'
loaders [2023-04-27 08:53:57,591: INFO/ForkPoolWorker-22] Task swh.loader.core.tasks.LoadContent[6bbc3f2c-473f-4066-83dd-93b36fdb8e86] succeeded in 0.6247061230242252s: {'status': 'eventful'}
loaders [2023-04-27 08:53:57,594: INFO/MainProcess] Task swh.loader.core.tasks.LoadContent[4912703f-6272-4fb0-b2f9-fd22e008b5d6] received
loaders [2023-04-27 08:53:57,705: INFO/ForkPoolWorker-22] Load origin 'https://cdn.datatables.net/fixedcolumns/4.1.0/js/fixedColumns.jqueryui.js' with type 'content'
loaders [2023-04-27 08:53:58,185: INFO/ForkPoolWorker-22] Task swh.loader.core.tasks.LoadContent[8ab72ae4-7177-4124-8b7c-780a51daedaf] succeeded in 0.5911702439188957s: {'status': 'eventful'}

[3]

loaders [2023-04-27 08:54:22,388: INFO/MainProcess] Task swh.loader.core.tasks.LoadDirectory[14fee1e0-85b0-44f0-9710-cf3d41589e72] received
loaders [2023-04-27 08:54:22,520: INFO/ForkPoolWorker-1] Load origin 'http://cran.r-project.org/src/contrib/hms_1.1.2.tar.gz' with type 'directory'
loaders [2023-04-27 08:54:28,396: INFO/ForkPoolWorker-1] Task swh.loader.core.tasks.LoadDirectory[ffe72d69-03fb-4a16-a459-c4af2f531474] succeeded in 6.00879945512861s: {'status': 'eventful'}
loaders [2023-04-27 08:54:28,399: INFO/MainProcess] Task swh.loader.core.tasks.LoadDirectory[a2ec64df-9397-4009-ac89-19bac1647c82] received
loaders [2023-04-27 08:54:28,556: INFO/ForkPoolWorker-1] Load origin 'https://hackage.haskell.org/package/clientsession/clientsession-0.9.1.2.tar.gz' with type 'directory'
loaders [2023-04-27 08:54:29,237: INFO/ForkPoolWorker-1] Task swh.loader.core.tasks.LoadDirectory[14fee1e0-85b0-44f0-9710-cf3d41589e72] succeeded in 0.8383421106263995s: {'status': 'eventful'}
loaders [2023-04-27 08:54:29,240: INFO/MainProcess] Task swh.loader.core.tasks.LoadDirectory[8783c0a9-9685-47d1-9d8a-00dbce86879c] received
loaders [2023-04-27 08:54:29,396: INFO/ForkPoolWorker-1] Load origin 'http://cran.r-project.org/src/contrib/RhpcBLASctl_0.21-247.1.tar.gz' with type 'directory'
loaders [2023-04-27 08:54:33,514: INFO/ForkPoolWorker-1] Task swh.loader.core.tasks.LoadDirectory[a2ec64df-9397-4009-ac89-19bac1647c82] succeeded in 4.274567719548941s: {'status': 'eventful'}
loaders [2023-04-27 08:54:33,517: INFO/MainProcess] Task swh.loader.core.tasks.LoadDirectory[9964f7cd-facc-44bc-be7f-2858624ea210] received
loaders [2023-04-27 08:54:33,620: INFO/ForkPoolWorker-1] Load origin 'https://static.crates.io/crates/askama_escape/askama_escape-0.10.3.crate' with type 'directory'
loaders [2023-04-27 08:54:34,588: INFO/ForkPoolWorker-1] Task swh.loader.core.tasks.LoadDirectory[8783c0a9-9685-47d1-9d8a-00dbce86879c] succeeded in 1.0708410954102874s: {'status': 'eventful'}
loaders [2023-04-27 08:54:34,590: INFO/MainProcess] Task swh.loader.core.tasks.LoadDirectory[5c3905f2-f46f-42d8-b236-d8c4fa977d60] received
loaders [2023-04-27 08:54:34,944: INFO/ForkPoolWorker-1] Load origin 'http://cran.r-project.org/src/contrib/coda_0.19-4.tar.gz' with type 'directory'
loaders [2023-04-27 08:54:35,960: INFO/ForkPoolWorker-1] Task swh.loader.core.tasks.LoadDirectory[9964f7cd-facc-44bc-be7f-2858624ea210] succeeded in 1.3694131933152676s: {'status': 'eventful'}
loaders [2023-04-27 08:54:36,228: INFO/MainProcess] Task swh.loader.core.tasks.LoadDirectory[1bc54397-c1f3-4a7c-82e0-c05188743459] received
loaders [2023-04-27 08:54:36,539: INFO/ForkPoolWorker-2] Load origin 'http://cran.r-project.org/src/contrib/mitml_0.4-4.tar.gz' with type 'directory'
loaders [2023-04-27 08:54:46,558: INFO/ForkPoolWorker-2] Task swh.loader.core.tasks.LoadDirectory[5c3905f2-f46f-42d8-b236-d8c4fa977d60] succeeded in 10.318829977884889s: {'status': 'eventful'}

For now, there is no new extids yet, so the checks step is still pending.

2023-04-27 11:02:27 swh@db1:5432 λ select now(), count(*) from extid where extid_type like 'nar-%';
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2023-04-27 09:07:37.060205+00 |     0 |
+-------------------------------+-------+
(1 row)

Time: 14826.951 ms (00:14.827)
2023-04-27 11:02:11 swh@db1:5432 λ select now(), extid_type, count(*) from extid group by extid_type;
+-------------------------------+---------------------+----------+
|              now              |     extid_type      |  count   |
+-------------------------------+---------------------+----------+
| 2023-04-27 09:02:20.679245+00 | bzr-nodeid          | 13485841 |
| 2023-04-27 09:02:20.679245+00 | cran-sha256         |    12103 |
| 2023-04-27 09:02:20.679245+00 | dsc-sha256          |    34306 |
| 2023-04-27 09:02:20.679245+00 | hg-nodeid           |  5695377 |
| 2023-04-27 09:02:20.679245+00 | maven-jar           |  5891103 |
| 2023-04-27 09:02:20.679245+00 | npm-archive-sha1    | 17742559 |
| 2023-04-27 09:02:20.679245+00 | npm-manifest-sha256 |  8767120 |
| 2023-04-27 09:02:20.679245+00 | pypi-archive-sha256 |  5339849 |
+-------------------------------+---------------------+----------+
(8 rows)

Time: 6908.806 ms (00:06.909)

changed the description

Mmm, although there will be no new extid if we don't have any listed contents or directories with the checksum_layout 'nar'... Only 'standard' one so far.

2023-04-27 11:26:23 swh-scheduler@db1:5432 λ select * from listed_origins where lister_id=(select id from listers where name='nixguix' and instance_name='guix.gnu.org') and visit_type in ('content', 'directory') and (extra_loader_arguments#>>'{"checksums_computation"}'<>'standard' and extra_loader_arguments#>>'{"checksum_layout"}'<>'standard')
swh-scheduler λ ;
+-----------+-----+------------+------------------------+---------+------------+-----------+-------------+
| lister_id | url | visit_type | extra_loader_arguments | enabled | first_seen | last_seen | last_update |
+-----------+-----+------------+------------------------+---------+------------+-----------+-------------+
+-----------+-----+------------+------------------------+---------+------------+-----------+-------------+
(0 rows)

Time: 65.288 ms