Skip to content

staging: Deploy nixguix lister (and Content|Directory loader)

Then trigger the scheduling of the Guix manifest.

  • Tag and push
    • swh.loader.core v5.1 (includes {Directory|Content}Loader)
    • swh.core v2.17 (includes war archive support)
    • swh.lister v4.1 (includes latest nixguix lister changes)
  • wait for those to show up in pypi.org [1] [2] [3]
  • swh-apps: Build and release new images
    • lister
    • loader_package
  • swh-charts: Update charts to reference the new
    • lister image version
    • nixguix lister using that ^
    • loader_package image version
    • DirectoryLoader (and associated queue)
    • ContentLoader (and associated queue)
  • scheduler: Update swh.lister and swh.loader.core package
  • scheduler: Register new tasks ^ [4]
  • Schedule guix manifest to list [6]
  • Check listing is ok
    • scheduler picked it up and sent it to queue [7]
    • pod started listing
    • ongoing listing [8]
  • Check loading is ok
    • content [9]
    • directory [10]
  • Decomission loader-nixguix
    • Drop chart declaration
    • Disable scheduler task [5]

[1] https://pypi.org/project/swh.lister/

[2] https://pypi.org/project/swh.core/

[3] https://pypi.org/project/swh.loader.core/

[4]

(failed reverse-i-search)`task -type': /usr/bin/swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml ^Csk -h | grep schedule_origins
swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register
...
INFO:swh.scheduler.cli.task_type:Create task type load-content in scheduler
INFO:swh.scheduler.cli.task_type:Create task type load-directory in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-nixguix in scheduler
...  # other task types irrelevant to this task but registered nonetheless (useful for later)
INFO:swh.scheduler.cli.task_type:Create task type load-conda in scheduler
INFO:swh.scheduler.cli.task_type:Create task type load-cpan in scheduler
INFO:swh.scheduler.cli.task_type:Create task type load-puppet in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-conda in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-cpan in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-hackage in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-nuget in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-puppet in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-rubygems in scheduler
...

Note: output filtered ^ to focus on what's important here

[5]

16:49:09 swh-scheduler@db1:5432=> update task set status='disabled' where type ='load-nixguix';
UPDATE 1
Time: 697.848 ms

[6]

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ task add list-nixguix url=https://guix.gnu.org/sources.json origin_upstream='https://git.savannah.gnu.org/cgit/guix.git/'
Created 1 tasks

Task 33419816
  Next run: today (2022-10-26T14:53:17.085262+00:00)
  Interval: 1 day, 0:00:00
  Type: list-nixguix
  Policy: recurring
  Args:
  Keyword args:
    origin_upstream: 'https://git.savannah.gnu.org/cgit/guix.git/'
    url: 'https://guix.gnu.org/sources.json'

[7]

root@scheduler0:~# journalctl -xef -u swh-scheduler-runner | grep -C1 list-nixguix
░░ The job identifier is 3555866.
Oct 26 14:53:24 scheduler0 swh[1111446]: INFO:swh.scheduler.celery_backend.runner:Grabbed 1 tasks list-nixguix
Oct 26 14:53:25 scheduler0 swh[1111446]: INFO:swh.scheduler.cli.admin.runner:Scheduled 1 tasks

[8] Configuration could be improved to use tokens (not using the anonymous mode)

listers [2022-10-26 14:55:15,625: INFO/MainProcess] Task swh.lister.nixguix.tasks.NixGuixListerTask[b7702f7b-96fe-4c8a-89c6-8ba7eb78a0d2] received
listers [2022-10-26 14:55:17,793: WARNING/ForkPoolWorker-1] No tokens set in configuration, using anonymous mode
listers [2022-10-26 14:55:21,422: WARNING/ForkPoolWorker-1] Cannot detect extension for <https://crates.io/api/v1/crates/error-chain/0.8.1/download>. Fallback to http head query
listers [2022-10-26 14:55:22,967: WARNING/ForkPoolWorker-1] Cannot detect extension for <https://crates.io/api/v1/crates/ordered-float/1.0.2/download>. Fallback to http head query
listers [2022-10-26 14:55:24,482: WARNING/ForkPoolWorker-1] Cannot detect extension for <https://crates.io/api/v1/crates/time-macros/0.2.3/download>. Fallback to http head query
...

[8]

17:00:47 swh-scheduler@db1:5432=> select now(), count(*) from listed_origins where lister_id = ( select id from listers where name='nixguix') ;
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-10-26 15:00:52.052646+00 |   170 |
+-------------------------------+-------+
(1 row)

Time: 13.391 ms

[9]

loaders [2022-10-26 15:41:00,310: INFO/MainProcess] Task swh.loader.core.tasks.LoadContent[24fe08ac-1210-46c5-ad0e-1a73a69d14c0] received
loaders [2022-10-26 15:41:01,036: INFO/ForkPoolWorker-1] Load origin 'http://download.tuxfamily.org/user42/ffap-rfc-space.el' with type 'content'
loaders [2022-10-26 15:41:01,668: INFO/ForkPoolWorker-1] Task swh.loader.core.tasks.LoadContent[24fe08ac-1210-46c5-ad0e-1a73a69d14c0] succeeded in 1.1599025342147797s: {'status': 'eventful'}

[10]

loaders [2022-10-26 15:41:35,928: INFO/MainProcess] Task swh.loader.core.tasks.LoadDirectory[112ff40f-f14e-4135-a8f3-a72247199f43] received
loaders [2022-10-26 15:41:36,674: INFO/ForkPoolWorker-1] Load origin 'https://files.pythonhosted.org/packages/source/m/mysqlclient/mysqlclient-2.0.1.tar.gz' with type 'directory'
loaders [2022-10-26 15:41:37,852: INFO/ForkPoolWorker-1] Task swh.loader.core.tasks.LoadDirectory[4729fa18-f0ae-4f1a-bab5-e9270fab2e1e] succeeded in 1.8904169800225645s: {'status': 'eventful'}
loaders [2022-10-26 15:41:37,858: INFO/MainProcess] Task swh.loader.core.tasks.LoadDirectory[6494d027-ee82-4ec6-b3ad-83869741852f] received
loaders [2022-10-26 15:41:38,188: INFO/ForkPoolWorker-1] Load origin 'http://cran.r-project.org/src/contrib/gtools_3.9.3.tar.gz' with type 'directory'
loaders [2022-10-26 15:42:03,619: INFO/ForkPoolWorker-1] Task swh.loader.core.tasks.LoadDirectory[112ff40f-f14e-4135-a8f3-a72247199f43] succeeded in 25.76121896598488s: {'status': 'eventful'}
loaders [2022-10-26 15:42:03,631: INFO/MainProcess] Task swh.loader.core.tasks.LoadDirectory[143a9816-6f87-415e-919f-c7390345c755] received
loaders [2022-10-26 15:42:04,018: INFO/ForkPoolWorker-1] Load origin 'https://static.crates.io/crates/error-chain/error-chain-0.8.1.crate' with type 'directory'
loaders [2022-10-26 15:42:09,071: INFO/ForkPoolWorker-1] Task swh.loader.core.tasks.LoadDirectory[6494d027-ee82-4ec6-b3ad-83869741852f] succeeded in 5.445178766967729s: {'status': 'eventful'}
loaders [2022-10-26 15:42:09,084: INFO/MainProcess] Task swh.loader.core.tasks.LoadDirectory[6a971088-f129-444b-8776-9a392dcb8d8c] received
loaders [2022-10-26 15:42:09,425: INFO/ForkPoolWorker-1] Load origin 'https://archive.apache.org/dist/maven/shared/maven-filtering-3.1.1-source-release.zip' with type 'directory'
Edited by Antoine R. Dumont