Skip to content
Snippets Groups Projects
New look: Off

Deploy swh.loader.core v2.0.0

    • Closed Issue created by Antoine R. Dumont @ardumont

    Changelog:

    • D6396: new Maven loader
    • D6750: Updated Debian loader to be able to deal with old origins with md5 manifest
    • D6748, D6749, Fix debian hash manifest to use the correct intrinsic version
    • D6751: Bump ExtID version for extid entries dsc-sha256

    Dev/Ops actions:

    • Actually land the related bunch of diffs (see Changelog entry)
    • Tag swh.loader.core
    • build is fine? [1]
    • Actually deploy on workers
    • for platform in {staging, production}
      • Restart impacted services:
        • swh-worker@loader_{$pkg_loader} for pkg_loader in {archive,cran,debian,deosit,nixguix,opam,npm,pypi,...}
        • swh-worker@lister
      • Ensure loading is still ok (staging: #3774 (closed), prod: #3774 (closed))

    We must use built-in now.

    Related to T3770

    Impact:

    • Schedule back origins with visit_type "deb" (ongoing)

    Dependencies pulled:

    • swh.model v3.1 (hashes: allowing optional md5 hash computation)
    • swh.lister v2.6.1 (debian: improved last_update computations)

    Note: v2.0 since it's an api change for the debian loader.

    Migrated from T3774 (view on Phabricator)

    ✓ 10 of 10 checklist items completed · Edited

    Designs

      Child items 0

      2. No child items are currently assigned. Use child items to break down this issue into smaller parts.

      Activity

      • Phabricator Migration user marked this issue as related to #2400 (closed)

        marked this issue as related to #2400 (closed)

      • Antoine R. Dumont added Package Loader System administration priority:Normal labels

        added Package Loader System administration priority:Normal labels

      • Antoine R. Dumont changed the description

        changed the description

      • Antoine R. Dumont changed title from Deploy swh.loader.core v1.3.0 to Deploy swh.loader.core v2.0.0

        changed title from Deploy swh.loader.core v1.3.0 to Deploy swh.loader.core v2.0.0

      • Antoine R. Dumont changed the description

        changed the description

      • Antoine R. Dumont changed the description

        changed the description

      • Antoine R. Dumont marked the checklist item Tag swh.loader.core as completed

        marked the checklist item Tag swh.loader.core as completed

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        staging deployment done ^

      • Antoine R. Dumont added state:wip label

        added state:wip label

      • Antoine R. Dumont marked the checklist item build is fine? [1] as completed

        marked the checklist item build is fine? [1] as completed

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        Triggered some debian loading in there. And everything is working as expected.

        See new snapshot from the api standpoint [1] out of the loading log processes [2]. People can check other snapshot datapoint from there if they so wish.

        We do find the extid from the debian loading process [3]

        ---------------
worker0.internal.staging.swh.network
---------------
Dec 07 16:43:17 worker0 python3[4113153]: [2021-12-07 16:43:17,919: INFO/ForkPoolWorker-1] Task swh.loader.package.debian.tasks.LoadDebian[6ac9d5d1-21ec-4819-9b76-55940bccef99] succeeded in 14.641922675073147s: {'status': 'eventful', 'snap
shot_id': '8ff1baaaee6aa4d266eec37092dcc32fbfa78724'}
Dec 07 16:43:43 worker0 python3[4113153]: [2021-12-07 16:43:43,605: INFO/ForkPoolWorker-1] Task swh.loader.package.debian.tasks.LoadDebian[35f73c99-4634-48c3-9f34-4b4a9b580d1d] succeeded in 25.681323068216443s: {'status': 'eventful', 'snap
shot_id': '0599500306a73f4f09ec304111cd57d455956109'}
Dec 07 16:44:06 worker0 python3[4113153]: [2021-12-07 16:44:06,014: INFO/ForkPoolWorker-1] Task swh.loader.package.debian.tasks.LoadDebian[441e00e5-6a26-4123-913b-0685ed813aef] succeeded in 22.403235021978617s: {'status': 'eventful', 'snap
shot_id': 'b90080be67b420445fbfcc342da3578c1c85255c'}
Dec 07 16:44:27 worker0 python3[4113153]: [2021-12-07 16:44:27,092: INFO/ForkPoolWorker-1] Task swh.loader.package.debian.tasks.LoadDebian[dbfd6704-29f7-4ce6-9f02-332a221ce8c6] succeeded in 21.068266598507762s: {'status': 'eventful', 'snap
shot_id': 'a4804a446f3a0138f8ef3739079f9d31e4279c87'}
Dec 07 16:45:09 worker0 python3[4116027]: [2021-12-07 16:45:09,901: INFO/ForkPoolWorker-2] Task swh.loader.package.debian.tasks.LoadDebian[9f8d2be9-fbf1-47c8-8e83-c53cf0d99639] succeeded in 39.72432394698262s: {'status': 'eventful', 'snaps
hot_id': '3010ddb6495ba3644998abdaf4a18342950920b9'}
---------------
worker1.internal.staging.swh.network
---------------
Dec 07 16:43:03 worker1 python3[2112811]: [2021-12-07 16:43:03,178: INFO/ForkPoolWorker-1] Task swh.loader.package.debian.tasks.LoadDebian[23ad800c-e5e6-4f9e-bddd-b72f0238bccd] succeeded in 14.875466795638204s: {'status': 'eventful', 'snap
shot_id': '2422896a70b58247fd3b6e5f231a2feda9b42e16'}
Dec 07 16:43:17 worker1 python3[2112811]: [2021-12-07 16:43:17,960: INFO/ForkPoolWorker-1] Task swh.loader.package.debian.tasks.LoadDebian[4e461289-347f-48dc-b7a3-e3dca1e6ec89] succeeded in 14.77596678584814s: {'status': 'eventful', 'snaps
hot_id': '59d4c96988c47271f4dcb56a67b8a25b5970e967'}
Dec 07 16:43:40 worker1 python3[2112811]: [2021-12-07 16:43:40,118: INFO/ForkPoolWorker-1] Task swh.loader.package.debian.tasks.LoadDebian[209d90ef-31f0-472c-84de-7668076e34e0] succeeded in 22.15398936904967s: {'status': 'eventful', 'snaps
hot_id': 'df5293e41406e2787795482f1fd69116a5b0d1ad'}
Dec 07 16:44:19 worker1 python3[2112811]: [2021-12-07 16:44:19,018: INFO/ForkPoolWorker-1] Task swh.loader.package.debian.tasks.LoadDebian[f23b393d-f65f-432a-8657-70c2c33a7c01] succeeded in 38.895293049514294s: {'status': 'eventful', 'snap
shot_id': '0d53f57f773bc062e6457860a7f5114ba1034423'}
Dec 07 16:46:00 worker1 python3[2112811]: [2021-12-07 16:46:00,951: INFO/ForkPoolWorker-1] Task swh.loader.package.debian.tasks.LoadDebian[f3f8d73a-aaa1-46fb-bf88-08e49422484f] succeeded in 101.9279799386859s: {'status': 'eventful', 'snaps
hot_id': 'd039e8d13ed9d846ffa34e7946039a9973948ced'}
---------------
worker2.internal.staging.swh.network
---------------
Dec 07 16:44:03 worker2 python3[425021]: [2021-12-07 16:44:03,963: INFO/ForkPoolWorker-1] Task swh.loader.package.debian.tasks.LoadDebian[863a50a2-53d0-4683-9ed5-877808fc85e3] succeeded in 20.466854243015405s: {'status': 'eventful', 'snaps
hot_id': '57258c56c4d36178f9e88e14137d66ca42d7241d'}
Dec 07 16:44:17 worker2 python3[425021]: [2021-12-07 16:44:17,885: INFO/ForkPoolWorker-1] Task swh.loader.package.debian.tasks.LoadDebian[b66d0291-84b4-4669-a5c1-104d2f86a789] succeeded in 13.917391878960188s: {'status': 'eventful', 'snaps
hot_id': '45bbee846a7b0343a64a20c659f5c302677511d8'}
Dec 07 16:44:43 worker2 python3[425021]: [2021-12-07 16:44:43,768: INFO/ForkPoolWorker-1] Task swh.loader.package.debian.tasks.LoadDebian[b21a856c-65ea-4842-880f-575915c54db6] succeeded in 25.878415616985876s: {'status': 'eventful', 'snaps
hot_id': 'e7944a1b2897cd87d1cb3a615f2b4e6955a50ac5'}
Dec 07 16:45:07 worker2 python3[431567]: [2021-12-07 16:45:07,206: INFO/ForkPoolWorker-2] Task swh.loader.package.debian.tasks.LoadDebian[56570006-1a7a-4740-b6e6-c6d279ae7a3b] succeeded in 23.212842805020045s: {'status': 'eventful', 'snaps
hot_id': '61ffdef34dc78eb6a81ab640840aa82a154b0af8'}
Dec 07 16:45:42 worker2 python3[431567]: [2021-12-07 16:45:42,508: INFO/ForkPoolWorker-2] Task swh.loader.package.debian.tasks.LoadDebian[eb85ff05-3864-434d-a5cc-b95c0292d3c5] succeeded in 35.29841585399117s: {'status': 'eventful', 'snapsh
ot_id': 'd132e81b44841f86ee612884631b68959a3ed4ea'}
        • [3]
        17:52:16 swh@db1:5432=> select * from extid where extid_type='dsc-sha256' and extid_version=1 limit 10;
+------------+--------------------------------------------------------------------+-------------+--------------------------------------------+---------------+
| extid_type |                               extid                                | target_type |                   target                   | extid_version |
+------------+--------------------------------------------------------------------+-------------+--------------------------------------------+---------------+
| dsc-sha256 | \x050a22f0a680882727e7c3c2e81a1c893e377fd369dc216112402987ec4f21b7 | release     | \xfb14cb883cec73c0381a2af553dec98829721c2e |             1 |
| dsc-sha256 | \x068e9500ada30813dac1721e71ec4ad4129142b960b7fe19487db1892865c82e | release     | \x45462aaaeb3e0bc7a9f90642b1cf62580ce6d714 |             1 |
| dsc-sha256 | \x088b1e1cc8d27e8482f03e31261e7dcdeb015e081fb6f74a1332d33669cfceba | release     | \x1e15115a728e77a4b03f7586f915332287be0d47 |             1 |
| dsc-sha256 | \x0f30d1d87d5d46e55f39ab51a778a40876c3a02d7c7e29df09d6e883aa0327b8 | release     | \x8cd802ed8932b73e55fdf7a0b5e0f017d5314226 |             1 |
| dsc-sha256 | \x13284df3206b915a8e7e209bb47f6ad6baa4ba41bb67746708dd959336b1b82c | release     | \x25d02fb58201234edb6db0a3c04f76da560d60e1 |             1 |
| dsc-sha256 | \x16a86730ede839da7d5ea60399a676f9538a24d98f90e5b9905de27a0a22db34 | release     | \x5de8d90fd206ae03673b2aca0f685f804570b4d5 |             1 |
| dsc-sha256 | \x1bae852903a71bbaafdb5057651762027aa330f4bde6ed855c0bdfea09436a9a | release     | \xc933b8a4084129eb40b3d9e95360b3d0a831eb67 |             1 |
| dsc-sha256 | \x1bf6664cbe277c0aa903cb32e32a9e8d72a4b10afaa7eeac2abb8ab7647768f1 | release     | \x3ccff68f51a538bc446429cb89fb77a15b38f11b |             1 |
| dsc-sha256 | \x2cdb9da4431c0c2e701c3b9fb1dbc0fd2b239b4c14681efb6153f7998fc0b7b3 | release     | \xc5d370b60bc6e35da5671509051a8c172da3fa14 |             1 |
| dsc-sha256 | \x305326c656a2ce36ff48a89a4a03adefe54eca349f026ac463c2b396639ba9d7 | release     | \x43d9879a3b87d545388328e30e83cc3de3096f1b |             1 |
+------------+--------------------------------------------------------------------+-------------+--------------------------------------------+---------------+
(10 rows)

Time: 6.560 ms
      • Antoine R. Dumont marked the checklist item Actually deploy on workers as completed

        marked the checklist item Actually deploy on workers as completed

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        Same done in production.

        This runs smoothly. [1] [2] [3]

        18:11:37 softwareheritage@belvedere:5432=> select now(), count(*) from extid where extid_type='dsc-sha256' and extid_version=1;
+------------------------------+-------+
|             now              | count |
+------------------------------+-------+
| 2021-12-07 17:11:40.14624+00 |   780 |
+------------------------------+-------+
(1 row)

Time: 145.775 ms
18:11:40 softwareheritage@belvedere:5432=> select now(), count(*) from extid where extid_type='dsc-sha256' and extid_version=1;
+------------------------------+-------+
|             now              | count |
+------------------------------+-------+
| 2021-12-07 17:14:30.74022+00 |  1346 |
+------------------------------+-------+
(1 row)

Time: 506.089 ms
        • [3] actual recent loading with the corresponding version to deploy
        Dec 07 17:08:37 worker16 python3[3153320]: [2021-12-07 17:08:37,751: INFO/ForkPoolWorker-1] Task swh.loader.package.debian.tasks.LoadDebian[3bf41deb-d76c-4101-b07f-0361c93ab8a2] succeeded in 8.943436987698078s: {'status': 'eventful', 'snapshot_id': 'f988003d7f2f2d5bfb4c59a6753aaee6eeb5df18'}
Dec 07 17:08:37 worker16 python3[3153304]: [2021-12-07 17:08:37,757: INFO/MainProcess] Received task: swh.loader.package.debian.tasks.LoadDebian[ecea322f-db58-4324-95a3-21d569afdc07]
Dec 07 17:08:48 worker16 python3[3153320]: [2021-12-07 17:08:48,635: INFO/ForkPoolWorker-1] Task swh.loader.package.debian.tasks.LoadDebian[7d186a4d-fd35-4401-8d55-626f14f72b97] succeeded in 10.87301928922534s: {'status': 'eventful', 'snapshot_id': '1e31823ab071b2b94777af02ce939c306a8265c5'}
Dec 07 17:08:48 worker16 python3[3153304]: [2021-12-07 17:08:48,639: INFO/MainProcess] Received task: swh.loader.package.debian.tasks.LoadDebian[cf1b3763-4d3d-4426-ba8e-f3677de7850d]
Dec 07 17:08:55 worker16 python3[3153320]: [2021-12-07 17:08:55,248: INFO/ForkPoolWorker-1] Task swh.loader.package.debian.tasks.LoadDebian[ecea322f-db58-4324-95a3-21d569afdc07] succeeded in 6.606376327574253s: {'status': 'eventful', 'snapshot_id': 'ff5f7059a28a88d84dfd9b93b536facea8b0ab64'}
Dec 07 17:08:55 worker16 python3[3153304]: [2021-12-07 17:08:55,423: INFO/MainProcess] Received task: swh.loader.package.debian.tasks.LoadDebian[67271c1d-0abf-482d-b3f6-74e958b55de9]
Dec 07 17:09:05 worker16 python3[3153743]: [2021-12-07 17:09:05,732: INFO/ForkPoolWorker-2] Task swh.loader.package.debian.tasks.LoadDebian[cf1b3763-4d3d-4426-ba8e-f3677de7850d] succeeded in 10.276469510048628s: {'status': 'eventful', 'snapshot_id': 'd79d8965bdec9501ba636499d8592fa8425e8505'}
Dec 07 17:09:05 worker16 python3[3153304]: [2021-12-07 17:09:05,737: INFO/MainProcess] Received task: swh.loader.package.debian.tasks.LoadDebian[46839761-5fa7-4c5d-8787-746c050348a2]
Dec 07 17:09:12 worker16 python3[3153743]: [2021-12-07 17:09:12,545: INFO/ForkPoolWorker-2] Task swh.loader.package.debian.tasks.LoadDebian[67271c1d-0abf-482d-b3f6-74e958b55de9] succeeded in 6.80641707777977s: {'status': 'eventful', 'snapshot_id': 'cfbf4ae98f5bb68b309ce4443d64b62eed0623b4'}
Dec 07 17:09:12 worker16 python3[3153304]: [2021-12-07 17:09:12,549: INFO/MainProcess] Received task: swh.loader.package.debian.tasks.LoadDebian[4417c334-7820-4ae7-8192-c4ceab91a8aa]
      • Antoine R. Dumont changed the description

        changed the description

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        Schedule back origins with visit_type "deb" (ongoing)

        For production only, i've triggered back the scheduling of visit type 'deb' using the send-to-celery cli with the policy oldest_scheduled_first [1] Given enough time, this should pass through the existing origins faster (than waiting for schedule recurrent service).

        • [1]
        swhscheduler@saatchi:~$ visit_type=deb; sleep=3600; while true; do   for policy in oldest_scheduled_first; do     echo "$(date) scheduling $visit_type origins with policy ${policy}";     SWH_CONFIG_FILENAME=/etc/softwareheritage/scheduler/listener-runner.yml       swh scheduler -C /etc/softwareheritage/scheduler/listener-runner.yml         origin send-to-celery --policy $policy $visit_type;     echo "$(date) sleep $sleep" ;     sleep $sleep;     done; done
...
5000 slots available in celery queue
5000 visits to send to celery
...
Tue Dec  7 17:51:54 UTC 2021 scheduling deb origins with policy oldest_scheduled_first
2679 slots available in celery queue
2679 visits to send to celery
Tue Dec  7 17:52:30 UTC 2021 sleep 3600
...
      • Antoine R. Dumont marked the checklist item for platform in {staging, production} as completed

        marked the checklist item for platform in {staging, production} as completed

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        All ongoing ingestions [1] are now only doing 'uneventful' events. So that means we did finalize the re-ingestions.

        • [1]
        root@pergamon:~# clush -b -w @swh-workers 'systemctl status swh-worker@loader_debian | grep succeeded' | grep 'succeeded'
Dec 08 13:27:02 worker01 python3[678508]: [2021-12-08 13:27:02,300: INFO/ForkPoolWorker-3385] Task swh.loader.package.debian.tasks.LoadDebian[2851f2ec-6542-43b4-9d56-5b1e9c0fc0f9] succeeded in 0.6866542343050241s: {'status': 'uneventful', 'snapshot_id': '752d3b9b11a034efb2dcb6551592a93dc4478c49'}
Dec 08 13:27:03 worker01 python3[678508]: [2021-12-08 13:27:03,281: INFO/ForkPoolWorker-3385] Task swh.loader.package.debian.tasks.LoadDebian[2ea9899d-14c1-4f16-86ac-649f22673894] succeeded in 0.9759201072156429s: {'status': 'uneventful', 'snapshot_id': '0d03a01c343968e7ceeb0fd5a57da088d7fd258a'}
Dec 08 13:27:04 worker01 python3[678508]: [2021-12-08 13:27:04,417: INFO/ForkPoolWorker-3385] Task swh.loader.package.debian.tasks.LoadDebian[d02f247b-7818-4af0-ad6f-c2b2076be394] succeeded in 1.1328360177576542s: {'status': 'uneventful', 'snapshot_id': '9f1469e6bb8aa0ac066cd618d262ec380b32400b'}
Dec 08 13:27:05 worker01 python3[678508]: [2021-12-08 13:27:05,499: INFO/ForkPoolWorker-3385] Task swh.loader.package.debian.tasks.LoadDebian[b794637a-2753-49c8-aaaa-c001559f26de] succeeded in 1.0800890605896711s: {'status': 'uneventful', 'snapshot_id': 'bf7287328ce8f8aff05525994c8ca9d19d62b9cd'}
Dec 08 13:27:05 worker01 python3[678508]: [2021-12-08 13:27:05,960: INFO/ForkPoolWorker-3385] Task swh.loader.package.debian.tasks.LoadDebian[af15500d-67e4-477d-9836-f6fd1b6da9c0] succeeded in 0.45769464038312435s: {'status': 'uneventful', 'snapshot_id': '2b0ab1c09b69c98fcc44fb9919853ca257b83e4b'}
Dec 08 13:27:01 worker02 python3[2915835]: [2021-12-08 13:27:01,794: INFO/ForkPoolWorker-3383] Task swh.loader.package.debian.tasks.LoadDebian[7810e418-cdd1-4e0d-b79f-846589184e52] succeeded in 0.48604015447199345s: {'status': 'uneventful', 'snapshot_id': '5ffa608fc6c92b95ba8c776d638bb5891892353c'}
Dec 08 13:27:02 worker02 python3[2915835]: [2021-12-08 13:27:02,270: INFO/ForkPoolWorker-3383] Task swh.loader.package.debian.tasks.LoadDebian[b0c39a40-2a56-4cb8-90cc-df9fb8ec38dc] succeeded in 0.47339972853660583s: {'status': 'uneventful', 'snapshot_id': '8de2447c9c0bba5eb9c2e618f01aa7dd13d14d6e'}
Dec 08 13:27:03 worker02 python3[2915835]: [2021-12-08 13:27:03,456: INFO/ForkPoolWorker-3383] Task swh.loader.package.debian.tasks.LoadDebian[4d775a2f-ea55-4621-8da4-308b516b88ee] succeeded in 1.1840890143066645s: {'status': 'uneventful', 'snapshot_id': '5723d9d3dbe4a2ba7ff9fee5aba1a59782059157'}
Dec 08 13:27:04 worker02 python3[2915835]: [2021-12-08 13:27:04,458: INFO/ForkPoolWorker-3383] Task swh.loader.package.debian.tasks.LoadDebian[421f6886-969d-4755-887a-e2797efb942d] succeeded in 0.9995743539184332s: {'status': 'uneventful', 'snapshot_id': '2a2b5982fbb536c4b69c13905db7e4c9e0775402'}
Dec 08 13:27:05 worker02 python3[2915835]: [2021-12-08 13:27:05,212: INFO/ForkPoolWorker-3383] Task swh.loader.package.debian.tasks.LoadDebian[dab75d7f-bd9b-484e-8aeb-6785254a75cb] succeeded in 0.7513855118304491s: {'status': 'uneventful', 'snapshot_id': 'bfd01900b6a8bd8b662b5d5d28f07ae59d4a68b2'}
Dec 08 13:27:04 worker03 python3[650958]: [2021-12-08 13:27:04,926: INFO/ForkPoolWorker-3124] Task swh.loader.package.debian.tasks.LoadDebian[a918bc1f-0a19-45ea-a6fc-a4fc3db50748] succeeded in 0.41829228587448597s: {'status': 'uneventful', 'snapshot_id': '80356c23b68e5d33391ee0a64091d108007b3377'}
Dec 08 13:27:07 worker03 python3[650960]: [2021-12-08 13:27:07,544: INFO/ForkPoolWorker-3125] Task swh.loader.package.debian.tasks.LoadDebian[cf67e698-247d-4daa-9e33-19ed6675bc82] succeeded in 0.535936875268817s: {'status': 'uneventful', 'snapshot_id': 'e23f922781b49a2e5bda559f393dfa4c7bde0aad'}
Dec 08 13:27:08 worker03 python3[650960]: [2021-12-08 13:27:08,061: INFO/ForkPoolWorker-3125] Task swh.loader.package.debian.tasks.LoadDebian[fd2f4150-efa1-4965-a066-08e0f712c119] succeeded in 0.5147611014544964s: {'status': 'uneventful', 'snapshot_id': 'b00e058d78c3999365651f462b2aabc392b72a53'}
Dec 08 13:27:03 worker04 python3[475813]: [2021-12-08 13:27:03,006: INFO/ForkPoolWorker-3162] Task swh.loader.package.debian.tasks.LoadDebian[2809938b-e78c-4046-aa73-1c44eadbcc4a] succeeded in 0.4477949570864439s: {'status': 'uneventful', 'snapshot_id': '959bfbdd6f6aa05ca95d5c0d93a1a72cd475bd1c'}
Dec 08 13:27:04 worker04 python3[475813]: [2021-12-08 13:27:04,455: INFO/ForkPoolWorker-3162] Task swh.loader.package.debian.tasks.LoadDebian[fe3c0442-ad3c-4daf-b6ef-575bb43e649f] succeeded in 1.446052024140954s: {'status': 'uneventful', 'snapshot_id': '5242fb514ae516e61332da9659605da738201390'}
Dec 08 13:27:04 worker04 python3[475813]: [2021-12-08 13:27:04,959: INFO/ForkPoolWorker-3162] Task swh.loader.package.debian.tasks.LoadDebian[9d39fa5d-12e8-4ad1-a6ad-3ce88ce7754c] succeeded in 0.49902280420064926s: {'status': 'uneventful', 'snapshot_id': '57ef50e761dea2da237bdbfa7d73f2d73f5314e4'}
Dec 08 13:27:05 worker04 python3[475813]: [2021-12-08 13:27:05,598: INFO/ForkPoolWorker-3162] Task swh.loader.package.debian.tasks.LoadDebian[2225e665-e553-4347-9313-78a38c4a4ead] succeeded in 0.6339817196130753s: {'status': 'uneventful', 'snapshot_id': '7668185b442aee9eb6505263c6982225d2e747fe'}
Dec 08 13:27:07 worker04 python3[475819]: [2021-12-08 13:27:07,747: INFO/ForkPoolWorker-3163] Task swh.loader.package.debian.tasks.LoadDebian[cbc707c3-ea94-42c8-ab92-9e145827e86e] succeeded in 0.5991860795766115s: {'status': 'uneventful', 'snapshot_id': '01248f926d55af5f593b59c22130430b6b4e2235'}
Dec 08 13:27:04 worker05 python3[571934]: [2021-12-08 13:27:04,419: INFO/ForkPoolWorker-2957] Task swh.loader.package.debian.tasks.LoadDebian[91ec2c66-a367-4823-a942-ee47bbb87372] succeeded in 0.9995316155254841s: {'status': 'uneventful', 'snapshot_id': '8ca1c9b2675b00d87a8eb6a0dd3af2241bcad7e0'}
Dec 08 13:27:05 worker05 python3[571934]: [2021-12-08 13:27:05,338: INFO/ForkPoolWorker-2957] Task swh.loader.package.debian.tasks.LoadDebian[5a015a09-c139-41fe-8d58-1d4c7f0b885d] succeeded in 0.9165753610432148s: {'status': 'uneventful', 'snapshot_id': '5cd1e109da56780025577fe8d9ad916ceb50d3c6'}
Dec 08 13:27:06 worker05 python3[571935]: [2021-12-08 13:27:06,201: INFO/ForkPoolWorker-2958] Task swh.loader.package.debian.tasks.LoadDebian[8ed13bf6-d3e2-4403-8f60-cd58279db3aa] succeeded in 0.6698852982372046s: {'status': 'uneventful', 'snapshot_id': 'bb6b88d29e64188905827f2d01521edeec72f745'}
Dec 08 13:27:06 worker05 python3[571935]: [2021-12-08 13:27:06,839: INFO/ForkPoolWorker-2958] Task swh.loader.package.debian.tasks.LoadDebian[64895b40-718d-4a8a-842b-f20014412a05] succeeded in 0.6359769906848669s: {'status': 'uneventful', 'snapshot_id': '033ef7a537e6d35ae056d7d70f315122fc2f2e2e'}
Dec 08 13:27:07 worker05 python3[571935]: [2021-12-08 13:27:07,726: INFO/ForkPoolWorker-2958] Task swh.loader.package.debian.tasks.LoadDebian[4940c715-a34f-42f8-8d24-3256e4ccbf12] succeeded in 0.8814013879746199s: {'status': 'uneventful', 'snapshot_id': 'f380af10ad2cb22d6973a17c635ed060c3dce504'}
Dec 08 13:26:58 worker06 python3[628354]: [2021-12-08 13:26:58,773: INFO/ForkPoolWorker-3112] Task swh.loader.package.debian.tasks.LoadDebian[ff4e80a1-a907-4737-808b-6dbeb252e33d] succeeded in 1.2383377272635698s: {'status': 'uneventful', 'snapshot_id': '514af285887ea3048eb36b82c4f840aabe903934'}
Dec 08 13:26:59 worker06 python3[628354]: [2021-12-08 13:26:59,260: INFO/ForkPoolWorker-3112] Task swh.loader.package.debian.tasks.LoadDebian[0a9565cd-9607-4b09-9db7-212026a44364] succeeded in 0.48336851224303246s: {'status': 'uneventful', 'snapshot_id': '26c7a9c47c57c04c3c0a2745a979e3b9aa2c1ed3'}
Dec 08 13:27:05 worker06 python3[628364]: [2021-12-08 13:27:05,917: INFO/ForkPoolWorker-3113] Task swh.loader.package.debian.tasks.LoadDebian[21435ab8-f861-49d3-aa23-ad939cbe44e0] succeeded in 0.6593674421310425s: {'status': 'uneventful', 'snapshot_id': '97fe431c155eb1c465b692227abf8d05f008d7e8'}
Dec 08 13:27:06 worker06 python3[628364]: [2021-12-08 13:27:06,741: INFO/ForkPoolWorker-3113] Task swh.loader.package.debian.tasks.LoadDebian[fd4575be-d003-4cec-bf86-9dc71985c310] succeeded in 0.8210923839360476s: {'status': 'uneventful', 'snapshot_id': 'f6a5f5a596a740dbf51b6bc64287483d5ba6ac89'}
Dec 08 13:27:07 worker06 python3[628364]: [2021-12-08 13:27:07,709: INFO/ForkPoolWorker-3113] Task swh.loader.package.debian.tasks.LoadDebian[f185f4b2-4a67-4082-93ff-fb25fb31bd0e] succeeded in 0.9639273826032877s: {'status': 'uneventful', 'snapshot_id': 'e4e528a93576846e1df24762fde8a1c48e848a04'}
Dec 08 13:27:03 worker07 python3[581359]: [2021-12-08 13:27:03,330: INFO/ForkPoolWorker-3229] Task swh.loader.package.debian.tasks.LoadDebian[318f4890-059a-4e87-943a-71fc4c207eb4] succeeded in 0.7694193031638861s: {'status': 'uneventful', 'snapshot_id': 'bff0d39bff71617648cb0fd155042556518a54be'}
Dec 08 13:27:04 worker07 python3[581359]: [2021-12-08 13:27:04,289: INFO/ForkPoolWorker-3229] Task swh.loader.package.debian.tasks.LoadDebian[b7b5ff48-3e90-4347-a545-3026a96e5de5] succeeded in 0.9565333966165781s: {'status': 'uneventful', 'snapshot_id': '459ae8071708be5e59b51fda5a890c79658e8380'}
Dec 08 13:27:05 worker07 python3[581359]: [2021-12-08 13:27:05,172: INFO/ForkPoolWorker-3229] Task swh.loader.package.debian.tasks.LoadDebian[e306ceae-7fae-4d61-b417-1b8e59cc8402] succeeded in 0.8805344887077808s: {'status': 'uneventful', 'snapshot_id': 'c4315c997c0501d8c166821681cda62ae23a066c'}
Dec 08 13:27:05 worker07 python3[581359]: [2021-12-08 13:27:05,790: INFO/ForkPoolWorker-3229] Task swh.loader.package.debian.tasks.LoadDebian[7f99d09a-f8ec-4422-b5d8-b18b0c1a67a0] succeeded in 0.6147643756121397s: {'status': 'uneventful', 'snapshot_id': '55956164efc84e1c64d259f893fcddbff9bb4b27'}
Dec 08 13:27:06 worker07 python3[581359]: [2021-12-08 13:27:06,338: INFO/ForkPoolWorker-3229] Task swh.loader.package.debian.tasks.LoadDebian[44ebccd0-bb41-4d7d-a9c7-efd14254ee59] succeeded in 0.5415774844586849s: {'status': 'uneventful', 'snapshot_id': 'e8d1434a658153e7835d7b4842747358e2b92720'}
Dec 08 13:27:02 worker08 python3[619010]: [2021-12-08 13:27:02,117: INFO/ForkPoolWorker-2977] Task swh.loader.package.debian.tasks.LoadDebian[3807496b-25a2-4ed6-b507-78ecb4093d63] succeeded in 0.7612722851336002s: {'status': 'uneventful', 'snapshot_id': 'f46a4a5790849901aca0e4d774e0a15b9638ed9f'}
Dec 08 13:27:02 worker08 python3[619010]: [2021-12-08 13:27:02,773: INFO/ForkPoolWorker-2977] Task swh.loader.package.debian.tasks.LoadDebian[8fb21bfc-e136-4322-b307-9908f73738c9] succeeded in 0.653038902208209s: {'status': 'uneventful', 'snapshot_id': 'cf45f8cd5cf7a7a52f5287a09fe0849f906b37b2'}
Dec 08 13:27:03 worker08 python3[619010]: [2021-12-08 13:27:03,499: INFO/ForkPoolWorker-2977] Task swh.loader.package.debian.tasks.LoadDebian[becb5703-b95e-484a-bf76-5bb2f682d58b] succeeded in 0.7234356962144375s: {'status': 'uneventful', 'snapshot_id': '214a284c603c8f2c2c060c0b3d9a08560f3ec562'}
Dec 08 13:27:04 worker08 python3[619010]: [2021-12-08 13:27:04,363: INFO/ForkPoolWorker-2977] Task swh.loader.package.debian.tasks.LoadDebian[c0f9cfde-cc5c-405d-b375-4c003f246a34] succeeded in 0.8620108757168055s: {'status': 'uneventful', 'snapshot_id': 'a76113266ad3784ba102d3fe179bc1d3e6a22421'}
Dec 08 13:27:07 worker08 python3[619011]: [2021-12-08 13:27:07,397: INFO/ForkPoolWorker-2978] Task swh.loader.package.debian.tasks.LoadDebian[a0b9e263-855a-40b2-8f3a-6eedbc752c0f] succeeded in 0.41254594922065735s: {'status': 'uneventful', 'snapshot_id': '32c58ec2eecef97eb2d3109f1552618b3d0ab4da'}
Dec 08 13:27:02 worker09 python3[717236]: [2021-12-08 13:27:02,501: INFO/ForkPoolWorker-3383] Task swh.loader.package.debian.tasks.LoadDebian[d3a2cd7a-83dd-481e-8a18-6108831a86e4] succeeded in 0.5456999782472849s: {'status': 'uneventful', 'snapshot_id': '0c3a0cb49508cccb3f49b2b0557d5a4f88775eb2'}
Dec 08 13:27:03 worker09 python3[717236]: [2021-12-08 13:27:03,285: INFO/ForkPoolWorker-3383] Task swh.loader.package.debian.tasks.LoadDebian[502f2cdb-de37-431d-b0d2-150dc895c48e] succeeded in 0.7818146981298923s: {'status': 'uneventful', 'snapshot_id': 'fb11397830f8e99147688a06c736940a2097f246'}
Dec 08 13:27:04 worker09 python3[717236]: [2021-12-08 13:27:04,407: INFO/ForkPoolWorker-3383] Task swh.loader.package.debian.tasks.LoadDebian[1694982b-d70e-4078-8c63-d504d9863d66] succeeded in 1.1197837069630623s: {'status': 'uneventful', 'snapshot_id': 'd2caefbffba919dd2dae684a8b10eded1d708bef'}
Dec 08 13:27:05 worker09 python3[717236]: [2021-12-08 13:27:05,077: INFO/ForkPoolWorker-3383] Task swh.loader.package.debian.tasks.LoadDebian[f0646b05-a8e9-418e-a280-95e31cc18b86] succeeded in 0.6671788021922112s: {'status': 'uneventful', 'snapshot_id': 'fb3a00a371a474e6e55e7aeb75b703da944d4826'}
Dec 08 13:27:05 worker09 python3[717236]: [2021-12-08 13:27:05,733: INFO/ForkPoolWorker-3383] Task swh.loader.package.debian.tasks.LoadDebian[91c64ff8-1748-4ad4-a6ac-81cb7ca51d1a] succeeded in 0.6532656252384186s: {'status': 'uneventful', 'snapshot_id': '30321e4c7c1292cb961d43e1d63c466d06cc2491'}
Dec 08 13:27:02 worker10 python3[726377]: [2021-12-08 13:27:02,609: INFO/ForkPoolWorker-3508] Task swh.loader.package.debian.tasks.LoadDebian[c9876458-3963-416b-9b53-1ba3acfb5d9f] succeeded in 0.7058506589382887s: {'status': 'uneventful', 'snapshot_id': 'd979577d663c6e9f205a22a6e676f3260455cb4f'}
Dec 08 13:27:03 worker10 python3[726377]: [2021-12-08 13:27:03,463: INFO/ForkPoolWorker-3508] Task swh.loader.package.debian.tasks.LoadDebian[5af1fb92-2d50-448c-979b-42e5dc063837] succeeded in 0.851406442001462s: {'status': 'uneventful', 'snapshot_id': '0db89b1d80fad4337486f5930f9764886b914309'}
Dec 08 13:27:03 worker10 python3[726377]: [2021-12-08 13:27:03,875: INFO/ForkPoolWorker-3508] Task swh.loader.package.debian.tasks.LoadDebian[2128051b-bd23-47b9-84d3-81b263f1f86b] succeeded in 0.410512525588274s: {'status': 'uneventful', 'snapshot_id': '23d109f5f4fa64556e8a9b82c78496deacea7be5'}
Dec 08 13:27:04 worker10 python3[726377]: [2021-12-08 13:27:04,681: INFO/ForkPoolWorker-3508] Task swh.loader.package.debian.tasks.LoadDebian[6d5f14f7-b89a-45b6-a4ba-faaaa43ccf8b] succeeded in 0.8033899273723364s: {'status': 'uneventful', 'snapshot_id': '690a593853db202b68c025012eec7a08ecab443e'}
Dec 08 13:27:07 worker10 python3[726380]: [2021-12-08 13:27:07,761: INFO/ForkPoolWorker-3509] Task swh.loader.package.debian.tasks.LoadDebian[233b8dc9-b12d-486e-83b0-beb830cab2a1] succeeded in 0.39271807111799717s: {'status': 'uneventful', 'snapshot_id': '1e3d8177cb219f4b4e135e53eb8da12487cbfa30'}
Dec 08 13:27:05 worker11 python3[619427]: [2021-12-08 13:27:05,199: INFO/ForkPoolWorker-3440] Task swh.loader.package.debian.tasks.LoadDebian[0032693c-c78a-4c27-aba4-dba542d20def] succeeded in 0.6839302219450474s: {'status': 'uneventful', 'snapshot_id': '4c48ebe7b210cced593e93e62e56c3b3e23a8c40'}
Dec 08 13:27:05 worker11 python3[619427]: [2021-12-08 13:27:05,620: INFO/ForkPoolWorker-3440] Task swh.loader.package.debian.tasks.LoadDebian[56a15b5d-85e8-4b8c-90b8-d49ab42b9cfd] succeeded in 0.4181954823434353s: {'status': 'uneventful', 'snapshot_id': 'e3e60219c25f7db13c4824a9dea7221eb76c97d0'}
Dec 08 13:27:06 worker11 python3[619427]: [2021-12-08 13:27:06,559: INFO/ForkPoolWorker-3440] Task swh.loader.package.debian.tasks.LoadDebian[f599be5e-41d5-4287-9394-6e73381abf14] succeeded in 0.9354355446994305s: {'status': 'uneventful', 'snapshot_id': 'd2d68c1b0d5e9ce66a151b0dfd45e3132b0a7c9f'}
Dec 08 13:27:07 worker11 python3[619427]: [2021-12-08 13:27:07,092: INFO/ForkPoolWorker-3440] Task swh.loader.package.debian.tasks.LoadDebian[e30673bb-e2ff-47ef-bffa-0b699d6483af] succeeded in 0.5300557464361191s: {'status': 'uneventful', 'snapshot_id': 'ef414d1ab088611ec28c05b8288ec8e15fe92b13'}
Dec 08 13:27:08 worker11 python3[619428]: [2021-12-08 13:27:08,066: INFO/ForkPoolWorker-3441] Task swh.loader.package.debian.tasks.LoadDebian[1ac95939-9371-4028-929c-25700bb1d835] succeeded in 0.7705352250486612s: {'status': 'uneventful', 'snapshot_id': 'b8cca9b11e1b6fcb59a3c6eb4c0e56a06a6d062a'}
Dec 08 08:33:05 worker12 python3[616843]: [2021-12-08 08:33:05,221: INFO/ForkPoolWorker-1290] Task swh.loader.package.debian.tasks.LoadDebian[3b2943a5-e15d-43bc-a3fa-72bcea087a00] succeeded in 0.5947206597775221s: {'status': 'uneventful', 'snapshot_id': 'e799f9c058bac95f07464d063e6a331516a356cd'}
Dec 08 08:33:05 worker12 python3[616843]: [2021-12-08 08:33:05,686: INFO/ForkPoolWorker-1290] Task swh.loader.package.debian.tasks.LoadDebian[754c2508-b547-4a64-8942-64f56e039aa1] succeeded in 0.4633220173418522s: {'status': 'uneventful', 'snapshot_id': 'c60c8acd560645f597822bb31ab13d0ba0b4390b'}
Dec 08 08:33:06 worker12 python3[616844]: [2021-12-08 08:33:06,238: INFO/ForkPoolWorker-1291] Task swh.loader.package.debian.tasks.LoadDebian[31735e21-c363-4499-8105-87fc6f1c661f] succeeded in 0.36779797449707985s: {'status': 'uneventful', 'snapshot_id': '460dc8a988484bf5673e29779a897eab18e32bed'}
Dec 08 13:27:05 worker13 python3[3433650]: [2021-12-08 13:27:05,367: INFO/ForkPoolWorker-3165] Task swh.loader.package.debian.tasks.LoadDebian[1d861689-95bf-4591-b2ac-1abf6659a453] succeeded in 0.24523821100592613s: {'status': 'uneventful', 'snapshot_id': 'd80894356a0d8460b844c802bba5f852e92fbb7c'}
Dec 08 13:27:05 worker13 python3[3433650]: [2021-12-08 13:27:05,950: INFO/ForkPoolWorker-3165] Task swh.loader.package.debian.tasks.LoadDebian[90fdd79f-daef-426f-bb8d-4a800e77fcfa] succeeded in 0.5802949145436287s: {'status': 'uneventful', 'snapshot_id': '6fc6aec7a4d9d36ff92c9dc240243d623711c472'}
Dec 08 13:27:06 worker13 python3[3433651]: [2021-12-08 13:27:06,954: INFO/ForkPoolWorker-3166] Task swh.loader.package.debian.tasks.LoadDebian[028ca150-321a-4533-a550-9a9b85c3721a] succeeded in 0.7972856387495995s: {'status': 'uneventful', 'snapshot_id': '40c3453cb93018fd6bfc9f193cc90e1e1750ce65'}
Dec 08 13:27:01 worker14 python3[3329206]: [2021-12-08 13:27:01,937: INFO/ForkPoolWorker-3314] Task swh.loader.package.debian.tasks.LoadDebian[9f1d347a-4f2c-4d36-b593-7c7a4695be55] succeeded in 0.6414489522576332s: {'status': 'uneventful', 'snapshot_id': '42d625f498a08b9069fa81cac8eaa6654fee8db9'}
Dec 08 13:27:02 worker14 python3[3329206]: [2021-12-08 13:27:02,576: INFO/ForkPoolWorker-3314] Task swh.loader.package.debian.tasks.LoadDebian[5526810f-befe-4e40-a844-52f372c0ee52] succeeded in 0.6344279609620571s: {'status': 'uneventful', 'snapshot_id': 'e78dcd97461adb72fe28287cba664f4488010566'}
Dec 08 13:27:03 worker14 python3[3329206]: [2021-12-08 13:27:03,318: INFO/ForkPoolWorker-3314] Task swh.loader.package.debian.tasks.LoadDebian[d5e3b59f-1e64-4f89-bf2e-ed241332f5a0] succeeded in 0.7364939358085394s: {'status': 'uneventful', 'snapshot_id': 'f2f5a9c1f42e993ea6505ef889b67787dcf316a6'}
Dec 08 13:27:03 worker14 python3[3329206]: [2021-12-08 13:27:03,700: INFO/ForkPoolWorker-3314] Task swh.loader.package.debian.tasks.LoadDebian[f959dca2-7868-46c8-bf32-fa2ba868cfbf] succeeded in 0.3786452692002058s: {'status': 'uneventful', 'snapshot_id': 'dc68be26941b57d8ce34e08acf377419ff011819'}
Dec 08 13:27:07 worker14 python3[3329210]: [2021-12-08 13:27:07,774: INFO/ForkPoolWorker-3315] Task swh.loader.package.debian.tasks.LoadDebian[50a039a6-7fb8-4c37-b33a-691ff29476b8] succeeded in 0.8697554171085358s: {'status': 'uneventful', 'snapshot_id': '1ee73bd14d18e5a3e7e8679a89a6c58b6745caf1'}
Dec 08 13:27:05 worker15 python3[3285376]: [2021-12-08 13:27:05,391: INFO/ForkPoolWorker-3570] Task swh.loader.package.debian.tasks.LoadDebian[43676611-486f-4d72-b9f5-19c832b3c3f2] succeeded in 0.27889907732605934s: {'status': 'uneventful', 'snapshot_id': 'd53d2bcf287631949e21b06603d961ed485111d5'}
Dec 08 13:27:05 worker15 python3[3285376]: [2021-12-08 13:27:05,812: INFO/ForkPoolWorker-3570] Task swh.loader.package.debian.tasks.LoadDebian[82bcb01c-5164-4ad7-a6e7-7ccea5e0bc4f] succeeded in 0.41776033863425255s: {'status': 'uneventful', 'snapshot_id': '2cfcd9f4461b66266e32542cbdffc995ba5a6fff'}
Dec 08 13:27:06 worker15 python3[3285377]: [2021-12-08 13:27:06,618: INFO/ForkPoolWorker-3571] Task swh.loader.package.debian.tasks.LoadDebian[5f6d9fbb-f137-4ea7-8de9-dab9c3503fe5] succeeded in 0.5858875531703234s: {'status': 'uneventful', 'snapshot_id': '6d45399f8c3394b15f4670d79970b30768311753'}
Dec 08 13:27:07 worker15 python3[3285377]: [2021-12-08 13:27:07,272: INFO/ForkPoolWorker-3571] Task swh.loader.package.debian.tasks.LoadDebian[b7dcb95b-9d66-4067-ad7b-9a715c1cb018] succeeded in 0.6478608753532171s: {'status': 'uneventful', 'snapshot_id': 'b6dcc15565d4c22c7d7c4436fee204f752bab7af'}
Dec 08 13:27:08 worker15 python3[3285377]: [2021-12-08 13:27:08,013: INFO/ForkPoolWorker-3571] Task swh.loader.package.debian.tasks.LoadDebian[718cec28-0b9f-49eb-b047-e9cfcf8ad11f] succeeded in 0.7377434447407722s: {'status': 'uneventful', 'snapshot_id': '6866378a88fca3cf8ccee2c11e772a65cdb47834'}
Dec 08 13:26:52 worker16 python3[3260822]: [2021-12-08 13:26:52,822: INFO/ForkPoolWorker-3344] Task swh.loader.package.debian.tasks.LoadDebian[2b6d2f6b-fe88-433c-a577-b8def481bb6e] succeeded in 0.5624852143228054s: {'status': 'uneventful', 'snapshot_id': '46098d4e9aa794ccf3b5ec819a93107e9367ff06'}
      • Antoine R. Dumont marked the checklist item Schedule back origins with visit_type "deb" (ongoing) as completed

        marked the checklist item Schedule back origins with visit_type "deb" (ongoing) as completed

      • Antoine R. Dumont closed

        closed

      • Antoine R. Dumont removed state:wip label

        removed state:wip label

      • Antoine R. Dumont mentioned in issue #2400 (closed)

        mentioned in issue #2400 (closed)

      Please register or sign in to reply