staging: Deploy golang stack
-
Package swh.loader.core v4.2 -
Edit chart to add the golang loader and lister -
Add golang lister task to recurring listing [1] -
Lister triggered [2] -
#4545 (closed): Let loaders ingest [3]
For information, sentry issues should pop up along the way [4]
- [1]
swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ task add list-golang-incremental
Created 1 tasks
Task 33419557
Next run: today (2022-09-22T16:19:15.520781+00:00)
Interval: 1 day, 0:00:00
Type: list-golang-incremental
Policy: recurring
Args:
Keyword args:- [2]
17:54:45 swh-scheduler@db1:5432=> select now(), lister_id, visit_type, count(*) from listed_origins where lister_id=(select id from listers where name='golang') group by lister_id, visit_type;
+-------------------------------+--------------------------------------+------------+-------+
| now | lister_id | visit_type | count |
+-------------------------------+--------------------------------------+------------+-------+
| 2022-09-22 16:29:35.206863+00 | 46b62892-60b2-4252-b37a-5c3cc84c9a6d | golang | 98612 |
+-------------------------------+--------------------------------------+------------+-------+
(1 row)
Time: 967.805 ms
18:29:36 swh-scheduler@db1:5432=> select now(), lister_id, visit_type, count(*) from listed_origins where lister_id=(select id from listers where name='golang') group by lister_id, visit_type;
+------------------------------+--------------------------------------+------------+-------+
| now | lister_id | visit_type | count |
+------------------------------+--------------------------------------+------------+-------+
| 2022-09-22 16:29:57.37355+00 | 46b62892-60b2-4252-b37a-5c3cc84c9a6d | golang | 99616 |
+------------------------------+--------------------------------------+------------+-------+
(1 row)
Time: 795.562 ms- [3]
│ loaders [2022-09-23 08:14:40,355: INFO/ForkPoolWorker-1] Task swh.loader.package.golang.tasks.LoadGolang[3e4aaa49-bebf-4343-9768-05c1d2ed6862] succeeded in 0.7992246079957113s: {'status': 'failed'} │
│ loaders [2022-09-23 08:14:40,367: INFO/MainProcess] Task swh.loader.package.golang.tasks.LoadGolang[d7c25e0e-42a6-4ed2-b390-c9de03ad7976] received │
│ loaders [2022-09-23 08:14:57,376: INFO/ForkPoolWorker-1] Task swh.loader.package.golang.tasks.LoadGolang[a4db278c-4ba6-48c8-b3f2-c1e0f71043a2] succeeded in 17.01177186399582s: {'status': 'eventful', 'snapshot_id': 'db5d1dbb46a4246d4972 │
│ loaders [2022-09-23 08:14:57,398: INFO/MainProcess] Task swh.loader.package.golang.tasks.LoadGolang[860bbdb0-f7b3-4a26-aae8-bfece0c146e4] received │
│ loaders [2022-09-23 08:15:05,873: INFO/ForkPoolWorker-1] Task swh.loader.package.golang.tasks.LoadGolang[d7c25e0e-42a6-4ed2-b390-c9de03ad7976] succeeded in 8.471439613000257s: {'status': 'eventful', 'snapshot_id': 'c9151adbafd2216b1f85 │
│ loaders [2022-09-23 08:15:05,900: INFO/MainProcess] Task swh.loader.package.golang.tasks.LoadGolang[2e523a68-8eca-44d6-b6fb-07789bcc77d7] received │
│ loaders [2022-09-23 08:15:06,327: ERROR/ForkPoolWorker-1] Failed to load any release for https://pkg.go.dev/github.com/btcboost/copernicus │
│ loaders [2022-09-23 08:15:06,450: INFO/ForkPoolWorker-1] Task swh.loader.package.golang.tasks.LoadGolang[860bbdb0-f7b3-4a26-aae8-bfece0c146e4] succeeded in 0.5533570959960343s: {'status': 'failed'} │
│ loaders [2022-09-23 08:15:06,469: INFO/MainProcess] Task swh.loader.package.golang.tasks.LoadGolang[4d61ebd6-a8fb-4af8-b54e-c777e48e276b] received │
│ loaders [2022-09-23 08:15:26,138: INFO/ForkPoolWorker-1] Task swh.loader.package.golang.tasks.LoadGolang[2e523a68-8eca-44d6-b6fb-07789bcc77d7] succeeded in 19.67315485799918s: {'status': 'eventful', 'snapshot_id': '533b0f65c5af9d458d01 │
Migrated from T4545 (view on Phabricator)