Deploy pubdev stack to production

changed milestone to %Extend archive coverage [Roadmap - Collect]

Scheduling the lister:

swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/    task add list-pubdev
Created 1 tasks

Task 415386050
  Next run: today (2023-05-26T09:47:04.288287+00:00)
  Interval: 1 day, 0:00:00
  Type: list-pubdev
  Policy: recurring
  Args:
  Keyword args:

marked the checklist item Schedule one task with type 'list-pubdev' as completed

Lister image is loop-crashing on a missing configuration deps it seems [1]. Let's add the deps and rebuild the image.

[1]

 prepare-configuration sed: -e expression #1, char 25: unknown command: `/'

I misread the issue and thought sed was missing for some reason. It was actually a missing secret in the application in charge of installing secrets in the production cluster. It's fixed now.

mentioned in commit swh-apps@fba1adea

mentioned in commit swh/infra/ci-cd/swh-charts@aa533639

Lister running:

listers {"asctime": "2023-05-26 12:41:01,857", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/worker/consumer/connection.py", "lineno": 22, "funcName": "start", "task_name": null, "task_id": nu
ll, "name": "celery.worker.consumer.connection", "levelname": "INFO", "message": "Connected to amqp://swhconsumer:**@rabbitmq.internal.softwareheritage.org:5672//"}
listers {"asctime": "2023-05-26 12:41:01,888", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/apps/worker.py", "lineno": 176, "funcName": "on_consumer_ready", "task_name": null, "task_id": null
, "name": "celery.apps.worker", "levelname": "INFO", "message": "lister@lister-pubdev-7f9b7b9644-qnb6h ready."}
listers {"asctime": "2023-05-26 12:41:02,741", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/worker/strategy.py", "lineno": 161, "funcName": "task_message_handler", "task_name": null, "task_id
": null, "name": "celery.worker.strategy", "levelname": "INFO", "message": "Task swh.lister.pubdev.tasks.PubDevListerTask[738d9e35-a0d0-4c07-8be4-9c90d55c771d] received", "data": {"id": "738d9e35-a0d0-4c07-8be4-9c90d55c771d", "name": "swh.
lister.pubdev.tasks.PubDevListerTask", "args": "[]", "kwargs": "{}"}}

changed the description

It's still ongoing but origins got listed already:

2023-05-26 14:50:28 swh-scheduler@db1:5432 λ select now(), visit_type, count(*) from listed_origins where lister_id=(select id from listers where name='pubdev') group by visit_type;
+-------------------------------+------------+-------+
|              now              | visit_type | count |
+-------------------------------+------------+-------+
| 2023-05-26 12:50:40.771039+00 | pubdev     | 41980 |
+-------------------------------+------------+-------+
(1 row)

Time: 946.446 ms

hahaha, scratch that, Those numbers are extract from the staging db... So it's ongoing but nothing is listed yet for some reason.

mentioned in issue swh/meta#4465 (closed)

changed the description

Double checking the configuration, it's configured ok.

Reading the code, it seems that's the lister's behavior [0].

It first reads a list of packages (http request) [1] and for each package, fetch extra package information (another http request) [2]. It seems to flush every 1k origins found [3], so 1k requests prior to actually write something in the db (well 1001 for the first round but who's counting ;).

Stracing the process from the rancher-node-metal02 (where it runs), it's not stale either. So it's all fine.

I'm just gonna let it run and get back to this later.

[0] https://gitlab.softwareheritage.org/swh/devel/swh-lister/-/blob/master/swh/lister/pubdev/lister.py

[1] https://pub.dev/api/package-names

[2] https://pub.dev/api/packages/$package (e.g. xapptor_logic)

[3] https://gitlab.softwareheritage.org/swh/devel/swh-lister/-/blob/master/swh/lister/pattern.py#L353

Regarding resources, it's all well and dandy too (so no issue there either). (I had already checked it was fine with the staging one and kept the same resource configuration).

[1] https://grafana.softwareheritage.org/goto/VwxsC2w4z?orgId=1

changed the description

added 2h of time spent

"tout vient a point a qui sait attendre". It's done.

It's probably worth an issue on that lister so it flushes more regularly its origins though... That's [2] for ya'.

[0]

2023-05-26 16:02:55 softwareheritage-scheduler@belvedere:5432 λ select now(), visit_type, count(*) from listed_origins where lister_id=(select id from listers where name='pubdev') group by visit_type;
+-------------------------------+------------+-------+
|              now              | visit_type | count |
+-------------------------------+------------+-------+
| 2023-05-26 15:18:53.425956+00 | pubdev     | 41736 |
+-------------------------------+------------+-------+
(1 row)

Time: 742.238 ms

[1]

listers {"asctime": "2023-05-26 14:31:05,021", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/app/trace.py", "lineno": 131, "funcName": "info", "task_name": null, "task_id": null, "name": "cele
ry.app.trace", "levelname": "INFO", "message": "Task swh.lister.pubdev.tasks.PubDevListerTask[738d9e35-a0d0-4c07-8be4-9c90d55c771d] succeeded in 6602.262384598958s: {'pages': 1, 'origins': 41736}", "data": {"id": "738d9e35-a0d0-4c07-8be4-9
c90d55c771d", "name": "swh.lister.pubdev.tasks.PubDevListerTask", "return_value": "{'pages': 1, 'origins': 41736}", "runtime": 6602.262384598958, "args": "[]", "kwargs": "{}"}}

[2] swh/devel/swh-lister#4697

mentioned in issue swh/devel/swh-lister#4697

Deploy pubdev stack to production

Designs

Child items ...

Activity