staging: Deploy new Arch Linux stack

• swh-apps: Build docker image
• swh-charts: Reference docker image
• swh/infra/ci-cd/swh-charts!213 (merged): swh-charts: Adapt values to reference the new lister and loader
• Deploy (push branch and let argocd do its job)
• scheduler: Register new task types
• Trigger a listing [2]
• Tag grafana with "deployment service=lister-arch service=loader-arch environment=staging" [1]
• Checks
  • lister
  • loader: nope, failing.

Dashboard to follow [1]

[1] https://grafana.softwareheritage.org/goto/e-c3EAWIz?orgId=1

[2] from the toolbox pod

swh@swh-toolbox-849fd6b5d5-wxn8m:~$ swh scheduler -C $SWH_CONFIG_FILENAME task add list-arch
Created 1 tasks

Task 33423094
  Next run: today (2023-11-10T15:06:16.301232+00:00)
  Interval: 1 day, 0:00:00
  Type: list-arch
  Policy: recurring
  Args:
  Keyword args:

Refs. swh/meta#4233

changed milestone to %Extend archive coverage [Roadmap - Collect]

added activity::Deployment label

removed activity::Deployment label

added activity::Deployment label

mentioned in commit swh/infra/ci-cd/swh-charts@55b4215e

mentioned in commit swh/infra/ci-cd/swh-charts@dde24783

mentioned in merge request swh/infra/ci-cd/swh-charts!213 (merged)

changed the description

mentioned in issue swh/meta#4233

marked the checklist item swh/infra/ci-cd/swh-charts!213 (merged): swh-charts: Adapt values to reference the new lister and loader as completed

marked the checklist item Deploy (push branch and let argocd do its job) as completed

mentioned in commit swh/infra/ci-cd/swh-charts@12ed8aa4

Listing started and new origins got registered.

2023-11-10 17:07:26 swh-scheduler@db1:5432 λ select now(), name, instance_name, visit_type, enabled, count(*) from listed_origins lo inner join listers l on l.id=lo.lister_id where lister_id in (select id from listers where name='arch') group by name, instance_name, visit_type, enabled order by count asc;
+-------------------------------+------+---------------+------------+---------+-------+
|              now              | name | instance_name | visit_type | enabled | count |
+-------------------------------+------+---------------+------------+---------+-------+
| 2023-11-10 16:07:30.073042+00 | arch | arch          | arch       | t       |   263 |
+-------------------------------+------+---------------+------------+---------+-------+
(1 row)

Time: 29.870 ms

mentioned in commit swh/infra/ci-cd/swh-charts@74c06dff

They algo got scheduled [1]

And are getting ingested. But it seems there are issues as most (if not all origins get a checksum mismatch [2].

It seems the image is missing some compression tools which fails the ingestion. [3] [4] [5] I've deactivate the loader for now. And i'm upgrading the image.

[1]

scheduler-schedule-recurrent INFO:swh.scheduler.celery_backend.recurrent_visits:arch: 263 visits scheduled in queue swh.loader.package.arch.tasks.LoadArch

[2]

swh-cassandra/loader-arch-85c4967dfd-rt2m4[loaders]: {"asctime": "2023-11-10 16:14:01,464", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/package/loader.py", "lineno": 706, "funcName": "load", "task_name": null, "task_id": null, "name": "swh.loader.package.loader", "levelname": "ERROR", "message": "Failed to load branch releases/3.9.1-2/python-3.9.1-2-x86_64.pkg.tar.zst for https://archlinux.org/packages/core/x86_64/python", "exc_info": "Traceback (most recent call last):\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/loader/package/loader.py\", line 691, in load\n    res = self._load_release(p_info, origin)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/loader/package/loader.py\", line 876, in _load_release\n    dl_artifacts = self.download_package(p_info, tmpdir)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/loader/package/loader.py\", line 420, in download_package\n    download(\n  File \"/opt/swh/.local/lib/python3.10/site-packages/tenacity/__init__.py\", line 289, in wrapped_f\n    return self(f, *args, **kw)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/tenacity/__init__.py\", line 379, in __call__\n    do = self.iter(retry_state=retry_state)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/tenacity/__init__.py\", line 314, in iter\n    return fut.result()\n  File \"/usr/local/lib/python3.10/concurrent/futures/_base.py\", line 451, in result\n    return self.__get_result()\n  File \"/usr/local/lib/python3.10/concurrent/futures/_base.py\", line 403, in __get_result\n    raise self._exception\n  File \"/opt/swh/.local/lib/python3.10/site-packages/tenacity/__init__.py\", line 382, in __call__\n    result = fn(*args, **kwargs)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/loader/package/utils.py\", line 139, in download\n    raise ValueError(\nValueError: Failure when fetching https://archive.archlinux.org/packages/p/python/python-3.9.1-2-x86_64.pkg.tar.zst. Checksum mismatched: 32000000 != 33233037"}

[3] https://sentry.softwareheritage.org/organizations/swh/issues/107630/?project=9&query=is%3Aunresolved+unpacking&statsPeriod=14d

[4] https://sentry.softwareheritage.org/organizations/swh/issues/108247/?project=9&query=is%3Aunresolved+unpacking&statsPeriod=14d

[5] https://sentry.softwareheritage.org/organizations/swh/issues/108030/?project=9&query=is%3Aunresolved+unpacking&statsPeriod=14d

mentioned in commit swh-apps@13f19d19

mentioned in commit swh/infra/ci-cd/swh-charts@c002c374

Listing sounds about ok, still growing:

2023-11-10 17:38:45 swh-scheduler@db1:5432 λ select now(), name, instance_name, visit_type, enabled, count(*) from listed_origins lo inner join listers l on l.id=lo.lister_id where lister_id in (select id from listers where name='arch') group by name, instance_name, visit_type, enabled order by count asc;
+-------------------------------+------+---------------+------------+---------+-------+
|              now              | name | instance_name | visit_type | enabled | count |
+-------------------------------+------+---------------+------------+---------+-------+
| 2023-11-10 16:38:46.488422+00 | arch | arch          | arch       | t       | 25673 |
+-------------------------------+------+---------------+------------+---------+-------+
(1 row)

Time: 76.253 ms

changed the description

mentioned in commit swh/infra/ci-cd/swh-charts@b80f458a