staging: Deploy gitweb lister

That's also an excuse to have another round of checks on the pipeline concurrency issue (should be fixed). And the auto-deploy of the swh-next-version branch of swh-charts.

Plan:

• Tag swh.lister v5.8.0
• debian package: Loop, fix, package 'til it works [1]
• scheduler0.staging: Upgrade python3-swh.lister (depends on ^)
• scheduler0.staging: Register new task type (depends on ^)
• swh/infra/ci-cd/swh-charts!93 (diffs) swh-charts: Update docker image
• swh/infra/ci-cd/swh-charts!92 (diffs): swh-charts: Reference new lister
• argocd: Deploy
• Schedule one new instance for listing (depends on the new task type being registered)
• Checks

[1] https://jenkins.softwareheritage.org/job/swh-apps/job/build-docker-image/258/console

marked the checklist item Tag swh.lister v5.8.0. as completed

mentioned in commit swh/infra/ci-cd/swh-charts@845b1279

mentioned in merge request swh/infra/ci-cd/swh-charts!92 (merged)

changed the description

changed the description

mentioned in commit swh/infra/ci-cd/swh-charts@8a9ee2bb

marked the checklist item swh/infra/ci-cd/swh-charts!93 (diffs) swh-charts: Update docker image as completed

marked the checklist item swh/infra/ci-cd/swh-charts!92 (diffs): swh-charts: Reference new lister as completed

marked the checklist item argocd: Deploy as completed

mentioned in commit swh/devel/swh-lister@e2587e30

mentioned in commit swh/devel/swh-lister@a6d89f1e

debian package: Loop, fix, package 'til it works

See 2 previous commits in the issue history. History of builds [1]

Unstable got unstuck [1], stable build ongoing [2].

[1] https://jenkins.softwareheritage.org/view/swh-debian%20(draft)/job/debian/job/packages/job/DLS/job/gbp-buildpackage/

[2] https://jenkins.softwareheritage.org/view/swh-debian%20(draft)/job/debian/job/packages/job/DLS/job/gbp-buildpackage/253/console

[3] https://jenkins.softwareheritage.org/view/swh-debian%20(draft)/job/debian/job/packages/job/DLS/job/gbp-buildpackage/254/console

marked the checklist item debian package: Loop, fix, package 'til it works as completed

New lister packaged and installed:

Unpacking python3-swh.lister (5.8.0-2~swh1~bpo10+1) over (5.7.4-1~swh1~bpo10+1) ...
Setting up python3-regex (0.1.20190207-1) ...
Setting up python3-tzlocal (1.5.1-1) ...
Setting up python3-dateparser (0.7.1-1) ...
Setting up python3-swh.lister (5.8.0-2~swh1~bpo10+1) ...
root@scheduler0:~# dpkg -l python3-swh.lister* | grep ii
ii  python3-swh.lister        5.8.0-2~swh1~bpo10+1 all          Software Heritage Listers (bitbucket, git(lab|hub), pypi, etc...)

New gitweb task type lister registered

swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register 
...
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gitweb
INFO:swh.scheduler.cli.task_type:Create task type list-gitweb in scheduler
...

marked the checklist item scheduler0.staging: Upgrade python3-swh.lister (depends on ^) as completed

marked the checklist item scheduler0.staging: Register new task type (depends on ^) as completed

changed the description

Schedule one new instance for listing (depends on the new task type being registered)

Scheduled [1] and listing went fine [2]. Some origins got scheduled too [3]

[3]

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/   add-forge-now --preset staging   schedule-first-visits     --type-name git     --lister-name gitweb     --lister-instance-name git.distorted.org.uk
100 slots available in celery queue
10 visits to send to celery

[2]

$ kail --context archive-staging-rke2 -n swh -d lister-gitweb --since 24h | grep "Listed\|succeeded\|Fetched"
swh/lister-gitweb-8bcbfc857-q87zm[listers]: {"asctime": "2023-07-11 13:20:59,393", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/app/trace.py", "lineno": 131, "funcName": "info", "task_name": null, "task_id": null, "name": "celery.app.trace", "levelname": "INFO", "message": "Task swh.lister.gitweb.tasks.GitwebListerTask[45bf72b1-ab0f-4ed5-ad4e-428b500188a9] succeeded in 4.421983942389488s: {'pages': 1, 'origins': 10}", "data": {"id": "45bf72b1-ab0f-4ed5-ad4e-428b500188a9", "name": "swh.lister.gitweb.tasks.GitwebListerTask", "return_value": "{'pages': 1, 'origins': 10}", "runtime": 4.421983942389488, "args": "[]", "kwargs": "{'url': 'https://git.distorted.org.uk/~mdw', 'max_pages': 3, 'enable_origins': False, 'max_origins_per_page': 10}"}}

[1]

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/   add-forge-now --preset staging     register-lister gitweb       url=https://git.distorted.org.uk/~mdw
Created 1 tasks

Task 33421851
  Next run: today (2023-07-11T13:19:29.086999+00:00)
  Interval: 1 day, 0:00:00
  Type: list-gitweb
  Policy: oneshot
  Args:
  Keyword args:
    enable_origins: False
    max_origins_per_page: 10
    max_pages: 3
    url: 'https://git.distorted.org.uk/~mdw'

marked the checklist item Schedule one new instance for listing (depends on the new task type being registered) as completed

marked the checklist item Checks as completed

Ingestion is fine too

$ kail --context archive-staging-rke2 -n swh-cassandra -d loader-add-forge-now --since 24h | grep "Listed\|succeeded\|Fetched"  | cut -d':' -f2- | jq -r .message
Listed 2 refs for repo https://git.distorted.org.uk/~mdw/backup-rotation
Fetched 13 objects; 12 are new
Task swh.loader.git.tasks.UpdateGitRepository[dc864157-92d3-438c-b721-b9d451f9b792] succeeded in 8.395855624228716s: {'status': 'eventful'}
Listed 7 refs for repo https://git.distorted.org.uk/~mdw/become
Listed 5 refs for repo https://git.distorted.org.uk/~mdw/anag
Fetched 218 objects; 214 are new
Task swh.loader.git.tasks.UpdateGitRepository[5e7d551c-c938-4c79-bf58-43565047f3eb] succeeded in 27.867278615012765s: {'status': 'eventful'}
Listed 2 refs for repo https://git.distorted.org.uk/~mdw/atoms
Fetched 46 objects; 46 are new
Task swh.loader.git.tasks.UpdateGitRepository[4defb0e7-436b-4323-9b2e-be261f21c7d9] succeeded in 2.1545015489682555s: {'status': 'eventful'}
Listed 5 refs for repo https://git.distorted.org.uk/~mdw/autoys
Fetched 198 objects; 191 are new
Task swh.loader.git.tasks.UpdateGitRepository[f87efc78-f06c-4d0e-b61e-5f9fbc15b7d5] succeeded in 14.022260729689151s: {'status': 'eventful'}
Listed 4 refs for repo https://git.distorted.org.uk/~mdw/bascat
Fetched 54 objects; 54 are new

mentioned in commit swh/devel/swh-environment@df358713

closed

Listed a subset of another instance and ingestion went fine too. It's readily deployable in production.

kail --context archive-staging-rke2 -n swh-cassandra -d loader-add-forge-now --since 24h | grep "Listed\|succeeded\|Fetched"  | cut -d':' -f2- | jq -r .message
Listed 2 refs for repo https://git.distorted.org.uk/~mdw/backup-rotation
Fetched 13 objects; 12 are new
Task swh.loader.git.tasks.UpdateGitRepository[dc864157-92d3-438c-b721-b9d451f9b792] succeeded in 8.395855624228716s: {'status': 'eventful'}
Listed 7 refs for repo https://git.distorted.org.uk/~mdw/become
Listed 5 refs for repo https://git.distorted.org.uk/~mdw/anag
Fetched 218 objects; 214 are new
Task swh.loader.git.tasks.UpdateGitRepository[5e7d551c-c938-4c79-bf58-43565047f3eb] succeeded in 27.867278615012765s: {'status': 'eventful'}
Listed 2 refs for repo https://git.distorted.org.uk/~mdw/atoms
Fetched 46 objects; 46 are new
Task swh.loader.git.tasks.UpdateGitRepository[4defb0e7-436b-4323-9b2e-be261f21c7d9] succeeded in 2.1545015489682555s: {'status': 'eventful'}
Listed 5 refs for repo https://git.distorted.org.uk/~mdw/autoys
Fetched 198 objects; 191 are new
Task swh.loader.git.tasks.UpdateGitRepository[f87efc78-f06c-4d0e-b61e-5f9fbc15b7d5] succeeded in 14.022260729689151s: {'status': 'eventful'}
Listed 4 refs for repo https://git.distorted.org.uk/~mdw/bascat
Fetched 54 objects; 54 are new
Task swh.loader.git.tasks.UpdateGitRepository[2c026e93-fb9f-4cf7-9db9-b3188e79a082] succeeded in 8.820461101364344s: {'status': 'eventful'}
Listed 2 refs for repo https://git.distorted.org.uk/~mdw/bridge-toys
Listed 49 refs for repo https://git.distorted.org.uk/~mdw/adns
Fetched 40 objects; 40 are new
Task swh.loader.git.tasks.UpdateGitRepository[7877505e-1a8a-47d3-8cb8-b2c80c570230] succeeded in 3.0848254952579737s: {'status': 'eventful'}
Listed 3 refs for repo https://git.distorted.org.uk/~mdw/ca
Fetched 201 objects; 201 are new
Task swh.loader.git.tasks.UpdateGitRepository[da75a4ac-d611-4b46-b0b9-153e5f4eeb40] succeeded in 7.747681633103639s: {'status': 'eventful'}
Listed 56 refs for repo https://git.distorted.org.uk/~mdw/catacomb
Fetched 962 objects; 955 are new
Task swh.loader.git.tasks.UpdateGitRepository[1751044e-eb7b-45a6-9a89-7146e18276ac] succeeded in 74.8138900604099s: {'status': 'eventful'}
Fetched 6433 objects; 5452 are new
Task swh.loader.git.tasks.UpdateGitRepository[d9b81e3a-1731-4771-902f-15a5c56e83ef] succeeded in 324.9713585691061s: {'status': 'eventful'}
Listed 5 refs for repo git://git.shadowcat.co.uk/catagits/App-IdiotBox.git
Fetched 362 objects; 199 are new
Task swh.loader.git.tasks.UpdateGitRepository[1e5e9f19-1965-4d70-8c1f-238ea8a45264] succeeded in 8.24041585996747s: {'status': 'eventful'}
Listed 5 refs for repo git://git.shadowcat.co.uk/catagits/BackPAN-Web.git
Fetched 146 objects; 134 are new
Task swh.loader.git.tasks.UpdateGitRepository[f8afa887-da5d-493a-84f1-ce6212957456] succeeded in 15.333778845146298s: {'status': 'eventful'}
Listed 78 refs for repo git://git.shadowcat.co.uk/catagits/Catalyst-Action-REST.git
Listed 3 refs for repo git://git.shadowcat.co.uk/catagits/Catalyst-Action-Serialize-Data-Serializer.git
Fetched 3801 objects; 37 are new
Task swh.loader.git.tasks.UpdateGitRepository[d563a83b-2187-44a6-8b11-5d5dfe3e3139] succeeded in 7.32580466195941s: {'status': 'eventful'}
Listed 3 refs for repo git://git.shadowcat.co.uk/catagits/Catalyst-Authentication-Credential-FBConnect.git
Fetched 79 objects; 76 are new
Task swh.loader.git.tasks.UpdateGitRepository[ba084f3d-fa93-4e78-abdd-d504fa9f0b2f] succeeded in 1.9993559278082103s: {'status': 'eventful'}
Listed 6 refs for repo git://git.shadowcat.co.uk/catagits/Catalyst-Authentication-Credential-HTTP-Proxy.git
Fetched 277 objects; 247 are new
Task swh.loader.git.tasks.UpdateGitRepository[1d5f0713-d6e4-4e84-8abb-55c16aadb626] succeeded in 23.840192390838638s: {'status': 'eventful'}
Listed 25 refs for repo git://git.shadowcat.co.uk/catagits/Catalyst-Authentication-Credential-HTTP.git
Fetched 4674 objects; 670 are new
Task swh.loader.git.tasks.UpdateGitRepository[268a1655-b9eb-497a-9be6-9db5cd953260] succeeded in 58.90320560336113s: {'status': 'eventful'}
Listed 6 refs for repo git://git.shadowcat.co.uk/catagits/Catalyst-Authentication-Credential-OAuth.git
Fetched 101 objects; 98 are new
Task swh.loader.git.tasks.UpdateGitRepository[ba1c5558-ce95-481e-ae39-a43abe3439d9] succeeded in 11.822136281058192s: {'status': 'eventful'}
Listed 19 refs for repo git://git.shadowcat.co.uk/catagits/Catalyst-Authentication-Credential-OpenID.git
Fetched 462 objects; 114 are new
Task swh.loader.git.tasks.UpdateGitRepository[c91db74d-e099-4fd7-9005-626bfe4b9c27] succeeded in 3.362644884735346s: {'status': 'eventful'}

marked this issue as related to #4986 (closed)

mentioned in issue swh/meta#5047 (closed)

changed milestone to %Extend archive coverage [Roadmap - Collect]

added activity::Deployment label