Add forge now - Process https://git.proxmox.com/

https://archive.softwareheritage.org/admin/add-forge/request/148/

Type: gitweb

changed milestone to %Extend archive coverage [Roadmap - Collect]

added AddForgeNow label

assigned to @guillaume

On staging environment origins listing failed:

swh@swh-toolbox-75b8779f69-9rq2s:~$ swh scheduler -C $SWH_CONFIG_FILENAME \
  add-forge-now --preset staging \
      register-lister gitweb \
        instance=git.proxmox.com
Created 1 tasks

Task 33422611
  Next run: today (2023-09-26T08:24:34.549638+00:00)
  Interval: 1 day, 0:00:00
  Type: list-gitweb
  Policy: oneshot
  Args:
  Keyword args:
    enable_origins: False
    instance: 'git.proxmox.com'
    max_origins_per_page: 10
    max_pages: 3

swh@swh-toolbox-75b8779f69-cc9f6:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-listed-origins gitweb git.proxmox.com
Forge git.proxmox.com (gitweb) has 0 listed origin in the scheduler database.

{"asctime": "2023-09-26 08:40:33,710", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/lister/pattern.py", "lineno": 228, "funcName": "run", "task_name": null, "task_id": null, "name": "swh.lister.pattern", "levelname": "INFO", "message": "Max origins per page set to 10 and reached, aborting page processing"}
{"asctime": "2023-09-26 08:40:33,714", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/lister/pattern.py", "lineno": 236, "funcName": "run", "task_name": null, "task_id": null, "name": "swh.lister.pattern", "levelname": "INFO", "message": "Disabling origins before sending them to the scheduler"}
{"asctime": "2023-09-26 08:40:33,716", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/lister/pattern.py", "lineno": 357, "funcName": "send_origins", "task_name": null, "task_id": null, "name": "swh.lister.pattern", "levelname": "WARNING", "message": "Skipping invalid origin: fast: git://git.proxmox.com/git/aab.git"}
{"asctime": "2023-09-26 08:40:33,716", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/lister/pattern.py", "lineno": 357, "funcName": "send_origins", "task_name": null, "task_id": null, "name": "swh.lister.pattern", "levelname": "WARNING", "message": "Skipping invalid origin: fast: git://git.proxmox.com/git/arch-pacman.git"}
{"asctime": "2023-09-26 08:40:33,717", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/lister/pattern.py", "lineno": 357, "funcName": "send_origins", "task_name": null, "task_id": null, "name": "swh.lister.pattern", "levelname": "WARNING", "message": "Skipping invalid origin: fast: git://git.proxmox.com/git/ceph.git"}
{"asctime": "2023-09-26 08:40:33,717", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/lister/pattern.py", "lineno": 357, "funcName": "send_origins", "task_name": null, "task_id": null, "name": "swh.lister.pattern", "levelname": "WARNING", "message": "Skipping invalid origin: fast: git://git.proxmox.com/git/corosync-pve.git"}
{"asctime": "2023-09-26 08:40:33,717", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/lister/pattern.py", "lineno": 357, "funcName": "send_origins", "task_name": null, "task_id": null, "name": "swh.lister.pattern", "levelname": "WARNING", "message": "Skipping invalid origin: fast: git://git.proxmox.com/git/corosync-qdevice.git"}
{"asctime": "2023-09-26 08:40:33,718", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/lister/pattern.py", "lineno": 357, "funcName": "send_origins", "task_name": null, "task_id": null, "name": "swh.lister.pattern", "levelname": "WARNING", "message": "Skipping invalid origin: fast: git://git.proxmox.com/git/criu.git"}
{"asctime": "2023-09-26 08:40:33,718", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/lister/pattern.py", "lineno": 357, "funcName": "send_origins", "task_name": null, "task_id": null, "name": "swh.lister.pattern", "levelname": "WARNING", "message": "Skipping invalid origin: fast: git://git.proxmox.com/git/dab-pve-appliances.git"}
{"asctime": "2023-09-26 08:40:33,718", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/lister/pattern.py", "lineno": 357, "funcName": "send_origins", "task_name": null, "task_id": null, "name": "swh.lister.pattern", "levelname": "WARNING", "message": "Skipping invalid origin: fast: git://git.proxmox.com/git/dab.git"}
{"asctime": "2023-09-26 08:40:33,719", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/lister/pattern.py", "lineno": 357, "funcName": "send_origins", "task_name": null, "task_id": null, "name": "swh.lister.pattern", "levelname": "WARNING", "message": "Skipping invalid origin: fast: git://git.proxmox.com/git/efi-boot-shim.git"}
{"asctime": "2023-09-26 08:40:33,719", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/lister/pattern.py", "lineno": 357, "funcName": "send_origins", "task_name": null, "task_id": null, "name": "swh.lister.pattern", "levelname": "WARNING", "message": "Skipping invalid origin: fast: git://git.proxmox.com/git/extjs.git"}
{"asctime": "2023-09-26 08:40:33,759", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/app/trace.py", "lineno": 131, "funcName": "info", "task_name": null, "task_id": null, "name": "celery.app.trace", "levelname": "INFO", "message": "Task swh.lister.gitweb.tasks.GitwebListerTask[fda2d684-9cf9-47eb-b89e-0858949e915f] succeeded in 3.9602681705728173s: {'pages': 1, 'origins': 0}", "data": {"id": "fda2d684-9cf9-47eb-b89e-0858949e915f", "name": "swh.lister.gitweb.tasks.GitwebListerTask", "return_value": "{'pages': 1, 'origins': 0}", "runtime": 3.9602681705728173, "args": "()", "kwargs": "{'instance': 'git.proxmox.com', 'max_pages': 3, 'enable_origins': False, 'max_origins_per_page': 10}"}}

There is some string prefixes before the git clone URLs in each repository page, we must strip them to avoid such errors. Fix is incoming.

swh/devel/swh-lister!505 (merged)

Thanks.

added 30m of time spent

mentioned in issue #5050 (closed)

mentioned in commit anlambert/swh-lister@3ea5dbcf

mentioned in merge request swh/devel/swh-lister!504 (merged)

mentioned in commit anlambert/swh-lister@3fc097f7

mentioned in merge request swh/devel/swh-lister!505 (merged)

mentioned in commit anlambert/swh-lister@0b9349d6

mentioned in commit anlambert/swh-lister@8b85c415

mentioned in commit anlambert/swh-lister@59a97964

On staging environment, after the deployment of the new lister version:

swh-scheduler=> select * from task where id = 33422611;
-[ RECORD 1 ]----+-----------------------------------------------------------------------------------------------------------------------------
id               | 33422611
type             | list-gitweb
arguments        | {"args": [], "kwargs": {"instance": "git.proxmox.com", "max_pages": 3, "enable_origins": false, "max_origins_per_page": 10}}
next_run         | 2023-10-05 07:24:37.214913+00
current_interval | 1 day
status           | completed
policy           | oneshot
retries_left     | 0
priority         | 

swh-scheduler=> update task set next_run=now(), status='next_run_not_scheduled' where id = 33422611;
UPDATE 1

Check the pod's logs:

ᐅ kb --context archive-staging-rke2 logs pods/lister-gitweb-666ff6bff8-7dwgf -n swh | tail -1 | jq
Defaulted container "listers" out of: listers, prepare-configuration (init)
{
  "asctime": "2023-10-12 10:23:02,734",
  "threadName": "MainThread",
  "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/app/trace.py",
  "lineno": 131,
  "funcName": "info",
  "task_name": null,
  "task_id": null,
  "name": "celery.app.trace",
  "levelname": "INFO",
  "message": "Task swh.lister.gitweb.tasks.GitwebListerTask[2fa4235f-dd8c-4ee0-a05e-aca9859bdae6] succeeded in 4.090913348016329s: {'pages': 1, 'origins': 10}",
  "data": {
    "id": "2fa4235f-dd8c-4ee0-a05e-aca9859bdae6",
    "name": "swh.lister.gitweb.tasks.GitwebListerTask",
    "return_value": "{'pages': 1, 'origins': 10}",
    "runtime": 4.090913348016329,
    "args": "()",
    "kwargs": "{'instance': 'git.proxmox.com', 'max_pages': 3, 'enable_origins': False, 'max_origins_per_page': 10}"
  }
}

Check the listed origins:

swh@swh-toolbox-555968744f-dl6lx:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-listed-origins gitweb git.proxmox.com -l
url                                   last_seen                         last_update
-----------------------------------   --------------------------------  --------------------------------
https://git.proxmox.com/git/aab.git   2023-10-12 10:23:02.603243+00:00  2022-11-12 10:23:01.276737+00:00
[...]
https://git.proxmox.com/git/extjs.git 2023-10-12 10:23:02.603243+00:00  2023-08-12 10:23:02.584534+00:00

Forge git.proxmox.com (gitweb) has 10 listed origins in the scheduler database.

Schedule the first ingestion:

swh@swh-toolbox-555968744f-dl6lx:~$ swh scheduler -C $SWH_CONFIG_FILENAME \
  add-forge-now --preset staging \
  schedule-first-visits \
    --type-name git \
    --lister-name gitweb \
    --lister-instance-name git.proxmox.com

100 slots available in celery queue
10 visits to send to celery

added 15m of time spent

On staging environment the first ingestion is successfully completed:

swh@swh-toolbox-555968744f-dl6lx:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-ingested-origins gitweb git.proxmox.com

Forge git.proxmox.com (gitweb) has 10 scheduled ingests in the scheduler.
failed       : 0
None         : 0
not_found    : 0
successful   : 10
total        : 10
success rate : 100.0%

added 5m of time spent

On production environment:

swh@swh-toolbox-b9856b644-bbz4v:~$ swh scheduler -C $SWH_CONFIG_FILENAME \
  add-forge-now --preset production \
  register-lister gitweb \
    instance=git.proxmox.com
Created 1 tasks

Task 415845442
  Next run: today (2023-10-12T12:07:30.808831+00:00)
  Interval: 1 day, 0:00:00
  Type: list-gitweb
  Policy: recurring
  Args:
  Keyword args:
    instance: 'git.proxmox.com'

Check the listed origins:

swh@swh-toolbox-b9856b644-bbz4v:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-listed-origins gitweb git.proxmox.com -l
url                                         last_seen                         last_update
------------------------------------------  --------------------------------  --------------------------------
https://git.proxmox.com/git/aab.git         2023-10-12 12:08:06.637631+00:00  2022-11-12 12:07:40.809321+00:00

https://git.proxmox.com/git/zfsonlinux.git  2023-10-12 12:08:06.637631+00:00  2023-10-11 14:07:52.454254+00:00

Forge git.proxmox.com (gitweb) has 176 listed origins in the scheduler database.

Schedule the first ingestion:

swh@swh-toolbox-b9856b644-bbz4v:~$ swh scheduler -C $SWH_CONFIG_FILENAME \
  add-forge-now --preset production \
    schedule-first-visits \
      --type-name git \
      --lister-name gitweb \
      --lister-instance-name git.proxmox.com

10000 slots available in celery queue
176 visits to send to celery

added 10m of time spent

On production environment the first ingestion is completed with one failure:

swh@swh-toolbox-b9856b644-bbz4v:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-ingested-origins gitweb git.proxmox.com

Forge git.proxmox.com (gitweb) has 176 scheduled ingests in the scheduler.
failed       : 0
None         : 0
not_found    : 1
successful   : 175
total        : 176
success rate : 99.43181818181817%

added 5m of time spent

closed