Skip to content
Snippets Groups Projects
New look: Off

[Add Forge Now] process https://git.chrismorgan.info

    • Open Issue created by Ghost User @ghost

    https://archive.softwareheritage.org/admin/add-forge/request/433/
    Type: gitweb

    Designs

      Child items ...

      2. Activity

      • A deleted user changed milestone to %Extend archive coverage [Roadmap - Collect]

        changed milestone to %Extend archive coverage [Roadmap - Collect]

      • A deleted user added AddForgeNow label

        added AddForgeNow label

        • Ghost User
          Ghost User @ghost ·
          Author

          Request completly messed up in staging environment.
          For the pipeline details, see:
          https://gitlab.softwareheritage.org/swh/infra/add-forge-now-requests/-/pipelines/7732.

        • Guillaume Samson
          Guillaume Samson @guillaume ·
          Owner

          On a local environment, I tried all the combinations of url, instance and base_git_url, all failed to retrieve any origins.

        • Paul Wise
          Paul Wise @pabs ·
          Developer

          The URLs of the gitweb summaries of the repos themselves work fine with git:

          $ for repo in $(gitweb=https://git.chrismorgan.info/ ; git=https://git.chrismorgan.info/ ; curl -k -s "$gitweb?a=project_index" | cut -f1 -d ' ' | sed "s=^=$git=") ; do echo $repo ; git ls-remote $repo | head -n1 ; echo ; done
https://git.chrismorgan.info/symlink
dcc8849be3e8599c398361a487d22829307bef14	HEAD

https://git.chrismorgan.info/crev-proofs
e8294f5cd2b5b579a43e1a87430451e160d8b6bc	HEAD

https://git.chrismorgan.info/tesid
0000000d63631779a143778dc09275292ef8ed75	HEAD

https://git.chrismorgan.info/sanitise-file-name
1ebdf9a8317b7d1dd6e66dc59b751eba2eec727c	HEAD

https://git.chrismorgan.info/gitweb
c83ccd7eeefbc7b43977a7542ba6b0ac87dfe1aa	HEAD

https://git.chrismorgan.info/u-plus
20220401431af94d10883592520af737ba151a9f	HEAD

https://git.chrismorgan.info/verhoeff
58d8934f5550910a19d02afd420ea2cdbdf299f1	HEAD

https://git.chrismorgan.info/anymap
2e9a570491664eea18ad61d98aa1c557d5e23e67	HEAD

https://git.chrismorgan.info/remote-dom-vm
f7c340413724802fa755c8947d0e74845477c94f	HEAD

https://git.chrismorgan.info/human-string-filler
a02907505ed1b65bb08d6a77c3b9676ab6c07d02	HEAD
        • Paul Wise
          Paul Wise @pabs ·
          Developer

          Cloning all these URLs with git works fine too.

        • Paul Wise
          Paul Wise @pabs ·
          Developer

          @guillaume could you retry this with the URLs suggested above?

        • Please register or sign in to reply
      • Guillaume Samson
        Guillaume Samson @guillaume ·
        Owner

        Even with a base_git_url parameter, the lister-gitweb fails to retrieve the origins:

        swh@9f93399fecbb:~$ swh scheduler \
  add-forge-now --preset staging \
    register-lister gitweb \
      instance=git.chrismorgan.info \
      base_git_url=https://git.chrismorgan.info
Created 1 tasks

Task 3
  Next run: today (2025-03-07T09:43:21.904940+00:00)
  Interval: 1 day, 0:00:00
  Type: list-gitweb
  Policy: oneshot
  Args:
  Keyword args:
    base_git_url: 'https://git.chrismorgan.info'
    enable_origins: False
    instance: 'git.chrismorgan.info'
    max_origins_per_page: 5
    max_pages: 2
        swh-lister-1  | [2025-03-07 09:43:25,555: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/remote-dom-vm with params None
swh-lister-1  | [2025-03-07 09:43:26,003: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/remote-dom-vm
swh-lister-1  | [2025-03-07 09:43:26,003: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/tesid with params None
swh-lister-1  | [2025-03-07 09:43:26,459: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/tesid
swh-lister-1  | [2025-03-07 09:43:26,459: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/gitweb with params None
swh-lister-1  | [2025-03-07 09:43:26,921: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/gitweb
swh-lister-1  | [2025-03-07 09:43:26,922: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/crev-proofs with params None
swh-lister-1  | [2025-03-07 09:43:27,374: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/crev-proofs
swh-lister-1  | [2025-03-07 09:43:27,375: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/anymap with params None
swh-lister-1  | [2025-03-07 09:43:27,839: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/anymap
swh-lister-1  | [2025-03-07 09:43:27,839: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/human-string-filler with params None
swh-lister-1  | [2025-03-07 09:43:28,287: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/human-string-filler
swh-lister-1  | [2025-03-07 09:43:28,287: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/sanitise-file-name with params None
swh-lister-1  | [2025-03-07 09:43:28,734: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/sanitise-file-name
swh-lister-1  | [2025-03-07 09:43:28,734: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/symlink with params None
swh-lister-1  | [2025-03-07 09:43:29,180: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/symlink
swh-lister-1  | [2025-03-07 09:43:29,180: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/u-plus with params None
swh-lister-1  | [2025-03-07 09:43:29,627: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/u-plus
swh-lister-1  | [2025-03-07 09:43:29,628: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/verhoeff with params None
swh-lister-1  | [2025-03-07 09:43:30,073: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/verhoeff
swh-lister-1  | [2025-03-07 09:43:30,073: INFO/ForkPoolWorker-2] Disabling origins before sending them to the scheduler
swh-lister-1  | [2025-03-07 09:43:30,087: INFO/ForkPoolWorker-2] Task swh.lister.gitweb.tasks.GitwebListerTask[cc8d8738-e3c4-4bed-b9be-866bd19e753f] succeeded in 5.7395192169997244s: {'pages': 1, 'origins': 0}
        swh@9f93399fecbb:~$ swh scheduler -C $SWH_CONFIG_FILENAME \
  add-forge-now --preset staging \
    register-lister gitweb \
      url=https://git.chrismorgan.info \
      base_git_url=https://git.chrismorgan.info
Created 1 tasks

Task 4
  Next run: today (2025-03-07T09:46:03.138384+00:00)
  Interval: 1 day, 0:00:00
  Type: list-gitweb
  Policy: oneshot
  Args:
  Keyword args:
    base_git_url: 'https://git.chrismorgan.info'
    enable_origins: False
    max_origins_per_page: 5
    max_pages: 2
    url: 'https://git.chrismorgan.info'
        swh-lister-1  | [2025-03-07 09:46:07,841: INFO/MainProcess] Task swh.lister.gitweb.tasks.GitwebListerTask[138eaed1-0fa7-40c0-b1b0-d15474a146aa] received
swh-lister-1  | [2025-03-07 09:46:07,843: DEBUG/ForkPoolWorker-2] Loading config file /srv/softwareheritage/config.yml
swh-lister-1  | [2025-03-07 09:46:07,850: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info with params None
swh-lister-1  | [2025-03-07 09:46:08,948: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/remote-dom-vm with params None
swh-lister-1  | [2025-03-07 09:46:09,389: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/remote-dom-vm
swh-lister-1  | [2025-03-07 09:46:09,389: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/tesid with params None
swh-lister-1  | [2025-03-07 09:46:09,829: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/tesid
swh-lister-1  | [2025-03-07 09:46:09,830: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/gitweb with params None
swh-lister-1  | [2025-03-07 09:46:10,287: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/gitweb
swh-lister-1  | [2025-03-07 09:46:10,287: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/crev-proofs with params None
swh-lister-1  | [2025-03-07 09:46:10,731: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/crev-proofs 
swh-lister-1  | [2025-03-07 09:46:10,731: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/anymap with params None
swh-lister-1  | [2025-03-07 09:46:11,197: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/anymap
swh-lister-1  | [2025-03-07 09:46:11,197: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/human-string-filler with params None
swh-lister-1  | [2025-03-07 09:46:11,656: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/human-string-filler
swh-lister-1  | [2025-03-07 09:46:11,656: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/sanitise-file-name with params None
swh-lister-1  | [2025-03-07 09:46:12,101: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/sanitise-file-name
swh-lister-1  | [2025-03-07 09:46:12,101: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/symlink with params None
swh-lister-1  | [2025-03-07 09:46:12,543: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/symlink
swh-lister-1  | [2025-03-07 09:46:12,543: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/u-plus with params None
swh-lister-1  | [2025-03-07 09:46:12,994: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/u-plus
swh-lister-1  | [2025-03-07 09:46:12,995: DEBUG/ForkPoolWorker-2] Fetching URL https://git.chrismorgan.info/verhoeff with params None
swh-lister-1  | [2025-03-07 09:46:13,439: DEBUG/ForkPoolWorker-2] No git urls found on https://git.chrismorgan.info/verhoeff
swh-lister-1  | [2025-03-07 09:46:13,439: INFO/ForkPoolWorker-2] Disabling origins before sending them to the scheduler
swh-lister-1  | [2025-03-07 09:46:13,452: INFO/ForkPoolWorker-2] Task swh.lister.gitweb.tasks.GitwebListerTask[138eaed1-0fa7-40c0-b1b0-d15474a146aa] succeeded in 5.609464245000709s: {'pages': 1, 'origins': 0}

        And cloning seems to fail too:

        /tmp ᐅ git clone https://git.chrismorgan.info/remote-dom-vm                                              
Cloning into 'remote-dom-vm'...
fatal: unable to access 'https://git.chrismorgan.info/remote-dom-vm/': The requested URL returned error: 500
/tmp ᐅ git clone https://git.chrismorgan.info/anymap                                          
Cloning into 'anymap'...
fatal: unable to access 'https://git.chrismorgan.info/anymap/': The requested URL returned error: 500

        So ingesting these origins with SCN fails:

        ᐅ kbp logs loader-save-code-now-7f4f5cfdc8-cp8hx -n swh-cassandra -c loaders | awk 'END{print}' | jq
{
  "asctime": "2025-03-07 09:58:32,338",
  "threadName": "MainThread",
  "pathname": "/opt/swh/venv/lib/python3.11/site-packages/celery/app/trace.py",
  "lineno": 128,
  "funcName": "info",
  "task_name": null,
  "task_id": null,
  "name": "celery.app.trace",
  "levelname": "INFO",
  "message": "Task swh.loader.git.tasks.UpdateGitRepository[e042b316-2179-45ca-a91f-d32e96b7205b] succeeded in 1.2533790124580264s: {'status': 'failed', 'error': 'unexpected http resp 500 for https://git.chrismorgan.info/anymap/info/refs?service=git-upload-pack'}",
  "data": {
    "id": "e042b316-2179-45ca-a91f-d32e96b7205b",
    "name": "swh.loader.git.tasks.UpdateGitRepository",
    "return_value": "{'status': 'failed', 'error': 'unexpected http resp 500 for https://git.chrismorgan.info/anymap/info/refs?service=git-upload-pack'}",
    "runtime": 1.2533790124580264,
    "args": "()",
    "kwargs": "{'url': 'https://git.chrismorgan.info/anymap'}"
  }
}
        Edited by Guillaume Samson
      Please register or sign in to reply