Skip to content
Snippets Groups Projects
New look: Off

[Add Forge Now] process https://git.basedflare.com

    • Open Issue created by Ghost User @ghost

    https://archive.softwareheritage.org/admin/add-forge/request/824/
    Type: gitea

    Designs

      Child items 0

      2. No child items are currently assigned. Use child items to break down this issue into smaller parts.

      Activity

      • A deleted user changed milestone to %Extend archive coverage [Roadmap - Collect]

        changed milestone to %Extend archive coverage [Roadmap - Collect]

      • A deleted user added AddForgeNow label

        added AddForgeNow label

        • Ghost User
          Ghost User @ghost ·
          Author

          Request completly messed up in staging environment.
          For the pipeline details, see:
          https://gitlab.softwareheritage.org/swh/infra/add-forge-now-requests/-/pipelines/11289.

        • Guillaume Samson
          Guillaume Samson @guillaume ·
          Owner

          Gitea lister failed (there is a bot check):

          ᐅ stern --context archive-staging-rke2 -n swh-cassandra -l app=lister-gitea --only-log-lines --no-follow --tail 4
lister-gitea-56b4f69696-ndcqw listers {"asctime": "2024-10-10 08:38:26,272", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/worker/strategy.py", "lineno": 161, "funcName": "task_message_handler", "task_name": null, "task_id": null, "name": "celery.worker.strategy", "levelname": "INFO", "message": "Task swh.lister.gitea.tasks.FullGiteaRelister[7ab33c00-6527-4326-bef1-fbf0d17446f5] received", "data": {"id": "7ab33c00-6527-4326-bef1-fbf0d17446f5", "name": "swh.lister.gitea.tasks.FullGiteaRelister", "args": "()", "kwargs": "{'instance': 'git.basedflare.com', 'max_pages': 2, 'enable_origins': False, 'max_origins_per_page': 5}", "eta": null}}
lister-gitea-56b4f69696-ndcqw listers {"asctime": "2024-10-10 08:38:26,454", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/lister/gogs/lister.py", "lineno": 111, "funcName": "__init__", "task_name": null, "task_id": null, "name": "swh.lister.gogs.lister", "levelname": "WARNING", "message": "No authentication token set in configuration, using anonymous mode"}
lister-gitea-56b4f69696-ndcqw listers {"asctime": "2024-10-10 08:38:26,529", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/lister/pattern.py", "lineno": 197, "funcName": "http_request", "task_name": null, "task_id": null, "name": "swh.lister.pattern", "levelname": "WARNING", "message": "Unexpected HTTP status code 403 on https://git.basedflare.com/.basedflare/bot-check?/api/v1/repos/search?limit=50&page=1: b'{\"ch\":\"311afa843333a54c7ad8b31949ce06f1#5280ec916a103c32cc39bddaaf8be17ad36a0d74eff51d85610354c1d39a9291#1728592704#e74c53ffe4efbb13a60bb4c57b7e392f2fb32c70dfa9ec9d08ff215c43c58ecb\",\"ca\":false,\"pow\":\"argon2#3#1#512\"}'"}
lister-gitea-56b4f69696-ndcqw listers {"asctime": "2024-10-10 08:38:26,584", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/app/trace.py", "lineno": 267, "funcName": "_log_error", "task_name": null, "task_id": null, "name": "celery.app.trace", "levelname": "ERROR", "message": "Task swh.lister.gitea.tasks.FullGiteaRelister[7ab33c00-6527-4326-bef1-fbf0d17446f5] raised unexpected: HTTPError('403 Client Error: Forbidden for url: https://git.basedflare.com/.basedflare/bot-check?/api/v1/repos/search?limit=50&page=1')", "exc_info": "Traceback (most recent call last):\n  File \"/opt/swh/.local/lib/python3.10/site-packages/celery/app/trace.py\", line 453, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/sentry_sdk/utils.py\", line 1761, in runner\n    return sentry_patched_function(*args, **kwargs)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/sentry_sdk/integrations/celery/__init__.py\", line 414, in _inner\n    reraise(*exc_info)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/sentry_sdk/utils.py\", line 1690, in reraise\n    raise value\n  File \"/opt/swh/.local/lib/python3.10/site-packages/sentry_sdk/integrations/celery/__init__.py\", line 409, in _inner\n    return f(*args, **kwargs)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/scheduler/task.py\", line 61, in __call__\n    result = super().__call__(*args, **kwargs)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/celery/app/trace.py\", line 736, in __protected_call__\n    return self.run(*args, **kwargs)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/lister/gitea/tasks.py\", line 16, in list_gitea_full\n    return lister.run().dict()\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/lister/pattern.py\", line 219, in run\n    for page in self.get_pages():\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/lister/gogs/lister.py\", line 162, in get_pages\n    body, links = self.page_request(\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/lister/gogs/lister.py\", line 130, in page_request\n    response = self.http_request(url, params=params)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/tenacity/__init__.py\", line 336, in wrapped_f\n    return copy(f, *args, **kw)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/tenacity/__init__.py\", line 475, in __call__\n    do = self.iter(retry_state=retry_state)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/tenacity/__init__.py\", line 376, in iter\n    result = action(retry_state)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/tenacity/__init__.py\", line 398, in <lambda>\n    self._add_action_func(lambda rs: rs.outcome.result())\n  File \"/usr/local/lib/python3.10/concurrent/futures/_base.py\", line 451, in result\n    return self.__get_result()\n  File \"/usr/local/lib/python3.10/concurrent/futures/_base.py\", line 403, in __get_result\n    raise self._exception\n  File \"/opt/swh/.local/lib/python3.10/site-packages/tenacity/__init__.py\", line 478, in __call__\n    result = fn(*args, **kwargs)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/lister/pattern.py\", line 203, in http_request\n    response.raise_for_status()\n  File \"/opt/swh/.local/lib/python3.10/site-packages/requests/models.py\", line 1024, in raise_for_status\n    raise HTTPError(http_error_msg, response=self)\nrequests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://git.basedflare.com/.basedflare/bot-check?/api/v1/repos/search?limit=50&page=1", "data": {"hostname": "lister@lister-gitea-56b4f69696-ndcqw", "id": "7ab33c00-6527-4326-bef1-fbf0d17446f5", "name": "swh.lister.gitea.tasks.FullGiteaRelister", "exc": "HTTPError('403 Client Error: Forbidden for url: https://git.basedflare.com/.basedflare/bot-check?/api/v1/repos/search?limit=50&page=1')", "traceback": "Traceback (most recent call last):\n  File \"/opt/swh/.local/lib/python3.10/site-packages/celery/app/trace.py\", line 453, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/sentry_sdk/utils.py\", line 1761, in runner\n    return sentry_patched_function(*args, **kwargs)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/sentry_sdk/integrations/celery/__init__.py\", line 414, in _inner\n    reraise(*exc_info)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/sentry_sdk/utils.py\", line 1690, in reraise\n    raise value\n  File \"/opt/swh/.local/lib/python3.10/site-packages/sentry_sdk/integrations/celery/__init__.py\", line 409, in _inner\n    return f(*args, **kwargs)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/scheduler/task.py\", line 61, in __call__\n    result = super().__call__(*args, **kwargs)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/celery/app/trace.py\", line 736, in __protected_call__\n    return self.run(*args, **kwargs)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/lister/gitea/tasks.py\", line 16, in list_gitea_full\n    return lister.run().dict()\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/lister/pattern.py\", line 219, in run\n    for page in self.get_pages():\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/lister/gogs/lister.py\", line 162, in get_pages\n    body, links = self.page_request(\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/lister/gogs/lister.py\", line 130, in page_request\n    response = self.http_request(url, params=params)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/tenacity/__init__.py\", line 336, in wrapped_f\n    return copy(f, *args, **kw)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/tenacity/__init__.py\", line 475, in __call__\n    do = self.iter(retry_state=retry_state)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/tenacity/__init__.py\", line 376, in iter\n    result = action(retry_state)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/tenacity/__init__.py\", line 398, in <lambda>\n    self._add_action_func(lambda rs: rs.outcome.result())\n  File \"/usr/local/lib/python3.10/concurrent/futures/_base.py\", line 451, in result\n    return self.__get_result()\n  File \"/usr/local/lib/python3.10/concurrent/futures/_base.py\", line 403, in __get_result\n    raise self._exception\n  File \"/opt/swh/.local/lib/python3.10/site-packages/tenacity/__init__.py\", line 478, in __call__\n    result = fn(*args, **kwargs)\n  File \"/opt/swh/.local/lib/python3.10/site-packages/swh/lister/pattern.py\", line 203, in http_request\n    response.raise_for_status()\n  File \"/opt/swh/.local/lib/python3.10/site-packages/requests/models.py\", line 1024, in raise_for_status\n    raise HTTPError(http_error_msg, response=self)\nrequests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://git.basedflare.com/.basedflare/bot-check?/api/v1/repos/search?limit=50&page=1\n", "args": "()", "kwargs": "{'instance': 'git.basedflare.com', 'max_pages': 2, 'enable_origins': False, 'max_origins_per_page': 5}", "description": "raised unexpected", "internal": false}}
          ᐅ curl -I https://git.basedflare.com                                                                            
HTTP/2 302 
content-length: 0
location: /.basedflare/bot-check?/
cache-control: no-cache

ᐅ curl -I "https://git.basedflare.com/.basedflare/bot-check?/"                                                   
HTTP/2 403 
content-type: text/html; charset=utf-8
content-length: 0
x-cache-status: HIT
        • Paul Wise
          Paul Wise @pabs ·
          Developer

          The bot check page requires a proof-of-work challenge, either via JavaScript, or running an argon2 Linux command, pasting the result and accepting a cookie in response.

          Example of the Linux command-line:

          echo "Q0g9IiQyIjtCPSQocHJpbnRmIDAlLjBzICQoc2VxIDEgJDUpKTtlY2hvICJXb3JraW5nLi4uIjtJPTA7d2hpbGUgdHJ1ZTsgZG8gSD0kKGVjaG8gLW4gJENIJEkgfCBhcmdvbjIgJDEgLWlkIC10ICQ2IC1rICQ3IC1wIDEgLWwgMzIgLXIpO0U9JHtIOjA6JDV9O1tbICRFID09ICRCIF1dICYmIGVjaG8gIk91dHB1dDoiICYmIGVjaG8gJDEjJDIjJDMjJDQjJEkgJiYgZXhpdCAwOygoSSsrKSk7ZG9uZTsK" | base64 -d | bash -s 338e98a8e1c6115b8972e7dfb60acc21 25f1fa27b9e496c7d38deca37263718a2850b22f828f06c99bcd5744395f1f4b 1741366570 314d6673c3f8f3ea3bdabba9a9592ed049739517e3d34eddd5c1915602006835 3 1 512
        • Paul Wise
          Paul Wise @pabs ·
          Developer

          After base64 decoding and de-onelinering, the input to bash looks like this:

          CH="$2"
B=$(printf 0%.0s $(seq 1 $5))
echo "Working..."
I=0
while true; do
  H=$(echo -n $CH$I | argon2 $1 -id -t $6 -k $7 -p 1 -l 32 -r)
  E=${H:0:$5}
  [[ $E == $B ]] &&
    echo "Output:" &&
    echo $1#$2#$3#$4#$I &&
    exit 0
  ((I++))
done
          Edited by Paul Wise
        • Please register or sign in to reply
      Please register or sign in to reply