[Add Forge Now] process https://sourceware.org
Activity
- A deleted user
changed milestone to %Extend archive coverage [Roadmap - Collect]
- A deleted user
added AddForgeNow label
Request completly messed up in staging environment.
For the pipeline details, see:
https://gitlab.softwareheritage.org/swh/infra/add-forge-now-requests/-/pipelines/8187.On staging environment:
- create a task with an
urlparameter:
swh@swh-toolbox-b78d9ff5f-qkrtl:~$ swh scheduler -C $SWH_CONFIG_FILENAME \ add-forge-now --preset staging \ register-lister gitweb \ url=https://sourceware.org/git Created 1 tasks Task 33424707 Next run: today (2024-04-02T13:47:23.066145+00:00) Interval: 1 day, 0:00:00 Type: list-gitweb Policy: oneshot Args: Keyword args: enable_origins: False max_origins_per_page: 5 max_pages: 2 url: 'https://sourceware.org/git'- check the listed origins:
swh@swh-toolbox-b78d9ff5f-qkrtl:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-listed-origins gitweb sourceware.org -l url last_seen last_update --------------------------------------------- -------------------------------- -------------------------------- https://sourceware.org/git/abidb.git 2024-04-02 13:49:52.608365+00:00 2024-02-13 13:49:51.111616+00:00 https://sourceware.org/git/annobin.git 2024-04-02 13:49:52.608365+00:00 2024-04-02 11:49:51.825628+00:00 https://sourceware.org/git/annobin-htdocs.git 2024-04-02 13:49:52.608365+00:00 2024-02-02 13:49:51.469506+00:00 https://sourceware.org/git/archer.git 2024-04-02 13:49:52.608365+00:00 2020-04-02 13:49:52.202440+00:00 https://sourceware.org/git/automake.git 2024-04-02 13:49:52.608365+00:00 2008-04-02 13:49:52.601033+00:00 Forge sourceware.org (gitweb) has 5 listed origins in the scheduler database.- schedule the first ingestion:
swh@swh-toolbox-b78d9ff5f-qkrtl:~$ swh scheduler -C $SWH_CONFIG_FILENAME \ add-forge-now --preset staging \ schedule-first-visits \ --type-name git \ --lister-name gitweb \ --lister-instance-name sourceware.org 1000 slots available in celery queue 5 visits to send to celery- the first ingestion fails:
swh@swh-toolbox-b78d9ff5f-qkrtl:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-ingested-origins gitweb sourceware.org -l url last_visit_status last_visit --------------------------------------------- ------------------- -------------------------------- https://sourceware.org/git/abidb.git failed 2024-04-02 14:01:28.368914+00:00 https://sourceware.org/git/annobin.git failed 2024-04-02 14:02:21.380234+00:00 https://sourceware.org/git/annobin-htdocs.git failed 2024-04-02 14:02:50.285484+00:00 https://sourceware.org/git/archer.git failed 2024-04-02 14:00:28.047470+00:00 https://sourceware.org/git/automake.git failed 2024-04-02 14:05:39.194430+00:00 Forge sourceware.org (gitweb) has 5 scheduled ingests in the scheduler. failed : 5 None : 0 not_found : 0 successful : 0 total : 5 success rate: 0.00%- create a task with an
- Resolved by Guillaume Samson
Here are some tests from my laptop and from the
toolboxpod in the staging cluster:swh@swh-toolbox-b78d9ff5f-qkrtl:~$ git clone https://sourceware.org/git/automake.git Cloning into 'automake'... fatal: unable to access 'https://sourceware.org/git/automake.git/': Failed to connect to sourceware.org port 443: Connection refused swh@swh-toolbox-b78d9ff5f-qkrtl:~$ curl -I https://sourceware.org/git/ curl: (7) Failed to connect to sourceware.org port 443: Connection refused/tmp ᐅ git clone https://sourceware.org/git/automake.git Cloning into 'automake'... remote: Enumerating objects: 37518, done. remote: Counting objects: 100% (37518/37518), done. remote: Compressing objects: 100% (6457/6457), done. remote: Total 37518 (delta 31182), reused 37262 (delta 30951), pack-reused 0 Receiving objects: 100% (37518/37518), 7.97 MiB | 12.40 MiB/s, done. Resolving deltas: 100% (31182/31182), done. /tmp ᐅ curl -I https://sourceware.org/git/ HTTP/2 200 date: Tue, 02 Apr 2024 14:45:39 GMT server: Apache/2.4.37 (Red Hat Enterprise Linux) OpenSSL/1.1.1k mod_qos/11.74 mod_wsgi/4.6.4 Python/3.6 mod_perl/2.0.12 Perl/v5.26.3 vary: User-Agent,Accept-Encoding content-security-policy: default-src 'self' http: https: strict-transport-security: max-age=16070400 content-type: text/html; charset=utf-8 - Last reply by Guillaume Samson
After processing the listed origins with save-code-now:
swh@swh-toolbox-b78d9ff5f-qkrtl:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-ingested-origins gitweb sourceware.org -l url last_visit_status last_visit --------------------------------------------- ------------------- -------------------------------- https://sourceware.org/git/abidb.git failed 2024-04-03 07:45:15.306051+00:00 https://sourceware.org/git/annobin.git successful 2024-04-02 17:04:15.483557+00:00 https://sourceware.org/git/annobin-htdocs.git successful 2024-04-02 15:24:53.239563+00:00 https://sourceware.org/git/archer.git successful 2024-04-02 19:31:25.781092+00:00 https://sourceware.org/git/automake.git successful 2024-04-02 15:23:46.630160+00:00 Forge sourceware.org (gitweb) has 5 scheduled ingests in the scheduler. failed : 1 None : 0 not_found : 0 successful : 4 total : 5 success rate: 80.00%On production environment:
- create a new task:
swh@swh-toolbox-66489dbb5f-b76vw:~$ swh scheduler -C $SWH_CONFIG_FILENAME \ add-forge-now --preset production \ register-lister gitweb \ url=https://sourceware.org/git Created 1 tasks Task 416477977 Next run: today (2024-04-03T07:55:16.463081+00:00) Interval: 1 day, 0:00:00 Type: list-gitweb Policy: recurring Args: Keyword args: url: 'https://sourceware.org/git'- check the listed origins:
swh@swh-toolbox-66489dbb5f-b76vw:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-listed-origins gitweb sourceware.org -l url last_seen last_update --------------------------------------- -------------------------------- -------------------------------- https://sourceware.org/git/abidb.git 2024-04-03 07:56:26.305784+00:00 2024-02-14 07:55:58.782517+00:00 [...] https://sourceware.org/git/valgrind.git 2024-04-03 07:56:26.305784+00:00 2024-04-03 06:33:26.292161+00:00 Forge sourceware.org (gitweb) has 70 listed origins in the scheduler database.- schedule the first ingestion:
swh@swh-toolbox-66489dbb5f-b76vw:~$ swh scheduler -C $SWH_CONFIG_FILENAME \ add-forge-now --preset production \ schedule-first-visits \ --type-name git \ --lister-name gitweb \ --lister-instance-name sourceware.org 10000 slots available in celery queue 70 visits to send to celery- after a lots of save-code-now, the first ingestion is completed:
swh@swh-toolbox-66489dbb5f-b76vw:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-ingested-origins gitweb sourceware.org Forge sourceware.org (gitweb) has 70 scheduled ingests in the scheduler. failed : 2 None : 0 not_found : 0 successful : 68 total : 70 success rate: 97.14%- Resolved by Guillaume Samson
@guillaume some feedback from IRC that needs to be addressed:
- The connection refused errors seems to come from a combination of apache-modqos and fail2ban. modqos delays parallel requests and then when SWH tries to download really large repos (
abidb.gitandbunsendb.git) over http, that takes too long, triggering fail2ban and the connection refused errors. -
abidb.git(large dumps of Linux distro ABI info) andbunsendb.git(testsuite log archive) are not code so not interesting to SWH, so both should probably not be archived by SWH, and are both very large so probably should not be attempted to be archived by SWH, since they fail due to their size and also trigger the connection refused errors.
Could you block these from the ingestion process? This needs doing for both the sourceware cgit and gitweb instances.
Edited by Paul Wise - The connection refused errors seems to come from a combination of apache-modqos and fail2ban. modqos delays parallel requests and then when SWH tries to download really large repos (
- Last reply by Guillaume Samson
Please register or sign in to reply