[Add Forge Now] process https://git.cworth.org
Designs
- Show closed items
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- A deleted user changed milestone to %Extend archive coverage [Roadmap - Collect]
changed milestone to %Extend archive coverage [Roadmap - Collect]
- A deleted user added AddForgeNow label
added AddForgeNow label
- Author
Request completly messed up in staging environment.
For the pipeline details, see:
https://gitlab.softwareheritage.org/swh/infra/add-forge-now-requests/-/pipelines/7054. - Author
Request successfully processed in staging environment.
For the pipeline details, see:
https://gitlab.softwareheritage.org/swh/infra/add-forge-now-requests/-/pipelines/7054. - Author
Request successfully processed in production environment.
For the pipeline details, see:
https://gitlab.softwareheritage.org/swh/infra/add-forge-now-requests/-/pipelines/7054. - Guillaume Samson reopened
reopened
- Owner
Some repositories in this forge have an URL with a
/git
path in their page, all these repositories were successfully ingested.Staging:
swh@swh-toolbox-5569576f57-ltnkq:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-ingested-origins gitweb git.cworth.org -l url last_visit_status last_visit ----------------------------------- ------------------- -------------------------------- git://git.cworth.org/apitrace failed 2024-02-11 20:42:46.891532+00:00 git://git.cworth.org/apitrace-tests failed 2024-02-11 18:41:54.378474+00:00 git://git.cworth.org/cairo-spline failed 2024-02-11 18:41:54.887589+00:00 git://git.cworth.org/empires-api failed 2024-02-11 18:46:18.074677+00:00 git://git.cworth.org/empires-html failed 2024-02-11 18:46:18.555070+00:00 git://git.cworth.org/empires-server failed 2024-02-11 18:46:19.036699+00:00 git://git.cworth.org/git/acre successful 2024-02-11 18:41:40.423222+00:00 git://git.cworth.org/git/akamaru successful 2024-02-11 18:41:47.859906+00:00 git://git.cworth.org/git/cworth.org successful 2024-02-11 18:46:07.504491+00:00 git://git.cworth.org/git/dvonn successful 2024-02-11 18:46:17.617874+00:00 Forge git.cworth.org (gitweb) has 10 scheduled ingests in the scheduler. failed : 6 None : 0 not_found : 0 successful : 4 total : 10 success rate: 40.00%
All the others repositories failed but they can be cloned by adding
/git
path:swh@swh-toolbox-5569576f57-ltnkq:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-ingested-origins gitweb git.cworth.org -l | awk '/apitrace /' git://git.cworth.org/apitrace failed 2024-02-11 20:42:46.891532+00:00 swh@swh-toolbox-5569576f57-ltnkq:~$ git clone git://git.cworth.org/git/apitrace Cloning into 'apitrace'... remote: Enumerating objects: 13913, done. remote: Counting objects: 100% (13913/13913), done. remote: Compressing objects: 100% (4441/4441), done. remote: Total 13913 (delta 10400), reused 12820 (delta 9425), pack-reused 0 Receiving objects: 100% (13913/13913), 5.47 MiB | 4.35 MiB/s, done. Resolving deltas: 100% (10400/10400), done.
All repositories could probably be loaded with a
base_git_url
parameter. - Owner
On staging environment.
Disable the former task:swh-scheduler=> begin; update task set status = 'disabled' where id = '33424223'; BEGIN UPDATE 1 swh-scheduler=*> select * from task where id = '33424223'; id | type | arguments | next_run | current_interval | status | policy | retries_left | priority ----------+-------------+-----------------------------------------------------------------------------------------------------------------------------+-------------------------------+------------------+----------+---------+--------------+---------- 33424223 | list-gitweb | {"args": [], "kwargs": {"instance": "git.cworth.org", "max_pages": 3, "enable_origins": false, "max_origins_per_page": 10}} | 2024-02-10 19:20:47.210069+00 | 1 day | disabled | oneshot | 0 | (1 row) swh-scheduler=*> commit; COMMIT
Create a new task with a
base_git_url
parameter:swh@swh-toolbox-5569576f57-ltnkq:~$ swh scheduler -C $SWH_CONFIG_FILENAME \ add-forge-now --preset staging \ register-lister gitweb \ instance=git.cworth.org \ base_git_url=git://git.cworth.org/git Created 1 tasks Task 33424240 Next run: today (2024-02-12T15:20:16.351691+00:00) Interval: 1 day, 0:00:00 Type: list-gitweb Policy: oneshot Args: Keyword args: base_git_url: 'git://git.cworth.org/git' enable_origins: False instance: 'git.cworth.org' max_origins_per_page: 10 max_pages: 3
Check the new listed origins:
swh@swh-toolbox-5569576f57-ltnkq:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-listed-origins gitweb git.cworth.org -l url last_seen last_update --------------------------------------- -------------------------------- -------------------------------- git://git.cworth.org/apitrace 2024-02-11 17:05:46.383249+00:00 2014-02-11 17:05:43.896947+00:00 git://git.cworth.org/apitrace-tests 2024-02-11 17:05:46.383249+00:00 2014-02-11 17:05:44.192923+00:00 git://git.cworth.org/cairo-spline 2024-02-11 17:05:46.383249+00:00 2013-02-11 17:05:44.469596+00:00 git://git.cworth.org/empires-api 2024-02-11 17:05:46.383249+00:00 2021-02-11 17:05:45.476396+00:00 git://git.cworth.org/empires-html 2024-02-11 17:05:46.383249+00:00 2021-02-11 17:05:45.843825+00:00 git://git.cworth.org/empires-server 2024-02-11 17:05:46.383249+00:00 2021-02-11 17:05:46.331515+00:00 git://git.cworth.org/git/acre 2024-02-12 15:24:02.561838+00:00 2014-02-12 15:23:59.793908+00:00 git://git.cworth.org/git/akamaru 2024-02-12 15:24:02.561838+00:00 2010-02-12 15:24:00.103242+00:00 git://git.cworth.org/git/apitrace 2024-02-12 15:24:02.561838+00:00 2014-02-12 15:24:00.419968+00:00 git://git.cworth.org/git/apitrace-tests 2024-02-12 15:24:02.561838+00:00 2014-02-12 15:24:00.718090+00:00 git://git.cworth.org/git/cairo-spline 2024-02-12 15:24:02.561838+00:00 2013-02-12 15:24:00.998567+00:00 git://git.cworth.org/git/cworth.org 2024-02-12 15:24:02.561838+00:00 2022-02-12 15:24:01.307456+00:00 git://git.cworth.org/git/dvonn 2024-02-12 15:24:02.561838+00:00 2021-02-12 15:24:01.616488+00:00 git://git.cworth.org/git/empires-api 2024-02-12 15:24:02.561838+00:00 2021-02-12 15:24:01.926932+00:00 git://git.cworth.org/git/empires-html 2024-02-12 15:24:02.561838+00:00 2021-02-12 15:24:02.237423+00:00 git://git.cworth.org/git/empires-server 2024-02-12 15:24:02.561838+00:00 2021-02-12 15:24:02.551115+00:00 Forge git.cworth.org (gitweb) has 16 listed origins in the scheduler database.
Reschedule an origins visit:
swh@swh-toolbox-5569576f57-ltnkq:~$ swh scheduler -C $SWH_CONFIG_FILENAME \ add-forge-now --preset staging \ schedule-first-visits \ --type-name git \ --lister-name gitweb \ --lister-instance-name git.cworth.org 1000 slots available in celery queue 16 visits to send to celery
All the origins with the path
/git
were successfully completed:swh@swh-toolbox-5569576f57-ltnkq:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-ingested-origins gitweb git.cworth.org -l url last_visit_status last_visit --------------------------------------- ------------------- -------------------------------- git://git.cworth.org/apitrace failed 2024-02-12 15:27:13.802959+00:00 git://git.cworth.org/apitrace-tests failed 2024-02-12 15:27:14.257428+00:00 git://git.cworth.org/cairo-spline failed 2024-02-12 15:27:14.737221+00:00 git://git.cworth.org/empires-api failed 2024-02-12 15:27:16.176336+00:00 git://git.cworth.org/empires-html failed 2024-02-12 15:27:16.555874+00:00 git://git.cworth.org/empires-server failed 2024-02-12 15:27:16.659844+00:00 git://git.cworth.org/git/acre successful 2024-02-12 15:27:11.052893+00:00 git://git.cworth.org/git/akamaru successful 2024-02-12 15:27:11.696886+00:00 git://git.cworth.org/git/apitrace successful 2024-02-12 15:27:07.484738+00:00 git://git.cworth.org/git/apitrace-tests successful 2024-02-12 15:27:43.521400+00:00 git://git.cworth.org/git/cairo-spline successful 2024-02-12 15:27:06.782306+00:00 git://git.cworth.org/git/cworth.org successful 2024-02-12 15:27:12.686452+00:00 git://git.cworth.org/git/dvonn successful 2024-02-12 15:27:13.331739+00:00 git://git.cworth.org/git/empires-api successful 2024-02-12 15:27:10.391129+00:00 git://git.cworth.org/git/empires-html successful 2024-02-12 15:27:16.041541+00:00 git://git.cworth.org/git/empires-server successful 2024-02-12 15:27:05.114687+00:00 Forge git.cworth.org (gitweb) has 16 scheduled ingests in the scheduler. failed : 6 None : 0 not_found : 0 successful : 10 total : 16 success rate: 62.50%
- Owner
On production environment.
Disable the former task.softwareheritage-scheduler=> begin; update task set status = 'disabled' where id = '416333728'; BEGIN UPDATE 1 softwareheritage-scheduler=*> select * from task where id = '416333728'; id | type | arguments | next_run | current_interval | status | policy | retries_left | priority -----------+-------------+--------------------------------------------------------+-------------------------------+------------------+----------+-----------+--------------+---------- 416333728 | list-gitweb | {"args": [], "kwargs": {"instance": "git.cworth.org"}} | 2024-02-12 19:06:52.204989+00 | 1 day | disabled | recurring | 0 | (1 row) softwareheritage-scheduler=*> commit; COMMIT
Create a new task with a
base_git_url
parameter:swh@swh-toolbox-5dff9594c4-6xw2w:~$ swh scheduler -C $SWH_CONFIG_FILENAME \ add-forge-now --preset production \ register-lister gitweb \ instance=git.cworth.org \ base_git_url=git://git.cworth.org/git Created 1 tasks Task 416335711 Next run: today (2024-02-12T15:43:23.820615+00:00) Interval: 1 day, 0:00:00 Type: list-gitweb Policy: recurring Args: Keyword args: base_git_url: 'git://git.cworth.org/git' instance: 'git.cworth.org'
Check the new listed origins:
swh@swh-toolbox-5dff9594c4-6xw2w:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-listed-origins gitweb git.cworth.org -l url last_seen last_update ------------------------------------------------- -------------------------------- -------------------------------- git://git.cworth.org/apitrace 2024-02-11 19:06:52.163327+00:00 2014-02-11 19:06:33.584227+00:00 git://git.cworth.org/apitrace-tests 2024-02-11 19:06:52.163327+00:00 2014-02-11 19:06:33.875537+00:00 git://git.cworth.org/cairo-spline 2024-02-11 19:06:52.163327+00:00 2013-02-11 19:06:34.150093+00:00 git://git.cworth.org/empires-api 2024-02-11 19:06:52.163327+00:00 2021-02-11 19:06:35.018298+00:00 git://git.cworth.org/empires-html 2024-02-11 19:06:52.163327+00:00 2021-02-11 19:06:35.309046+00:00 git://git.cworth.org/empires-server 2024-02-11 19:06:52.163327+00:00 2021-02-11 19:06:35.608302+00:00 git://git.cworth.org/fips 2024-02-11 19:06:52.163327+00:00 2015-02-11 19:06:35.914378+00:00 git://git.cworth.org/git/acre 2024-02-12 15:44:49.609474+00:00 2014-02-12 15:44:30.753266+00:00 git://git.cworth.org/git/akamaru 2024-02-12 15:44:49.609474+00:00 2010-02-12 15:44:31.049033+00:00 git://git.cworth.org/git/apitrace 2024-02-12 15:44:49.609474+00:00 2014-02-12 15:44:31.328979+00:00 git://git.cworth.org/git/apitrace-tests 2024-02-12 15:44:49.609474+00:00 2014-02-12 15:44:31.617976+00:00 git://git.cworth.org/git/cairo-spline 2024-02-12 15:44:49.609474+00:00 2013-02-12 15:44:31.884036+00:00 git://git.cworth.org/git/cworth.org 2024-02-12 15:44:49.609474+00:00 2022-02-12 15:44:32.171735+00:00 git://git.cworth.org/git/dvonn 2024-02-12 15:44:49.609474+00:00 2021-02-12 15:44:32.470199+00:00 git://git.cworth.org/git/empires-api 2024-02-12 15:44:49.609474+00:00 2021-02-12 15:44:32.767247+00:00 git://git.cworth.org/git/empires-html 2024-02-12 15:44:49.609474+00:00 2021-02-12 15:44:33.064762+00:00 git://git.cworth.org/git/empires-server 2024-02-12 15:44:49.609474+00:00 2021-02-12 15:44:33.359194+00:00 git://git.cworth.org/git/fips 2024-02-12 15:44:49.609474+00:00 2015-02-12 15:44:33.657179+00:00 git://git.cworth.org/git/glaze 2024-02-12 15:44:49.609474+00:00 2015-02-12 15:44:34.010047+00:00 git://git.cworth.org/git/glenv 2024-02-12 15:44:49.609474+00:00 2014-02-12 15:44:34.305670+00:00 git://git.cworth.org/git/glfps 2024-02-12 15:44:49.609474+00:00 2014-02-12 15:44:34.597792+00:00 git://git.cworth.org/git/grrobot 2024-02-12 15:44:49.609474+00:00 2021-02-12 15:44:34.897355+00:00 git://git.cworth.org/git/gzip 2024-02-12 15:44:49.609474+00:00 2010-02-12 15:44:35.184803+00:00 git://git.cworth.org/git/hello 2024-02-12 15:44:49.609474+00:00 2008-02-12 15:44:35.502037+00:00 git://git.cworth.org/git/hgbook-git 2024-02-12 15:44:49.609474+00:00 2008-02-12 15:44:35.791938+00:00 git://git.cworth.org/git/ikiwiki-bootstrap 2024-02-12 15:44:49.609474+00:00 2008-02-12 15:44:36.081913+00:00 git://git.cworth.org/git/kub 2024-02-12 15:44:49.609474+00:00 2011-02-12 15:44:36.373833+00:00 git://git.cworth.org/git/lmno-api 2024-02-12 15:44:49.609474+00:00 2021-02-12 15:44:36.875423+00:00 git://git.cworth.org/git/lmno-server 2024-02-12 15:44:49.609474+00:00 2021-02-12 15:44:37.162276+00:00 git://git.cworth.org/git/lmno-todo 2024-02-12 15:44:49.609474+00:00 2021-02-12 15:44:37.450884+00:00 git://git.cworth.org/git/lmno.games 2024-02-12 15:44:49.609474+00:00 2022-02-12 15:44:37.810968+00:00 git://git.cworth.org/git/loa 2024-02-12 15:44:49.609474+00:00 2021-02-12 15:44:38.098914+00:00 git://git.cworth.org/git/loa-java 2024-02-12 15:44:49.609474+00:00 2009-02-12 15:44:38.374894+00:00 git://git.cworth.org/git/loudgame 2024-02-12 15:44:49.609474+00:00 2009-02-12 15:44:38.662984+00:00 git://git.cworth.org/git/mnemon 2024-02-12 15:44:49.609474+00:00 2012-02-12 15:44:38.965188+00:00 git://git.cworth.org/git/nodemuch 2024-02-12 15:44:49.609474+00:00 2011-02-12 15:44:39.274066+00:00 git://git.cworth.org/git/nogit 2024-02-12 15:44:49.609474+00:00 2021-02-12 15:44:39.570717+00:00 git://git.cworth.org/git/notmuch 2024-02-12 15:44:49.609474+00:00 2024-01-15 15:44:39.924353+00:00 git://git.cworth.org/git/notmuch-web 2024-02-12 15:44:49.609474+00:00 2012-02-12 15:44:40.652786+00:00 git://git.cworth.org/git/obsolete/notmuch-to-html 2024-02-12 15:44:49.609474+00:00 2015-02-12 15:44:41.599236+00:00 git://git.cworth.org/git/obsolete/notmuch-web 2024-02-12 15:44:49.609474+00:00 2012-02-12 15:44:41.880441+00:00 git://git.cworth.org/git/puzzless 2024-02-12 15:44:49.609474+00:00 2019-02-12 15:44:42.533941+00:00 git://git.cworth.org/git/rrsolve 2024-02-12 15:44:49.609474+00:00 2004-02-12 15:44:42.833951+00:00 git://git.cworth.org/git/scherzo 2024-02-12 15:44:49.609474+00:00 2014-02-12 15:44:43.126770+00:00 git://git.cworth.org/git/spritext 2024-02-12 15:44:49.609474+00:00 2008-02-12 15:44:43.456889+00:00 git://git.cworth.org/git/sup 2024-02-12 15:44:49.609474+00:00 2010-02-12 15:44:43.731692+00:00 git://git.cworth.org/git/svg2png 2024-02-12 15:44:49.609474+00:00 2010-02-12 15:44:44.016555+00:00 git://git.cworth.org/git/tar 2024-02-12 15:44:49.609474+00:00 2011-02-12 15:44:44.313342+00:00 git://git.cworth.org/git/ttt 2024-02-12 15:44:49.609474+00:00 2008-02-12 15:44:44.609252+00:00 git://git.cworth.org/git/turbot 2024-02-12 15:44:49.609474+00:00 2023-01-12 15:44:44.955763+00:00 git://git.cworth.org/git/turbot-web 2024-02-12 15:44:49.609474+00:00 2023-01-12 15:44:45.265044+00:00 git://git.cworth.org/git/vogl 2024-02-12 15:44:49.609474+00:00 2015-02-12 15:44:45.567837+00:00 git://git.cworth.org/git/wordgame 2024-02-12 15:44:49.609474+00:00 2008-02-12 15:44:45.863350+00:00 git://git.cworth.org/git/words 2024-02-12 15:44:49.609474+00:00 2020-02-12 15:44:46.149362+00:00 git://git.cworth.org/git/xoboot 2024-02-12 15:44:49.609474+00:00 2008-02-12 15:44:46.447976+00:00 git://git.cworth.org/git/zombocom-ai 2024-02-12 15:44:49.609474+00:00 2023-01-12 15:44:46.768840+00:00 git://git.cworth.org/git/zombocom-static 2024-02-12 15:44:49.609474+00:00 None git://git.cworth.org/glaze 2024-02-11 19:06:52.163327+00:00 2015-02-11 19:06:36.254596+00:00 git://git.cworth.org/glenv 2024-02-11 19:06:52.163327+00:00 2014-02-11 19:06:36.543727+00:00 git://git.cworth.org/glfps 2024-02-11 19:06:52.163327+00:00 2014-02-11 19:06:36.836413+00:00 git://git.cworth.org/grrobot 2024-02-11 19:06:52.163327+00:00 2021-02-11 19:06:37.141717+00:00 git://git.cworth.org/lmno-api 2024-02-11 19:06:52.163327+00:00 2021-02-11 19:06:38.907299+00:00 git://git.cworth.org/lmno-server 2024-02-11 19:06:52.163327+00:00 2021-02-11 19:06:39.197703+00:00 git://git.cworth.org/lmno-todo 2024-02-11 19:06:52.163327+00:00 2021-02-11 19:06:39.490198+00:00 git://git.cworth.org/lmno.games 2024-02-11 19:06:52.163327+00:00 2022-02-11 19:06:39.784075+00:00 git://git.cworth.org/nodemuch 2024-02-11 19:06:52.163327+00:00 2011-02-11 19:06:41.521087+00:00 git://git.cworth.org/nogit 2024-02-11 19:06:52.163327+00:00 2021-02-11 19:06:41.813986+00:00 git://git.cworth.org/notmuch 2024-02-11 19:06:52.163327+00:00 2024-01-14 19:06:42.154199+00:00 git://git.cworth.org/notmuch-web 2024-02-11 19:06:52.163327+00:00 2012-02-11 19:06:42.983889+00:00 git://git.cworth.org/obsolete/notmuch-to-html 2024-02-11 19:06:52.163327+00:00 2015-02-11 19:06:43.957287+00:00 git://git.cworth.org/obsolete/notmuch-web 2024-02-11 19:06:52.163327+00:00 2012-02-11 19:06:44.232031+00:00 git://git.cworth.org/puzzless 2024-02-11 19:06:52.163327+00:00 2019-02-11 19:06:44.802390+00:00 git://git.cworth.org/rrsolve 2024-02-11 19:06:52.163327+00:00 2004-02-11 19:06:45.090455+00:00 git://git.cworth.org/scherzo 2024-02-11 19:06:52.163327+00:00 2014-02-11 19:06:45.378913+00:00 git://git.cworth.org/turbot 2024-02-11 19:06:52.163327+00:00 2023-01-11 19:06:47.116556+00:00 git://git.cworth.org/turbot-web 2024-02-11 19:06:52.163327+00:00 2023-01-11 19:06:47.482775+00:00 git://git.cworth.org/vogl 2024-02-11 19:06:52.163327+00:00 2015-02-11 19:06:47.786847+00:00 git://git.cworth.org/words 2024-02-11 19:06:52.163327+00:00 2020-02-11 19:06:48.348720+00:00 git://git.cworth.org/zombocom-ai 2024-02-11 19:06:52.163327+00:00 2023-01-11 19:06:48.965748+00:00 git://git.cworth.org/zombocom-static 2024-02-11 19:06:52.163327+00:00 None git://git.notmuchmail.org/git/notmuch 2024-02-12 15:44:49.609474+00:00 2014-02-12 15:44:41.307771+00:00 git://git.notmuchmail.org/git/notmuch-wiki 2024-02-12 15:44:49.609474+00:00 2014-02-12 15:44:42.238607+00:00 Forge git.cworth.org (gitweb) has 82 listed origins in the scheduler database.
Reschedule an ingestion:
swh@swh-toolbox-5dff9594c4-6xw2w:~$ swh scheduler -C $SWH_CONFIG_FILENAME \ add-forge-now --preset production \ schedule-first-visits \ --type-name git \ --lister-name gitweb \ --lister-instance-name git.cworth.org 10000 slots available in celery queue 82 visits to send to celery
The new origins ingestion were completed with a few failures:
swh@swh-toolbox-5dff9594c4-6xw2w:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-ingested-origins gitweb git.cworth.org Forge git.cworth.org (gitweb) has 82 scheduled ingests in the scheduler. failed : 38 None : 0 not_found : 0 successful : 44 total : 82 success rate: 53.66% swh@swh-toolbox-5dff9594c4-6xw2w:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-ingested-origins gitweb git.cworth.org -l | \ awk 'BEGIN{with_git_path_failed=0;with_git_path_successful=0;no_git_path_failed=0;no_git_path_successful=0} { if ($2 == "failed" && $1 ~ /\/git\//) with_git_path_failed+=1 ; if ($2 == "successful" && $1 ~ /\/git\//) with_git_path_successful+=1 ; if ($2 == "failed" && $1 !~ /\/git\//) no_git_path_failed+=1 ; if ($2 == "successful" && $1 !~ /\/git\//) no_git_path_successful+=1 } END{printf "\nwith_git_path_failed %6s\nwith_git_path_successful %s\nno_git_path_failed %8s\nno_git_path_successful %4s\n", with_git_path_failed, with_git_path_successful, no_git_path_failed, no_git_path_successful}' with_git_path_failed 8 with_git_path_successful 44 no_git_path_failed 30 no_git_path_successful 0
- Guillaume Samson closed
closed