Add forge now - Process https://git.replicant.us/ (several forges)

https://archive.softwareheritage.org/admin/add-forge/request/20/

Warning : 7 different forges are hosted there:

• https://git.replicant.us/replicant
• https://git.replicant.us/replicant-next
• https://git.replicant.us/infrastructure
• https://git.replicant.us/contrib
• https://git.replicant.us/mirrors/AOSP
• https://git.replicant.us/mirrors/LineageOS
• https://git.replicant.us/mirrors/freesmartphone.orga (empty)

added AddForgeNow label

changed milestone to %Extend archive coverage [Roadmap - Collect]

added #4826 (closed) as child task

added #4827 (closed) as child task

added #4828 (closed) as child task

removed child task #4828 (closed)

removed child task #4827 (closed)

removed child task #4826 (closed)

changed the description

mentioned in commit swh/infra/ci-cd/swh-charts@f8484e12

staging:

• listing [1] [2]
• (ongoing) ingestion [3]

[1]

swhscheduler@scheduler0:~$ CONFIG=/etc/softwareheritage/scheduler/backend.yml
swhscheduler@scheduler0:~$ LISTER_NAME=cgit
swhscheduler@scheduler0:~$ LISTER_INSTANCE_NAME=git.replicant.us
swhscheduler@scheduler0:~$ FORGE_URL=https://$LISTER_INSTANCE_NAME/replicant
swhscheduler@scheduler0:~$ swh scheduler --config-file $CONFIG \
>   add-forge-now --preset $ENVIRONMENT \
>   register-lister $LISTER_NAME \
>       url=$FORGE_URL
Created 1 tasks

Task 33420933
  Next run: tomorrow (2023-04-04T15:31:30.392958+00:00)
  Interval: 1 day, 0:00:00
  Type: list-cgit
  Policy: oneshot
  Args:
  Keyword args:
    enable_origins: False
    max_origins_per_page: 10
    max_pages: 3
    url: 'https://git.replicant.us/replicant'

swhscheduler@scheduler0:~$ FORGE_URL=https://$LISTER_INSTANCE_NAME/replicant-next
swhscheduler@scheduler0:~$ swh scheduler --config-file $CONFIG \
>   add-forge-now --preset $ENVIRONMENT \
>   register-lister $LISTER_NAME \
>       url=$FORGE_URL
Created 1 tasks

Task 33420934
  Next run: tomorrow (2023-04-04T15:41:21.059702+00:00)
  Interval: 1 day, 0:00:00
  Type: list-cgit
  Policy: oneshot
  Args:
  Keyword args:
    enable_origins: False
    max_origins_per_page: 10
    max_pages: 3
    url: 'https://git.replicant.us/replicant-next'

swhscheduler@scheduler0:~$ FORGE_URL=https://$LISTER_INSTANCE_NAME/infrastructure
swhscheduler@scheduler0:~$ swh scheduler --config-file $CONFIG \
>   add-forge-now --preset $ENVIRONMENT \
>   register-lister $LISTER_NAME \
>       url=$FORGE_URL
Created 1 tasks

Task 33420935
  Next run: tomorrow (2023-04-04T15:42:49.364330+00:00)
  Interval: 1 day, 0:00:00
  Type: list-cgit
  Policy: oneshot
  Args:
  Keyword args:
    enable_origins: False
    max_origins_per_page: 10
    max_pages: 3
    url: 'https://git.replicant.us/infrastructure'

swhscheduler@scheduler0:~$ swh scheduler --config-file $CONFIG \
>   add-forge-now --preset $ENVIRONMENT \
>   register-lister $LISTER_NAME \
>       url=$FORGE_URL
Created 1 tasks

Task 33420935
  Next run: tomorrow (2023-04-04T15:42:49.364330+00:00)
  Interval: 1 day, 0:00:00
  Type: list-cgit
  Policy: oneshot
  Args:
  Keyword args:
    enable_origins: False
    max_origins_per_page: 10
    max_pages: 3
    url: 'https://git.replicant.us/infrastructure'

swhscheduler@scheduler0:~$ swh scheduler --config-file $CONFIG \
>   add-forge-now --preset $ENVIRONMENT \
>   register-lister $LISTER_NAME \
>       url=$FORGE_URL
Created 1 tasks

Task 33420936
  Next run: tomorrow (2023-04-04T15:44:45.193134+00:00)
  Interval: 1 day, 0:00:00
  Type: list-cgit
  Policy: oneshot
  Args:
  Keyword args:
    enable_origins: False
    max_origins_per_page: 10
    max_pages: 3
    url: 'https://git.replicant.us/infrastructure'

swhscheduler@scheduler0:~$ FORGE_URL=https://$LISTER_INSTANCE_NAME/mirrors/AOSP
swhscheduler@scheduler0:~$ swh scheduler --config-file $CONFIG \
>   add-forge-now --preset $ENVIRONMENT \
>   register-lister $LISTER_NAME \
>       url=$FORGE_URL
Created 1 tasks

Task 33420937
  Next run: tomorrow (2023-04-04T15:45:30.187832+00:00)
  Interval: 1 day, 0:00:00
  Type: list-cgit
  Policy: oneshot
  Args:
  Keyword args:
    enable_origins: False
    max_origins_per_page: 10
    max_pages: 3
    url: 'https://git.replicant.us/mirrors/AOSP'

swhscheduler@scheduler0:~$ FORGE_URL=https://$LISTER_INSTANCE_NAME/mirrors/LineageOS
swhscheduler@scheduler0:~$ swh scheduler --config-file $CONFIG \
>   add-forge-now --preset $ENVIRONMENT \
>   register-lister $LISTER_NAME \
>       url=$FORGE_URL

Created 1 tasks

Task 33420938
  Next run: tomorrow (2023-04-04T15:45:39.857669+00:00)
  Interval: 1 day, 0:00:00
  Type: list-cgit
  Policy: oneshot
  Args:
  Keyword args:
    enable_origins: False
    max_origins_per_page: 10
    max_pages: 3
    url: 'https://git.replicant.us/mirrors/LineageOS'

sql excerpt:
| 33420933 | list-cgit | {"args": [], "kwargs": {"url": "https://git.replicant.us/replicant", "max_pages": 3, "enable_origins": false, "max_origins_per_page": 10}}         | completed | 2023-04-03 15:38:10.195094+00 |
| 33420935 | list-cgit | {"args": [], "kwargs": {"url": "https://git.replicant.us/infrastructure", "max_pages": 3, "enable_origins": false, "max_origins_per_page": 10}}    | completed | 2023-04-03 15:44:15.66016+00  |
| 33420936 | list-cgit | {"args": [], "kwargs": {"url": "https://git.replicant.us/infrastructure", "max_pages": 3, "enable_origins": false, "max_origins_per_page": 10}}    | completed | 2023-04-03 15:44:56.700094+00 |
| 33420934 | list-cgit | {"args": [], "kwargs": {"url": "https://git.replicant.us/replicant-next", "max_pages": 3, "enable_origins": false, "max_origins_per_page": 10}}    | completed | 2023-04-03 15:42:26.332492+00 |
| 33420937 | list-cgit | {"args": [], "kwargs": {"url": "https://git.replicant.us/mirrors/AOSP", "max_pages": 3, "enable_origins": false, "max_origins_per_page": 10}}      | completed | 2023-04-03 15:45:52.467764+00 |
| 33420938 | list-cgit | {"args": [], "kwargs": {"url": "https://git.replicant.us/mirrors/LineageOS", "max_pages": 3, "enable_origins": false, "max_origins_per_page": 10}} | completed | 2023-04-03 15:46:30.787156+00 |

[2]

│ listers [2023-04-03 15:38:31,523: INFO/ForkPoolWorker-2] Task swh.lister.cgit.tasks.CGitListerTask[39583b02-8146-4d70-8a6e-e0f570a1f8d7] succeeded in 17.082834577187896s: {'pages': 3, 'origins': 29}

[3]

swhscheduler@scheduler0:~$ swh scheduler --config-file $CONFIG \
>   add-forge-now --preset $ENVIRONMENT \
>   schedule-first-visits \
>   --type-name git \
>   --lister-name $LISTER_NAME \
>   --lister-instance-name $LISTER_INSTANCE_NAME
100 slots available in celery queue
100 visits to send to celery

production:

• (ongoing) listing [1] [2]
• (ongoing) trigger ingestion

[1]

swhscheduler@saatchi:~$ CONFIG=/etc/softwareheritage/scheduler/backend.yml
swhscheduler@saatchi:~$ LISTER_NAME=cgit
swhscheduler@saatchi:~$ LISTER_INSTANCE_NAME=git.replicant.us
swhscheduler@saatchi:~$ ENVIRONMENT=production
swhscheduler@saatchi:~$ sleep_seconds=60;
swhscheduler@saatchi:~$ for url in replicant replicant-next infrastructure contrib mirrors/AOSP mirrors/LineageOS; do
>   FORGE_URL=https://$LISTER_INSTANCE_NAME/$url
>   swh scheduler --config-file $CONFIG \
>     add-forge-now --preset $ENVIRONMENT \
>     register-lister $LISTER_NAME \
>         url=$FORGE_URL
>   sleep $sleep_seconds;
> done
Created 1 tasks

Task 415364376
  Next run: tomorrow (2023-04-04T15:55:47.537635+00:00)
  Interval: 64 days, 0:00:00
  Type: list-cgit
  Policy: recurring
  Args:
  Keyword args:
    url: 'https://git.replicant.us/replicant'

Created 1 tasks

Task 415364377
  Next run: tomorrow (2023-04-04T15:56:48.324963+00:00)
  Interval: 64 days, 0:00:00
  Type: list-cgit
  Policy: recurring
  Args:
  Keyword args:
    url: 'https://git.replicant.us/replicant-next'

Created 1 tasks

Task 415364378
  Next run: tomorrow (2023-04-04T15:57:48.907023+00:00)
  Interval: 64 days, 0:00:00
  Type: list-cgit
  Policy: recurring
  Args:
  Keyword args:
    url: 'https://git.replicant.us/infrastructure'

Created 1 tasks

Task 415364379
  Next run: tomorrow (2023-04-04T15:58:49.511127+00:00)
  Interval: 64 days, 0:00:00
  Type: list-cgit
  Policy: recurring
  Args:
  Keyword args:
    url: 'https://git.replicant.us/contrib'

Created 1 tasks

Task 415364380
  Next run: tomorrow (2023-04-04T15:59:50.022746+00:00)
  Interval: 64 days, 0:00:00
  Type: list-cgit
  Policy: recurring
  Args:
  Keyword args:
    url: 'https://git.replicant.us/mirrors/AOSP'

Created 1 tasks

Task 415364381
  Next run: tomorrow (2023-04-04T16:00:50.612511+00:00)
  Interval: 64 days, 0:00:00
  Type: list-cgit
  Policy: recurring
  Args:
  Keyword args:
    url: 'https://git.replicant.us/mirrors/LineageOS'

[2]

root@pergamon:~# clush -b -w @swh-workers "systemctl status swh-worker@lister" | grep "cgit\.tasks"
Apr 03 15:57:16 worker03 python3[455170]: [2023-04-03 15:57:16,300: INFO/MainProcess] Received task: swh.lister.cgit.tasks.CGitListerTask[5246063b-0cda-47a4-aa0d-f9cb6d0a33e4]
Apr 03 15:58:39 worker03 python3[1400174]: [2023-04-03 15:58:39,568: INFO/ForkPoolWorker-9] Task swh.lister.cgit.tasks.CGitListerTask[5246063b-0cda-47a4-aa0d-f9cb6d0a33e4] succeeded in 83.2171219624579s: {'pages': 3, 'origins': 109}
Apr 03 16:04:33 worker04 python3[452837]: [2023-04-03 16:04:32,979: INFO/MainProcess] Received task: swh.lister.cgit.tasks.CGitListerTask[90cce5f2-06ba-4250-97a5-0c30dbb1ec21]
Apr 03 16:00:42 worker05 python3[479078]: [2023-04-03 16:00:42,105: INFO/MainProcess] Received task: swh.lister.cgit.tasks.CGitListerTask[5c82747a-8cc2-4604-97ac-ebbf26d71b97]
Apr 03 16:03:07 worker05 python3[1594858]: [2023-04-03 16:03:07,989: INFO/ForkPoolWorker-10] Task swh.lister.cgit.tasks.CGitListerTask[5c82747a-8cc2-4604-97ac-ebbf26d71b97] succeeded in 145.85710083227605s: {'pages': 1, 'origins': 36}
Apr 03 16:03:49 worker06 python3[450379]: [2023-04-03 16:03:49,897: INFO/MainProcess] Received task: swh.lister.cgit.tasks.CGitListerTask[6131a3a5-d223-48b5-b9c1-32a71397ee80]
Apr 03 16:02:41 worker07 python3[456627]: [2023-04-03 16:02:41,240: INFO/MainProcess] Received task: swh.lister.cgit.tasks.CGitListerTask[a50a12d9-63c5-48a1-8888-1574521586a0]
Apr 03 16:02:55 worker07 python3[1582403]: [2023-04-03 16:02:55,075: INFO/ForkPoolWorker-10] Task swh.lister.cgit.tasks.CGitListerTask[a50a12d9-63c5-48a1-8888-1574521586a0] succeeded in 13.803611684590578s: {'pages': 1, 'origins': 12}
Apr 03 16:03:15 worker12 python3[448607]: [2023-04-03 16:03:15,024: INFO/MainProcess] Received task: swh.lister.cgit.tasks.CGitListerTask[364b370b-889c-4762-861b-f7be24fdbdde]
Apr 03 16:05:11 worker12 python3[1477906]: [2023-04-03 16:05:11,414: INFO/ForkPoolWorker-9] Task swh.lister.cgit.tasks.CGitListerTask[364b370b-889c-4762-861b-f7be24fdbdde] succeeded in 116.37971831206232s: {'pages': 3, 'origins': 108}
Apr 01 10:57:13 worker14 python3[2941421]: [2023-04-01 10:57:13,761: INFO/MainProcess] Received task: swh.lister.cgit.tasks.CGitListerTask[e2437ec7-f8df-4c98-8474-a5b3533b9c40]
Apr 01 10:57:15 worker14 python3[3797194]: [2023-04-01 10:57:15,745: INFO/ForkPoolWorker-9] Task swh.lister.cgit.tasks.CGitListerTask[e2437ec7-f8df-4c98-8474-a5b3533b9c40] succeeded in 1.8842534399591386s: {'pages': 1, 'origins': 179}
Apr 03 09:27:12 worker16 python3[2930009]: [2023-04-03 09:27:12,309: INFO/MainProcess] Received task: swh.lister.cgit.tasks.CGitListerTask[b1c96fa5-0022-48f2-96f7-15365f2a07d3]
Apr 03 09:27:21 worker16 python3[3968944]: [2023-04-03 09:27:21,250: INFO/ForkPoolWorker-10] Task swh.lister.cgit.tasks.CGitListerTask[b1c96fa5-0022-48f2-96f7-15365f2a07d3] succeeded in 8.921204905025661s: {'pages': 1, 'origins': 32}

2023-04-03 18:08:49 softwareheritage-scheduler@belvedere:5432 λ select id, type, arguments, status, next_run from task where type='list-cgit' order by id desc limit 6;
+-----------+-----------+-------------------------------------------------------------------------------+------------------------+-------------------------------+
|    id     |   type    |                                   arguments                                   |         status         |           next_run            |
+-----------+-----------+-------------------------------------------------------------------------------+------------------------+-------------------------------+
| 415364381 | list-cgit | {"args": [], "kwargs": {"url": "https://git.replicant.us/mirrors/LineageOS"}} | next_run_not_scheduled | 2023-06-06 16:08:37.047079+00 |
| 415364380 | list-cgit | {"args": [], "kwargs": {"url": "https://git.replicant.us/mirrors/AOSP"}}      | next_run_scheduled     | 2023-04-03 16:03:45.370242+00 |
| 415364379 | list-cgit | {"args": [], "kwargs": {"url": "https://git.replicant.us/contrib"}}           | next_run_not_scheduled | 2023-06-06 16:05:11.415982+00 |
| 415364378 | list-cgit | {"args": [], "kwargs": {"url": "https://git.replicant.us/infrastructure"}}    | next_run_not_scheduled | 2023-06-06 16:02:55.077322+00 |
| 415364377 | list-cgit | {"args": [], "kwargs": {"url": "https://git.replicant.us/replicant-next"}}    | next_run_not_scheduled | 2023-06-06 16:03:07.99076+00  |
| 415364376 | list-cgit | {"args": [], "kwargs": {"url": "https://git.replicant.us/replicant"}}         | next_run_not_scheduled | 2023-06-06 15:58:39.57268+00  |
+-----------+-----------+-------------------------------------------------------------------------------+------------------------+-------------------------------+
(6 rows)

Time: 7.341 ms

[3]

swhscheduler@saatchi:~$ swh scheduler --config-file $CONFIG \
>   add-forge-now --preset $ENVIRONMENT \
>   schedule-first-visits \
>   --type-name git \
>   --lister-name $LISTER_NAME \
>   --lister-instance-name $LISTER_INSTANCE_NAME
10000 slots available in celery queue
1159 visits to send to celery
swhscheduler@saatchi:~$ swh scheduler --config-file $CONFIG   add-forge-now --preset $ENVIRONMENT   schedule-first-visits   --type-name git   --lister-name $LISTER_NAME   --lister-instance-name $LISTER_INSTANCE_NAME
10000 slots available in celery queue
345 visits to send to celery

changed the description

added 1h of time spent

status: 1342/1504

[1] archive.s.o (cgit table)

 git.replicant.us	git	1,342	

[2]

2023-04-05 11:24:12 softwareheritage-scheduler@belvedere:5432 λ select visit_type, count(*) from listed_origins where lister_id=(select id from listers where name='cgit' and instance_name='git.replicant.us') group by visit_type;
+------------+-------+
| visit_type | count |
+------------+-------+
| git        |  1504 |
+------------+-------+
(1 row)

Time: 750.544 ms

marked the checklist item https://git.replicant.us/mirrors/LineageOS as completed

marked the checklist item https://git.replicant.us/contrib as completed

marked the checklist item https://git.replicant.us/infrastructure as completed

marked the checklist item https://git.replicant.us/replicant-next as completed

marked the checklist item https://git.replicant.us/replicant as completed

assigned to @ardumont

Due to a high lag replication on the replica db, the numbers of // workers for the add-forge-now decreased. So it might take some more time to finish this ingestion.

Nonetheless most of the ingestion is done. So it'd be ok to update the archive-changelog about it now.

cc @bchauvet ^

added 15m of time spent

marked the checklist item https://git.replicant.us/mirrors/AOSP as completed

closed