Skip to content

Add Forge Now - Process https://gitlab.univ-lille.fr/

https://archive.softwareheritage.org/admin/add-forge/request/16/

staging:

  • Schedule oneshot forge instance listing [1]
  • Disable origins listed
  • Schedule some origins for ingestion
  • Stop once some passed (meaning purge rabbitmq queue) [2]

production (same m.o. than ^):

  • Schedule recurring forge instance listing [3]
  • Schedule origins for ingestion [4]
  • Ensure ingestion started [5] [6]

[1]

listers [2022-11-08 14:39:04,751: INFO/ForkPoolWorker-1] Task swh.lister.gitlab.tasks.IncrementalGitLabLister[28ad46c2-89fc-4d8d-91fa-6c6c50774b8a] succeeded in 197.7044994729804s: {'pages': 17, 'origins': 1592}

[2]

loaders [2022-11-08 14:42:45,892: INFO/MainProcess] Task swh.loader.git.tasks.UpdateGitRepository[9a313d94-0307-4fda-8343-fcd47d557300] received
loaders [2022-11-08 14:42:47,323: INFO/ForkPoolWorker-1] Load origin 'https://gitlab.univ-lille.fr/pierre.boulet/test-pb.git' with type 'git'
loaders Enumerating objects: 3, done.
loaders Total 3 (delta 0), reused 0 (delta 0), pack-reused 3
loaders [2022-11-08 14:42:48,197: INFO/ForkPoolWorker-1] Listed 2 refs for repo https://gitlab.univ-lille.fr/pierre.boulet/test-pb.git
loaders [2022-11-08 14:42:49,246: INFO/ForkPoolWorker-1] Fetched 4 objects; 4 are new
loaders [2022-11-08 14:42:49,346: INFO/ForkPoolWorker-1] Task swh.loader.git.tasks.UpdateGitRepository[9a313d94-0307-4fda-8343-fcd47d557300] succeeded in 3.3280571810901165s: {'status': 'eventful'}
loaders [2022-11-08 14:42:49,349: INFO/MainProcess] Task swh.loader.git.tasks.UpdateGitRepository[1e8b83d9-a9d7-48c9-be91-5158d060f5c8] received
loaders [2022-11-08 14:42:49,790: INFO/ForkPoolWorker-1] Load origin 'https://gitlab.univ-lille.fr/Mickael.Bertainchant/gitlab-pages-test.git' with type 'git'
loaders Enumerating objects: 7, done.
loaders Total 7 (delta 0), reused 0 (delta 0), pack-reused 7
loaders [2022-11-08 14:42:50,273: INFO/ForkPoolWorker-1] Listed 2 refs for repo https://gitlab.univ-lille.fr/Mickael.Bertainchant/gitlab-pages-test.git
loaders [2022-11-08 14:42:50,484: INFO/ForkPoolWorker-1] Fetched 8 objects; 0 are new
loaders [2022-11-08 14:42:50,590: INFO/ForkPoolWorker-1] Task swh.loader.git.tasks.UpdateGitRepository[1e8b83d9-a9d7-48c9-be91-5158d060f5c8] succeeded in 1.236720732995309s: {'status': 'eventful'}
loaders [2022-11-08 14:42:50,593: INFO/MainProcess] Task swh.loader.git.tasks.UpdateGitRepository[a5f73ce3-aa4b-4656-87a1-ebab0091effc] received
loaders [2022-11-08 14:42:53,102: INFO/ForkPoolWorker-1] Load origin 'https://gitlab.univ-lille.fr/marius.bilasco/test.git' with type 'git'
loaders Enumerating objects: 3, done.
loaders Total 3 (delta 0), reused 0 (delta 0), pack-reused 3
loaders [2022-11-08 14:42:53,942: INFO/ForkPoolWorker-1] Listed 2 refs for repo https://gitlab.univ-lille.fr/marius.bilasco/test.git
loaders [2022-11-08 14:42:54,465: INFO/ForkPoolWorker-1] Fetched 4 objects; 3 are new
loaders [2022-11-08 14:42:54,603: INFO/ForkPoolWorker-1] Task swh.loader.git.tasks.UpdateGitRepository[a5f73ce3-aa4b-4656-87a1-ebab0091effc] succeeded in 4.00705074891448s: {'status': 'eventful'}
loaders [2022-11-08 14:42:54,606: INFO/MainProcess] Task swh.loader.git.tasks.UpdateGitRepository[d4de0bd2-2dcd-4ff0-9b20-8c50adc29daa] received
loaders [2022-11-08 14:42:54,879: INFO/ForkPoolWorker-1] Load origin 'https://gitlab.univ-lille.fr/marius.bilasco/oftfea_for_review.git' with type 'git'

[3] recurring policy by default

swhscheduler@saatchi:~/addforgenow$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ task add list-gitlab-incremental url=https://gitlab.univ-lille.fr/api/v4/
Created 1 tasks

Task 415258832
  Next run: today (2022-11-08T14:45:32.187761+00:00)
  Interval: 1 day, 0:00:00
  Type: list-gitlab-incremental
  Policy: recurring
  Args:
  Keyword args:
    url: 'https://gitlab.univ-lille.fr/api/v4/'

[4]

15:46:32 softwareheritage-scheduler@belvedere:5432=> select now(), count(distinct url) from listed_origins where lister_id = ( select id from listers where name='gitlab' and instance_name='gitlab.univ-lille.fr') ;
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-11-08 14:47:05.726665+00 |  1592 |
+-------------------------------+-------+
(1 row)

Time: 297.619 ms
swhscheduler@saatchi:~/addforgenow$ ./gitlab-univ-lille-fr.sh
Tue Nov  8 14:47:13 UTC 2022 scheduling git origins with policy never_visited_oldest_update_first to queue add_forge_now:swh.loader.git.tasks.UpdateGitRepository for lister gitlab.univ-lille.fr (tablesample 1)
10000 slots available in celery queue
1592 visits to send to celery

[5]

loaders [2022-11-08 14:47:58,916: INFO/MainProcess] Task swh.loader.git.tasks.UpdateGitRepository[a26468e0-3764-4ddc-a453-9ac767a25d0e] received
loaders [2022-11-08 14:47:59,043: INFO/MainProcess] Task swh.loader.git.tasks.UpdateGitRepository[6a017715-23aa-423f-a70d-ba6fe55c331a] received
loaders [2022-11-08 14:47:59,353: INFO/ForkPoolWorker-1] Load origin 'https://gitlab.univ-lille.fr/pierre.boulet/test-pb.git' with type 'git'
loaders Enumerating objects: 3, done.
loaders Total 3 (delta 0), reused 0 (delta 0), pack-reused 3
loaders [2022-11-08 14:47:59,936: INFO/ForkPoolWorker-1] Listed 2 refs for repo https://gitlab.univ-lille.fr/pierre.boulet/test-pb.git
loaders [2022-11-08 14:48:00,554: INFO/ForkPoolWorker-1] Fetched 4 objects; 4 are new
loaders [2022-11-08 14:48:00,680: INFO/ForkPoolWorker-1] Task swh.loader.git.tasks.UpdateGitRepository[a26468e0-3764-4ddc-a453-9ac767a25d0e] succeeded in 1.5879477080889046s: {'status': 'eventful'}
loaders [2022-11-08 14:48:00,690: INFO/MainProcess] Task swh.loader.git.tasks.UpdateGitRepository[ca7195b8-4ba8-4a92-ab6c-770073b5b336] received
loaders [2022-11-08 14:48:00,910: INFO/ForkPoolWorker-1] Load origin 'https://gitlab.univ-lille.fr/Mickael.Bertainchant/gitlab-pages-test.git' with type 'git'
loaders Enumerating objects: 7, done.
loaders Total 7 (delta 0), reused 0 (delta 0), pack-reused 7
loaders [2022-11-08 14:48:01,170: INFO/ForkPoolWorker-1] Listed 2 refs for repo https://gitlab.univ-lille.fr/Mickael.Bertainchant/gitlab-pages-test.git
loaders [2022-11-08 14:48:01,264: INFO/ForkPoolWorker-1] Fetched 8 objects; 0 are new
loaders [2022-11-08 14:48:01,323: INFO/ForkPoolWorker-1] Task swh.loader.git.tasks.UpdateGitRepository[6a017715-23aa-423f-a70d-ba6fe55c331a] succeeded in 0.6349385678768158s: {'status': 'eventful'}
loaders [2022-11-08 14:48:01,335: INFO/MainProcess] Task swh.loader.git.tasks.UpdateGitRepository[06adcec0-7a3f-4936-8bca-60ad7f558d1a] received
loaders [2022-11-08 14:48:01,538: INFO/ForkPoolWorker-1] Load origin 'https://gitlab.univ-lille.fr/lahoucine.elaidous.etu/java-web.git' with type 'git'
loaders Enumerating objects: 794, done.
loaders Total 794 (delta 0), reused 0 (delta 0), pack-reused 794
loaders [2022-11-08 14:48:02,442: INFO/ForkPoolWorker-1] Listed 2 refs for repo https://gitlab.univ-lille.fr/lahoucine.elaidous.etu/java-web.git
loaders [2022-11-08 14:48:05,467: INFO/ForkPoolWorker-1] Fetched 795 objects; 0 are new
loaders [2022-11-08 14:48:05,559: INFO/ForkPoolWorker-1] Task swh.loader.git.tasks.UpdateGitRepository[ca7195b8-4ba8-4a92-ab6c-770073b5b336] succeeded in 4.22518645785749s: {'status': 'eventful'}
loaders [2022-11-08 14:48:05,577: INFO/MainProcess] Task swh.loader.git.tasks.UpdateGitRepository[84fb3b24-f490-431d-8067-795bd2da2307] received
loaders [2022-11-08 14:48:05,767: INFO/ForkPoolWorker-1] Load origin 'https://gitlab.univ-lille.fr/marius.bilasco/test.git' with type 'git'
loaders Enumerating objects: 3, done.
loaders Total 3 (delta 0), reused 0 (delta 0), pack-reused 3
loaders [2022-11-08 14:48:06,030: INFO/ForkPoolWorker-1] Listed 2 refs for repo https://gitlab.univ-lille.fr/marius.bilasco/test.git
loaders [2022-11-08 14:48:06,544: INFO/ForkPoolWorker-1] Fetched 4 objects; 2 are new
loaders [2022-11-08 14:48:06,623: INFO/ForkPoolWorker-1] Task swh.loader.git.tasks.UpdateGitRepository[06adcec0-7a3f-4936-8bca-60ad7f558d1a] succeeded in 1.0529507556930184s: {'status': 'eventful'}

[6]

15:53:48 softwareheritage-scheduler@belvedere:5432=> select now(), last_visit_status, count(ovs.url) from origin_visit_stats ovs join listed_origins lo on lo.url = ovs.url and lo.visit_type = ovs.visit_type where lister_id = (select id from listers where name='gitlab' and instance_name='gitlab.univ-lille.fr') group by last_visit_status;
+-------------------------------+-------------------+-------+
|              now              | last_visit_status | count |
+-------------------------------+-------------------+-------+
| 2022-11-08 15:09:04.339708+00 | successful        |   159 |
| 2022-11-08 15:09:04.339708+00 | (null)            |  1433 |
+-------------------------------+-------------------+-------+
(2 rows)

Time: 185654.043 ms (03:05.654)
Edited by Antoine R. Dumont