Skip to content

Add Forge Now - Process https://gitlab.univ-eiffel.fr/

https://archive.softwareheritage.org/admin/add-forge/request/23/

staging:

  • Upgrade to python3-swh.scheduler 1.4.0
  • Restart gunicorn-swh-scheduler (for that new version ^)
  • Trigger add-forge-now listing [1]
  • Ensure listing happened [2]
  • Schedule those origins for ingestion [3]
  • Ensure ingestion went fine [4]

production:

  • Upgrade to python3-swh.scheduler 1.4.0
  • Restart gunicorn-swh-scheduler
  • Trigger add-forge-now listing [5]
  • Ensure listing happened [6]
  • Schedule those origins for ingestion [7]
  • Ensure ingestion went fine [8]

Annex:

  • D8957: Fix documentation papercuts

[1]

>   add-forge-now --preset staging \
>     register-lister gitlab \
>       url=https://gitlab.univ-eiffel.fr/api/v4

Created 1 tasks

Task 33420091
  Next run: today (2022-12-14T13:57:03.880511+00:00)
  Interval: 90 days, 0:00:00
  Type: list-gitlab-full
  Policy: oneshot
  Args:
  Keyword args:
    enable_origins: False
    max_origins_per_page: 10
    max_pages: 3
    url: 'https://gitlab.univ-eiffel.fr/api/v4'

[2]

2022-12-14 15:02:04 swh-scheduler@db1:5432 λ select arguments from task where type='list-gitlab-full' limit 2;
+----------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                  arguments                                                                   |
+----------------------------------------------------------------------------------------------------------------------------------------------+
| {"args": [], "kwargs": {"url": "https://gitlab.inria.fr/api/v4/", "instance": "inria"}}                                                      |
| {"args": [], "kwargs": {"url": "https://gitlab.univ-eiffel.fr/api/v4", "max_pages": 3, "enable_origins": false, "max_origins_per_page": 10}} |
+----------------------------------------------------------------------------------------------------------------------------------------------+
(2 rows)

Time: 8.678 ms


2022-12-14 15:02:57 swh-scheduler@db1:5432 λ select lister_id, url, visit_type from listed_origins
swh-scheduler λ where lister_id = (select id from listers
swh-scheduler λ                    where name='gitlab'
swh-scheduler λ                    and instance_name='gitlab.univ-eiffel.fr');
+--------------------------------------+------------------------------------------------------------------+------------+
|              lister_id               |                               url                                | visit_type |
+--------------------------------------+------------------------------------------------------------------+------------+
| 4d4a96f8-6d39-4da9-8fc1-3248d46324a3 | https://gitlab.univ-eiffel.fr/claire.bruna-rosso/mega.git        | git        |
| 4d4a96f8-6d39-4da9-8fc1-3248d46324a3 | https://gitlab.univ-eiffel.fr/derycke/fiches.git                 | git        |
| 4d4a96f8-6d39-4da9-8fc1-3248d46324a3 | https://gitlab.univ-eiffel.fr/eco7/vehlib.git                    | git        |
| 4d4a96f8-6d39-4da9-8fc1-3248d46324a3 | https://gitlab.univ-eiffel.fr/pacien/markdown-course-website.git | git        |
+--------------------------------------+------------------------------------------------------------------+------------+
(4 rows)

Time: 382.363 ms

[3]

swhscheduler@scheduler0:~/addforgenow$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/   add-forge-now --preset staging   schedule-first-visits     --type-name git     --lister-name gitlab     --lister-instance-name gitlab.univ-eiffel.fr
100 slots available in celery queue
4 visits to send to celery

[4]

2022-12-14 15:23:09 swh-scheduler@db1:5432 λ select last_visit_status, count(ovs.url)
swh-scheduler λ     from origin_visit_stats ovs
swh-scheduler λ     join listed_origins lo USING(url, visit_type)
swh-scheduler λ     where lister_id = (select id from listers
swh-scheduler λ                        where name='gitlab' and
swh-scheduler λ                        instance_name='gitlab.univ-eiffel.fr')
swh-scheduler λ     and visit_type='git'
swh-scheduler λ     group by last_visit_status;
+-------------------+-------+
| last_visit_status | count |
+-------------------+-------+
| successful        |     4 |
+-------------------+-------+
(1 row)

Time: 2064.165 ms (00:02.064)

[5]

swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ \
>   add-forge-now \
>     register-lister gitlab \
>       url=https://gitlab.univ-eiffel.fr/api/v4
Created 1 tasks

Task 415262958
  Next run: today (2022-12-14T14:29:57.079968+00:00)
  Interval: 90 days, 0:00:00
  Type: list-gitlab-full
  Policy: recurring
  Args:
  Keyword args:
    url: 'https://gitlab.univ-eiffel.fr/api/v4'

Created 1 tasks

Task 415262959
  Next run: tomorrow (2022-12-15T14:29:57.107647+00:00)
  Interval: 1 day, 0:00:00
  Type: list-gitlab-incremental
  Policy: recurring
  Args:
  Keyword args:
    url: 'https://gitlab.univ-eiffel.fr/api/v4'

[6]

2022-12-14 15:30:45 softwareheritage-scheduler@belvedere:5432 λ select lister_id, url, visit_type from listed_origins
where lister_id = (select id from listers
                   where name='gitlab'
                   and instance_name='gitlab.univ-eiffel.fr');
+--------------------------------------+------------------------------------------------------------------+------------+
|              lister_id               |                               url                                | visit_type |
+--------------------------------------+------------------------------------------------------------------+------------+
| 09019653-55ae-4dc5-9339-6f0a12879479 | https://gitlab.univ-eiffel.fr/claire.bruna-rosso/mega.git        | git        |
| 09019653-55ae-4dc5-9339-6f0a12879479 | https://gitlab.univ-eiffel.fr/derycke/fiches.git                 | git        |
| 09019653-55ae-4dc5-9339-6f0a12879479 | https://gitlab.univ-eiffel.fr/eco7/vehlib.git                    | git        |
| 09019653-55ae-4dc5-9339-6f0a12879479 | https://gitlab.univ-eiffel.fr/pacien/markdown-course-website.git | git        |
+--------------------------------------+------------------------------------------------------------------+------------+
(4 rows)

Time: 396.130 ms

[7]

swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ \
>   add-forge-now \
>   schedule-first-visits \
>     --type-name git \
>     --lister-name gitlab \
>     --lister-instance-name gitlab.univ-eiffel.fr
10000 slots available in celery queue
4 visits to send to celery

[8]

2022-12-14 15:36:08 softwareheritage-scheduler@belvedere:5432 λ select last_visit_status, count(ovs.url)
    from origin_visit_stats ovs
    join listed_origins lo USING(url, visit_type)
    where lister_id = (select id from listers
                       where name='gitlab' and
                       instance_name='gitlab.univ-eiffel.fr')
    and visit_type='git'
    group by last_visit_status;
+-------------------+-------+
| last_visit_status | count |
+-------------------+-------+
| successful        |     4 |
+-------------------+-------+
(1 row)

Time: 115949.005 ms (01:55.949)
Edited by Antoine R. Dumont