Add forge now - Process https://gitlab.cri.epita.fr/cri

https://archive.softwareheritage.org/admin/add-forge/request/35/

Type: gitlab

changed milestone to %Extend archive coverage [Roadmap - Collect]

added AddForgeNow label

assigned to @guillaume

On staging environment:

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ \
>    add-forge-now --preset staging \
>      register-lister gitlab \
>        url=https://gitlab.cri.epita.fr/api/v4/
Created 1 tasks

Task 33421013
  Next run: today (2023-04-11T13:07:18.934339+00:00)
  Interval: 90 days, 0:00:00
  Type: list-gitlab-full
  Policy: oneshot
  Args:
  Keyword args:
    enable_origins: False
    max_origins_per_page: 10
    max_pages: 3
    url: 'https://gitlab.cri.epita.fr/api/v4/'

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ \
>   add-forge-now --preset staging \
>   schedule-first-visits \
>     --type-name git \
>     --lister-name gitlab \
>     --lister-instance-name gitlab.cri.epita.fr
100 slots available in celery queue
12 visits to send to celery

added 20m of time spent

On staging environment, all repositories ingests are successfully completed:

swh-scheduler=> select visit_type, url, last_visit_status from origin_visit_stats
    where visit_type='git' and
    url like 'https://gitlab.cri.epita.fr%';
 visit_type |                                 url                                  | last_visit_status 
------------+----------------------------------------------------------------------+-------------------
 git        | https://gitlab.cri.epita.fr/cri/docker/arch-cri.git                  | successful
 git        | https://gitlab.cri.epita.fr/cri/docker/ldap.git                      | successful
 git        | https://gitlab.cri.epita.fr/cri/docker/kerberos.git                  | successful
 git        | https://gitlab.cri.epita.fr/cri/packages/nuc-led-setter.git          | successful
 git        | https://gitlab.cri.epita.fr/cri/packages/machine-state.git           | successful
 git        | https://gitlab.cri.epita.fr/cri/docker/ansible.git                   | successful
 git        | https://gitlab.cri.epita.fr/cri/services/django-exam.git             | successful
 git        | https://gitlab.cri.epita.fr/cri/docker/gitolite.git                  | successful
 git        | https://gitlab.cri.epita.fr/cri/packages/face-crop.git               | successful
 git        | https://gitlab.cri.epita.fr/cri/iac/ansible/roles/creeps-gateway.git | successful
 git        | https://gitlab.cri.epita.fr/cri/services/blog.git                    | successful
 git        | https://gitlab.cri.epita.fr/cri/services/intranet.git                | successful
(12 rows)

swh-scheduler=>     select last_visit_status, count(ovs.url)
    from origin_visit_stats ovs
    join listed_origins lo USING(url, visit_type)
    where lister_id = (select id from listers
                       where name='gitlab' and
                       instance_name='gitlab.cri.epita.fr')
    and visit_type='git'
    group by last_visit_status;
 last_visit_status | count 
-------------------+-------
 successful        |    12
(1 row)

added 10m of time spent

On production environment:

swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ \
>   add-forge-now --preset production \
>   register-lister gitlab \
>     url=https://gitlab.cri.epita.fr/api/v4/
Created 1 tasks

Task 415367047
  Next run: today (2023-04-11T15:18:26.165227+00:00)
  Interval: 90 days, 0:00:00
  Type: list-gitlab-full
  Policy: recurring
  Args:
  Keyword args:
    url: 'https://gitlab.cri.epita.fr/api/v4/'

Created 1 tasks

Task 415367048
  Next run: tomorrow (2023-04-12T15:18:26.469067+00:00)
  Interval: 1 day, 0:00:00
  Type: list-gitlab-incremental
  Policy: recurring
  Args:
  Keyword args:
    url: 'https://gitlab.cri.epita.fr/api/v4/'

swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ \
>   add-forge-now --preset production \
>     schedule-first-visits \
>       --type-name git \
>       --lister-name gitlab \
>       --lister-instance-name gitlab.cri.epita.fr
10000 slots available in celery queue
102 visits to send to celery

added 15m of time spent

On production environment, most first ingests are completed:

softwareheritage-scheduler=>     select last_visit_status, count(ovs.url)
    from origin_visit_stats ovs
    join listed_origins lo USING(url, visit_type)
    where lister_id = (select id from listers
                       where name='gitlab' and
                       instance_name='gitlab.cri.epita.fr')
    and visit_type='git'
    group by last_visit_status;
 last_visit_status | count 
-------------------+-------
 successful        |   101
 not_found         |     1
(2 rows)

added 10m of time spent

closed

mentioned in commit swh/devel/swh-scheduler@6299df4d

mentioned in merge request swh/devel/swh-scheduler!343 (merged)