Add forge now - Process https://gitlab.mbb.cnrs.fr

https://archive.softwareheritage.org/admin/add-forge/request/40/

Type: gitlab

changed milestone to %Extend archive coverage [Roadmap - Collect]

added AddForgeNow label

assigned to @guillaume

On staging environment:

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ \
>    add-forge-now --preset staging \
>      register-lister gitlab \
>        url=https://gitlab.mbb.cnrs.fr/api/v4/
Created 1 tasks

Task 33420980
  Next run: today (2023-04-07T13:06:49.393189+00:00)
  Interval: 90 days, 0:00:00
  Type: list-gitlab-full
  Policy: oneshot
  Args:
  Keyword args:
    enable_origins: False
    max_origins_per_page: 10
    max_pages: 3
    url: 'https://gitlab.mbb.cnrs.fr/api/v4/'

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ \
>   add-forge-now --preset staging \
>   schedule-first-visits \
>     --type-name git \
>     --lister-name gitlab \
>     --lister-instance-name gitlab.mbb.cnrs.fr
100 slots available in celery queue
20 visits to send to celery

added 15m of time spent

On staging environment, all ingests are completed except one who is not_found, may be because this repository is empty.

swh-scheduler=> select visit_type, url, last_visit_status from origin_visit_stats
    where visit_type='git' and
    url like 'https://gitlab.mbb.cnrs.fr%';
 visit_type |                                 url                                 | last_visit_status 
------------+---------------------------------------------------------------------+-------------------
 git        | https://gitlab.mbb.cnrs.fr/courses/ncbi_entrez.git                  | successful
 git        | https://gitlab.mbb.cnrs.fr/julien/suproj.git                        | successful
 git        | https://gitlab.mbb.cnrs.fr/julien/r-package-checker.git             | successful
 git        | https://gitlab.mbb.cnrs.fr/mbb/tp_rocks.git                         | successful
 git        | https://gitlab.mbb.cnrs.fr/macse/macse.git                          | successful
 git        | https://gitlab.mbb.cnrs.fr/julien/climprod.git                      | successful
 git        | https://gitlab.mbb.cnrs.fr/mbb/doc_user.git                         | successful
 git        | https://gitlab.mbb.cnrs.fr/jlopez/sbio.git                          | not_found
 git        | https://gitlab.mbb.cnrs.fr/lxdsingu/atelier-lxdsingu.git            | successful
 git        | https://gitlab.mbb.cnrs.fr/aduhamet/snakemake_rapidrun_obitools.git | successful
 git        | https://gitlab.mbb.cnrs.fr/edna/test_metabarcoding_data.git         | successful
 git        | https://gitlab.mbb.cnrs.fr/courses/workshop_git.git                 | successful
 git        | https://gitlab.mbb.cnrs.fr/seaconnect/annotationvariant.git         | successful
 git        | https://gitlab.mbb.cnrs.fr/francois/Infusion.git                    | successful
 git        | https://gitlab.mbb.cnrs.fr/peguerin/snakemake_rapidrun_obitools.git | successful
 git        | https://gitlab.mbb.cnrs.fr/edna/custom_reference_database.git       | successful
 git        | https://gitlab.mbb.cnrs.fr/mbb/dnabushmeat.git                      | successful
 git        | https://gitlab.mbb.cnrs.fr/mbb/modulefiles.git                      | successful
 git        | https://gitlab.mbb.cnrs.fr/francois/spamm-ref.git                   | successful
 git        | https://gitlab.mbb.cnrs.fr/jlopez/Genepop.git                       | successful
(20 rows)

swh-scheduler=> select last_visit_status, count(ovs.url)
    from origin_visit_stats ovs
    join listed_origins lo USING(url, visit_type)
    where lister_id = (select id from listers
                       where name='gitlab' and
                       instance_name='gitlab.mbb.cnrs.fr')
    and visit_type='git'
    group by last_visit_status;
 last_visit_status | count 
-------------------+-------
 successful        |    19
 not_found         |     1
(2 rows)

added 10m of time spent

On production environment:

swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ \
>   add-forge-now --preset production \
>   register-lister gitlab \
>     url=https://gitlab.mbb.cnrs.fr/api/v4/
Created 1 tasks

Task 415366935
  Next run: today (2023-04-11T09:08:45.486459+00:00)
  Interval: 90 days, 0:00:00
  Type: list-gitlab-full
  Policy: recurring
  Args:
  Keyword args:
    url: 'https://gitlab.mbb.cnrs.fr/api/v4/'

Created 1 tasks

Task 415366936
  Next run: tomorrow (2023-04-12T09:08:45.600620+00:00)
  Interval: 1 day, 0:00:00
  Type: list-gitlab-incremental
  Policy: recurring
  Args:
  Keyword args:
    url: 'https://gitlab.mbb.cnrs.fr/api/v4/'

swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ \
>   add-forge-now --preset production \
>     schedule-first-visits \
>       --type-name git \
>       --lister-name gitlab \
>       --lister-instance-name gitlab.mbb.cnrs.fr
10000 slots available in celery queue
135 visits to send to celery

added 15m of time spent

On production environment, most of the first ingests are successfully completed:

softwareheritage-scheduler=>     select last_visit_status, count(ovs.url)
    from origin_visit_stats ovs
    join listed_origins lo USING(url, visit_type)
    where lister_id = (select id from listers
                       where name='gitlab' and
                       instance_name='gitlab.mbb.cnrs.fr')
    and visit_type='git'
    group by last_visit_status;
 last_visit_status | count 
-------------------+-------
 successful        |   132
 not_found         |     3
(2 rows)

added 25m of time spent

closed