Add forge now - Process https://annas-software.org/

https://archive.softwareheritage.org/admin/add-forge/request/48/

Type: gitlab

changed milestone to %Extend archive coverage [Roadmap - Collect]

added AddForgeNow label

assigned to @guillaume

On staging environment:

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ \
>    add-forge-now --preset staging \
>      register-lister gitlab \
>        url=https://annas-software.org/api/v4/
Created 1 tasks

Task 33421059
  Next run: today (2023-04-17T09:54:06.517379+00:00)
  Interval: 90 days, 0:00:00
  Type: list-gitlab-full
  Policy: oneshot
  Args:
  Keyword args:
    enable_origins: False
    max_origins_per_page: 10
    max_pages: 3
    url: 'https://annas-software.org/api/v4/'

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ \
>   add-forge-now --preset staging \
>   schedule-first-visits \
>     --type-name git \
>     --lister-name gitlab \
>     --lister-instance-name annas-software.org
100 slots available in celery queue
3 visits to send to celery

added 15m of time spent

The ingest of the main repository failed:

swh-scheduler=> select visit_type, url, last_visit_status from origin_visit_stats
    where visit_type='git' and
    url like 'https://annas-software.org%';
 visit_type |                                     url                                     | last_visit_status 
------------+-----------------------------------------------------------------------------+-------------------
 git        | https://annas-software.org/python-developer/annas-archive-scrapping-API.git | successful
 git        | https://annas-software.org/AnnaArchivist/annas-archive.git                  | failed
 git        | https://annas-software.org/jj/annas-archive.git                             | successful
(3 rows)

swh-scheduler=> select last_visit_status, count(ovs.url)
    from origin_visit_stats ovs
    join listed_origins lo USING(url, visit_type)
    where lister_id = (select id from listers
                       where name='gitlab' and
                       instance_name='annas-software.org')
    and visit_type='git'
    group by last_visit_status;
 last_visit_status | count 
-------------------+-------
 successful        |     2
 failed            |     1
(2 rows)

Looks like a temporary failure (network or disk space) but could not find any report on sentry. I submitted a save code now request for the repository on staging and it succeeded.

Thank. I'll add this forge in production.

added 10m of time spent

On production environment:

whscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ \
>   add-forge-now --preset production \
>   register-lister gitlab \
>     url=https://annas-software.org/api/v4/
Created 1 tasks

Task 415369288
  Next run: today (2023-04-18T12:12:59.533041+00:00)
  Interval: 90 days, 0:00:00
  Type: list-gitlab-full
  Policy: recurring
  Args:
  Keyword args:
    url: 'https://annas-software.org/api/v4/'

Created 1 tasks

Task 415369289
  Next run: tomorrow (2023-04-19T12:12:59.604117+00:00)
  Interval: 1 day, 0:00:00
  Type: list-gitlab-incremental
  Policy: recurring
  Args:
  Keyword args:
    url: 'https://annas-software.org/api/v4/'

swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ \
>   add-forge-now --preset production \
>     schedule-first-visits \
>       --type-name git \
>       --lister-name gitlab \
>       --lister-instance-name annas-software.org
10000 slots available in celery queue
3 visits to send to celery

added 15m of time spent

On production environment, all first ingests was successfully completed:

softwareheritage-scheduler=> select last_visit_status, count(ovs.url)
    from origin_visit_stats ovs
    join listed_origins lo USING(url, visit_type)
    where lister_id = (select id from listers
                       where name='gitlab' and
                       instance_name='annas-software.org')
    and visit_type='git'
    group by last_visit_status;
 last_visit_status | count 
-------------------+-------
 successful        |     3
(1 row)

https://archive.softwareheritage.org/browse/search/?q=https%3A%2F%2Fannas-software.org&with_visit=true&with_content=true

added 10m of time spent

closed