Add forge now - Process https://dev.sanctum.geek.nz/

changed milestone to %Extend archive coverage [Roadmap - Collect]

On staging environment:

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ \
>   add-forge-now --preset staging \
>     register-lister cgit \
>       url=https://dev.sanctum.geek.nz/
Created 1 tasks

Task 33421022
  Next run: tomorrow (2023-04-13T12:15:41.817573+00:00)
  Interval: 1 day, 0:00:00
  Type: list-cgit
  Policy: oneshot
  Args:
  Keyword args:
    enable_origins: False
    max_origins_per_page: 10
    max_pages: 3
    url: 'https://dev.sanctum.geek.nz/'

Update listing task date:

swh-scheduler=> update task set next_run=now(), status='next_run_not_scheduled' where id=33421022;
UPDATE 1
swh-scheduler=> select * from task where id=33421022;
-[ RECORD 1 ]----+-------------------------------------------------------------------------------------------------------------------------------------
id               | 33421022
type             | list-cgit
arguments        | {"args": [], "kwargs": {"url": "https://dev.sanctum.geek.nz/", "max_pages": 3, "enable_origins": false, "max_origins_per_page": 10}}
next_run         | 2023-04-12 12:21:56.490297+00
current_interval | 1 day
status           | next_run_scheduled
policy           | oneshot
retries_left     | 0
priority         |

Then trigger first ingestion:

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ \
>   add-forge-now --preset staging \
>   schedule-first-visits \
>     --type-name git \
>     --lister-name cgit \
>     --lister-instance-name dev.sanctum.geek.nz
100 slots available in celery queue
10 visits to send to celery

added 25m of time spent

On staging environment, most first ingests are successfully completed:

swh-scheduler=> select visit_type, url, last_visit_status from origin_visit_stats
    where visit_type='git' and
    url like 'https://dev.sanctum.geek.nz%';
 visit_type |                                 url                                 | last_visit_status 
------------+---------------------------------------------------------------------+-------------------
 git        | https://dev.sanctum.geek.nz/code/adt-perl-demo.git                  | successful
 git        | https://dev.sanctum.geek.nz/code/btree.git                          | successful
 git        | https://dev.sanctum.geek.nz/code/cat.git                            | successful
 git        | https://dev.sanctum.geek.nz/code/perlobj-demo.git                   | successful
 git        | https://dev.sanctum.geek.nz/code/funcptr.git                        | successful
 git        | https://dev.sanctum.geek.nz/code/texad.git                          | successful
 git        | https://dev.sanctum.geek.nz/code/tunics.git                         | successful
 git        | https://dev.sanctum.geek.nz/code/spsh.git                           | successful
 git        | https://dev.sanctum.geek.nz/code/POE-Component-Client-WebSocket.git | successful
 git        | https://dev.sanctum.geek.nz/code/dotfiles.git                       | failed
(10 rows)

swh-scheduler=>     select last_visit_status, count(ovs.url)
    from origin_visit_stats ovs
    join listed_origins lo USING(url, visit_type)
    where lister_id = (select id from listers
                       where name='cgit' and
                       instance_name='dev.sanctum.geek.nz')
    and visit_type='git'
    group by last_visit_status;
 last_visit_status | count 
-------------------+-------
 successful        |     9
 failed            |     1
(2 rows)

added 10m of time spent

On production environment:

swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ \
>   add-forge-now --preset production \
>   register-lister cgit \
>     url=https://dev.sanctum.geek.nz/
Created 1 tasks

Task 415367342
  Next run: tomorrow (2023-04-13T12:58:09.482069+00:00)
  Interval: 64 days, 0:00:00
  Type: list-cgit
  Policy: recurring
  Args:
  Keyword args:
    url: 'https://dev.sanctum.geek.nz/'

added 5m of time spent

Schedule the first ingests:

swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ \
>   add-forge-now --preset production \
>     schedule-first-visits \
>       --type-name git \
>       --lister-name cgit \
>       --lister-instance-name dev.sanctum.geek.nz
10000 slots available in celery queue
73 visits to send to celery

added 20m of time spent

On production environment, all the first ingests are successfully completed except one:

softwareheritage-scheduler=>     select last_visit_status, count(ovs.url)
    from origin_visit_stats ovs
    join listed_origins lo USING(url, visit_type)
    where lister_id = (select id from listers
                       where name='cgit' and
                       instance_name='dev.sanctum.geek.nz')
    and visit_type='git'
    group by last_visit_status;
         last_visit_status | count 
-------------------+-------
 successful        |    82
 failed            |     1
(2 rows)

https://archive.softwareheritage.org/browse/search/?q=https%3A%2F%2Fdev.sanctum.geek.nz&with_visit=true&with_content=true

added 15m of time spent

closed

Add forge now - Process https://dev.sanctum.geek.nz/

Designs

Child items ...

Activity