Ingest pub.dev (Dart, Flutter)

added Archive coverage PubDev lister PubDev loader labels

Lister runs on Docker report

Lister runs fine on Docker and found 33564 origins. It takes less than 20s to complete, it runs fast and without errors.

Lister succeeded in 14.662720984983025s: {'pages': 1, 'origins': 33564}

swh-scheduler=# select count(*) from listed_origins where visit_type = 'pubdev';
 count 
-------
 33564

Loader runs on Docker report

Loader runs fine on docker. I've first launched 100 and then 1900 loader tasks, it complete in less than an hour.

swh-scheduler=# select count(*) from origin_visit_stats where visit_type='pubdev' and last_visit_status='successful';
 count 
-------
  1450

swh-scheduler=# select count(*) from origin_visit_stats where visit_type='pubdev' and last_visit_status='failed';
 count 
-------
   550

changed the description

marked the checklist item swh/devel/swh-lister!290 (closed): Document lister as completed

marked the checklist item swh/devel/swh-loader-core!309 (closed): Document loader as completed

marked the checklist item #4465 (closed): Lister run in docker as completed

added priority:Normal label

changed the description

! In #4465 (closed), @franckbret wrote: Loader runs on Docker report

Loader runs fine on docker. I've first launched 100 and then 1900 loader tasks, it complete in less than an hour.
swh-scheduler=# select count(*) from origin_visit_stats where visit_type='pubdev' and last_visit_status='successful';
 count 
-------
  1450

swh-scheduler=# select count(*) from origin_visit_stats where visit_type='pubdev' and last_visit_status='failed';
 count 
-------
   550

swh/devel/swh-loader-core!312 (closed) fix the large amount of failed visits. I've launched 10000 tasks with following results:


swh-scheduler=# select count(*) from origin_visit_stats where visit_type='pubdev' and last_visit_status='successful';
 count 
-------
  9977
(1 row)

swh-scheduler=# select count(*) from origin_visit_stats where visit_type='pubdev' and last_visit_status='failed';                                                       
 count 
-------
    23
(1 row)

marked the checklist item #4465 (closed): Loader run in docker as completed

marked this issue as related to swh/infra/sysadm-environment#4517 (closed)

changed the description

marked this issue as related to #4628 (closed)

marked this issue as related to swh/devel/swh-lister#4665 (closed)

marked the checklist item Call for public review as completed

Public review successfully validated, ready for production :)

changed milestone to %Extend archive coverage [Roadmap - Collect]

Ingest pub.dev (Dart, Flutter)

Child items ...

Activity