Add forge now - Process https://forgemia.inra.fr/
Activity
changed milestone to %Extend archive coverage [Roadmap - Collect]
added AddForgeNow label
assigned to @guillaume
On staging environment:
swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ \ > add-forge-now --preset staging \ > register-lister gitlab \ > url=https://forgemia.inra.fr/api/v4/ Created 1 tasks Task 33420979 Next run: today (2023-04-07T09:47:16.993895+00:00) Interval: 90 days, 0:00:00 Type: list-gitlab-full Policy: oneshot Args: Keyword args: enable_origins: False max_origins_per_page: 10 max_pages: 3 url: 'https://forgemia.inra.fr/api/v4/'swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ \ > add-forge-now --preset staging \ > schedule-first-visits \ > --type-name git \ > --lister-name gitlab \ > --lister-instance-name forgemia.inra.fr 100 slots available in celery queue 30 visits to send to celeryEdited by Guillaume SamsonOn staging environment, all ingests are successfully completed except one who is not_found and one who is failed:
swh-scheduler=> select visit_type, url, last_visit_status from origin_visit_stats where visit_type='git' and url like 'https://forgemia.inra.fr%'; visit_type | url | last_visit_status ------------+------------------------------------------------------------------------------------------+------------------- git | https://forgemia.inra.fr/adminforgemia/doc-public.git | not_found git | https://forgemia.inra.fr/jean-baptiste.bohuon/test-public.git | successful git | https://forgemia.inra.fr/svdetection/popsim.git | successful git | https://forgemia.inra.fr/genotoul-bioinfo/dgenies.git | successful git | https://forgemia.inra.fr/genotoul-bioinfo/jflow.git | successful git | https://forgemia.inra.fr/genotoul-bioinfo/dgenies-viewer.git | successful git | https://forgemia.inra.fr/QTL/spell-qtl.git | successful git | https://forgemia.inra.fr/genotoul-bioinfo/jflow-toolshed.git | successful git | https://forgemia.inra.fr/guilhem.heinrich/casdar-agroecoperenne-action1.git | successful git | https://forgemia.inra.fr/millard/isocor.git | successful git | https://forgemia.inra.fr/genotoul-bioinfo/jvenn.git | successful git | https://forgemia.inra.fr/nicolas.verzelen/blockmodels4inventories.git | successful git | https://forgemia.inra.fr/siriusquality/sqcode/siriusquality-bioma-maizelai-component.git | successful git | https://forgemia.inra.fr/guilhem.heinrich/graph-exploration.git | successful git | https://forgemia.inra.fr/phenotic/publique/ordinalclustering.git | successful git | https://forgemia.inra.fr/tropo-group/tropolink.git | successful git | https://forgemia.inra.fr/in-sylva-development/in-sylva.search.app.git | successful git | https://forgemia.inra.fr/gauthier.quesnel/red-slides.git | successful git | https://forgemia.inra.fr/umr-gdec/magatt.git | successful git | https://forgemia.inra.fr/umr-1202-biogeco/snakemake_singularity_hpc.git | successful git | https://forgemia.inra.fr/jerome.mariette/ukfs.git | successful git | https://forgemia.inra.fr/urgi-is/rare-basket.git | successful git | https://forgemia.inra.fr/umrf/exploremetabar.git | successful git | https://forgemia.inra.fr/umr-astre/capiche.git | successful git | https://forgemia.inra.fr/sunrise/inference_methods.git | successful git | https://forgemia.inra.fr/edith.garot/these.git | successful git | https://forgemia.inra.fr/bibliome/alvisae-anaee.git | successful git | https://forgemia.inra.fr/umr-astre/culicoides-friction.git | successful git | https://forgemia.inra.fr/tropo-group/tropolink_frontend.git | successful git | https://forgemia.inra.fr/pappso/xtpcpp.git | failed (30 rows) swh-scheduler=> select last_visit_status, count(ovs.url) from origin_visit_stats ovs join listed_origins lo USING(url, visit_type) where lister_id = (select id from listers where name='gitlab' and instance_name='forgemia.inra.fr') and visit_type='git' group by last_visit_status; last_visit_status | count -------------------+------- successful | 28 failed | 1 not_found | 1 (3 rows)not_found repository errors:
│ loaders [2023-04-07 09:58:07,716: INFO/ForkPoolWorker-1] Load origin 'https://forgemia.inra.fr/adminforgemia/doc-public.git' with type 'git' │ loaders [2023-04-07 09:58:08,119: ERROR/ForkPoolWorker-1] Loading failure, updating to `not_found` status │ loaders Traceback (most recent call last): │ loaders File "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/git/loader.py", line 319, in fetch_data │ loaders fetch_info = self.fetch_pack_from_origin( │ loaders File "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/git/loader.py", line 237, in fetch_pack_from_origin │ loaders pack_result = client.fetch_pack( │ loaders File "/opt/swh/.local/lib/python3.10/site-packages/dulwich/client.py", line 2085, in fetch_pack │ loaders refs, server_capabilities, url = self._discover_references( │ loaders File "/opt/swh/.local/lib/python3.10/site-packages/dulwich/client.py", line 1941, in _discover_references │ loaders resp, read = self._http_request(url, headers) │ loaders File "/opt/swh/.local/lib/python3.10/site-packages/dulwich/client.py", line 2219, in _http_request │ loaders raise HTTPUnauthorized(resp.headers.get("WWW-Authenticate"), url) │ loaders dulwich.client.HTTPUnauthorized: No valid credentials provided │ loaders │ loaders During handling of the above exception, another exception occurred: │ loaders │ loaders Traceback (most recent call last): │ loaders File "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/loader.py", line 417, in load │ loaders more_data_to_fetch = self.fetch_data() │ loaders File "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/git/loader.py", line 323, in fetch_data │ loaders raise NotFound(e) │ loaders swh.loader.exception.NotFound: No valid credentials providedOn production environment:
swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ \ > add-forge-now --preset production \ > register-lister gitlab \ > url=https://forgemia.inra.fr/api/v4/ Created 1 tasks Task 415366919 Next run: today (2023-04-11T08:16:51.129125+00:00) Interval: 90 days, 0:00:00 Type: list-gitlab-full Policy: recurring Args: Keyword args: url: 'https://forgemia.inra.fr/api/v4/' Created 1 tasks Task 415366920 Next run: tomorrow (2023-04-12T08:16:51.278795+00:00) Interval: 1 day, 0:00:00 Type: list-gitlab-incremental Policy: recurring Args: Keyword args: url: 'https://forgemia.inra.fr/api/v4/'swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ \ > add-forge-now --preset production \ > schedule-first-visits \ > --type-name git \ > --lister-name gitlab \ > --lister-instance-name forgemia.inra.fr 10000 slots available in celery queue 1171 visits to send to celeryOn production environment, most of the first ingests are successfully completed:
softwareheritage-scheduler=> select last_visit_status, count(ovs.url) from origin_visit_stats ovs join listed_origins lo USING(url, visit_type) where lister_id = (select id from listers where name='gitlab' and instance_name='forgemia.inra.fr') and visit_type='git' group by last_visit_status; last_visit_status | count -------------------+------- successful | 1152 failed | 4 not_found | 15 (3 rows)
Please register or sign in to reply