gitweb: add all discovered URLs, including git://
In swh/infra/add-forge-now-requests#120 we discovered a gitweb instance that has git:// and https:// URLs on repo pages. The lister then ignores the git:// URLs and only registers the https:// URLs. The git:// URLs work but the https:// URLs do not work. So currently we cannot archive this forge.
Probably the right solution for now is to just add all the discovered URLs, so that no matter which ones work, we always are able to figure them out. The repos would get downloaded multiple times for forges where all of the URLs work.
A more advanced strategy for later would be for the gitweb lister to group all of the URLs for one repo, then the scheduler could try each of them in turn until one succeeds.
Activity
mentioned in merge request !551 (closed)
A more advanced strategy for later would be for the gitweb lister to group all of the URLs for one repo, then the scheduler could try each of them in turn until one succeeds.
I just pushed !551 (closed) in order to check the extracted clone URLs directly in the lister. For a given gitweb project, the first clonable git URL is sent to the scheduler for loading, HTTP[S] URLs being checked prior the GIT ones.
mentioned in merge request !552