Ingest OSDN
Summary:
OSDN is a SourceForge-style forge that has been around for many years and supports tarballs, git, hg, svn and maybe also bzr. The site has 55303 projects according to the project list, each of those can contain multiple repos as well as tarballs etc. It recently became unstable after OS China acquired it from the previous Japanese owners. Requests often either take a long time to load or return 500 errors, so it will not be easy to ingest via the main site. It has an API but that will have the website issues too. On the other hand, there are also pages listing all of the git, hg and svn (but that is broken, only shows one repo) and these don't appear to have any of the problems the main site has. The clone URLs look like this and work fine:
git: git://git.osdn.net/gitroot/hengbandforosx/web.git
hg: https://hg.osdn.net/view/vim/vim
svn: https://svn.osdn.net/svnroot/zevenet/
Plan:
-
Implement lister -
Implement loader -
docker: Run lister -
docker: Run loader -
Document lister -
Document loader -
Deploy on staging -
Call for public review -
Deploy on production