-
- Downloads
Add a non-incremental sourceforge lister
Following zack's work on T735, this change introduces an actual SWH lister for SourceForge. SourceForge provides a main sitemap that lists sharded sitemaps, which themselves list pages. Each page belongs to a project (or sub-project, though those are rare), information about which can be found by querying a REST API, which gives us the list of any and all VCS used for said project. Both sitemaps and pages have a "last modified" timestamp that will be used in a future patch to implement incremental listing. More precise information can be found as inline comments or docstrings.
parent
879170a5
No related branches found
No related tags found
Showing
- setup.py 1 addition, 0 deletionssetup.py
- swh/lister/sourceforge/__init__.py 12 additions, 0 deletionsswh/lister/sourceforge/__init__.py
- swh/lister/sourceforge/lister.py 224 additions, 0 deletionsswh/lister/sourceforge/lister.py
- swh/lister/sourceforge/tasks.py 20 additions, 0 deletionsswh/lister/sourceforge/tasks.py
- swh/lister/sourceforge/tests/__init__.py 0 additions, 0 deletionsswh/lister/sourceforge/tests/__init__.py
- swh/lister/sourceforge/tests/data/adobexmp.json 1 addition, 0 deletionsswh/lister/sourceforge/tests/data/adobexmp.json
- swh/lister/sourceforge/tests/data/backapps-website.json 1 addition, 0 deletionsswh/lister/sourceforge/tests/data/backapps-website.json
- swh/lister/sourceforge/tests/data/backapps.json 1 addition, 0 deletionsswh/lister/sourceforge/tests/data/backapps.json
- swh/lister/sourceforge/tests/data/main-sitemap.xml 10 additions, 0 deletionsswh/lister/sourceforge/tests/data/main-sitemap.xml
- swh/lister/sourceforge/tests/data/mojunk.json 1 addition, 0 deletionsswh/lister/sourceforge/tests/data/mojunk.json
- swh/lister/sourceforge/tests/data/mramm.json 1 addition, 0 deletionsswh/lister/sourceforge/tests/data/mramm.json
- swh/lister/sourceforge/tests/data/os3dmodels.json 1 addition, 0 deletionsswh/lister/sourceforge/tests/data/os3dmodels.json
- swh/lister/sourceforge/tests/data/subsitemap-0.xml 58 additions, 0 deletionsswh/lister/sourceforge/tests/data/subsitemap-0.xml
- swh/lister/sourceforge/tests/data/subsitemap-1.xml 38 additions, 0 deletionsswh/lister/sourceforge/tests/data/subsitemap-1.xml
- swh/lister/sourceforge/tests/test_lister.py 180 additions, 0 deletionsswh/lister/sourceforge/tests/test_lister.py
- swh/lister/sourceforge/tests/test_tasks.py 34 additions, 0 deletionsswh/lister/sourceforge/tests/test_tasks.py
Loading
-
Some references in the commit message have been migrated:
- T735 is now swh/meta#735 (closed)
-
mentioned in merge request !398 (closed)
Please register or sign in to comment