packagist: Reimplement lister using new Lister API
The previous implementation was generating tasks for a non implemented Packagist loader. The new implementation extracts source repository URL, VCS type and last update date for each package referenced by Packagist and send those info to the scheduler. Packages metadata are retrieved using Packagist API endpoints whose responses are served from static files, which are guaranteed to be efficient on the Packagist side (no dymamic queries). Furthermore, subsequent listing will send the "If-Modified-Since" HTTP header to only retrieve packages metadata updated since the previous listing operation in order to save bandwidth and return only origins which might have new released versions. Closes T2991
parent
82ab96ad
No related branches found
No related tags found
Showing
- swh/lister/packagist/__init__.py 1 addition, 3 deletionsswh/lister/packagist/__init__.py
- swh/lister/packagist/lister.py 154 additions, 74 deletionsswh/lister/packagist/lister.py
- swh/lister/packagist/models.py 0 additions, 17 deletionsswh/lister/packagist/models.py
- swh/lister/packagist/tasks.py 2 additions, 2 deletionsswh/lister/packagist/tasks.py
- swh/lister/packagist/tests/conftest.py 0 additions, 26 deletionsswh/lister/packagist/tests/conftest.py
- swh/lister/packagist/tests/data/den1n_contextmenu.json 78 additions, 0 deletionsswh/lister/packagist/tests/data/den1n_contextmenu.json
- swh/lister/packagist/tests/data/https_packagist.org/packages_list.json 0 additions, 9 deletions...ckagist/tests/data/https_packagist.org/packages_list.json
- swh/lister/packagist/tests/data/ljjackson_linnworks.json 83 additions, 0 deletionsswh/lister/packagist/tests/data/ljjackson_linnworks.json
- swh/lister/packagist/tests/data/lky_wx_article.json 239 additions, 0 deletionsswh/lister/packagist/tests/data/lky_wx_article.json
- swh/lister/packagist/tests/data/spryker-eco_computop-api.json 141 additions, 0 deletions...lister/packagist/tests/data/spryker-eco_computop-api.json
- swh/lister/packagist/tests/test_lister.py 133 additions, 78 deletionsswh/lister/packagist/tests/test_lister.py
- swh/lister/packagist/tests/test_tasks.py 9 additions, 9 deletionsswh/lister/packagist/tests/test_tasks.py
Loading
-
Some references in the commit message have been migrated:
- T2991 is now #2991 (closed)
Please register or sign in to comment