- Jan 28, 2021
-
-
Antoine Lambert authored
Port launchpad lister to the swh.lister.pattern.Lister API. Last update date of each listed git repositories is now sent to the scheduler. The lister can work in incremental mode, only modified repositories since the last listing operation will be returned in that case. Closes T2992
-
Antoine R. Dumont authored
In effect, it just allows to add credentials to cgit, cran and pypi listers. This fixes instances of error [1] [1] https://sentry.softwareheritage.org/share/issue/2c35a9f129cf4982a2dd003a232d507a/ Related to T2998
-
Antoine R. Dumont authored
This fixes the master build [1] [1] https://jenkins.softwareheritage.org/job/DLS/job/tests/1210/console
-
Antoine R. Dumont authored
while allowing legacy lister to still run (with --legacy)
-
- Jan 27, 2021
-
-
Antoine R. Dumont authored
Drop launchpad lister from the lister to check, its test setup is more involved than the other listers. As its setup is not done in that test, it's actually connecting anonymously to the launchpad server. So remove such lister from the test. This should also fix the debian build which refuses such access [1] [1] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DLS/job/gbp-buildpackage/97/console
-
Antoine R. Dumont authored
This current test was failing in a debian chroot with connection error.
-
Antoine R. Dumont authored
The listers depending on it got ported to the new lister api.
-
Antoine R. Dumont authored
Plus that instruction is not correct in most recent requests_mock version (failing the debian build)
-
Vincent Sellier authored
Related to T2988
-
Vincent Sellier authored
Related to T2988
-
- Jan 26, 2021
-
-
Antoine Lambert authored
Port debian lister to `swh.lister.pattern.Lister` API. The new implementation will produce one instance of ListedOrigin model per package, notably containing the set of parameters expected by the debian loader. The lister is also stateful, meaning only new packages and those with new found versions since the last listing will be returned. Closes T2979
-
Tenma authored
All the fixtures declared in them are not used anymore in the tests of the listers ported to the new Lister API.
-
Antoine R. Dumont authored
The previous pagination implementation has a hard-coded limit server side [1] [1] ``` {"error":"Offset pagination has a maximum allowed offset of 50000 for requests that return objects of type Project. Remaining records can be retrieved using keyset pagination."} ``` Related to T2994
-
Antoine Lambert authored
R package last update date can be found in the "Packaged" field of package info returned by tools::CRAN_package_db(). So retrieve it and parse it as a datetime to provide as last_update parameter value in ListedOrigin model. Closes T2989
-
Antoine Lambert authored
Related to T2989
-
Antoine R. Dumont authored
-
Antoine R. Dumont authored
Related to T2987
-
- Jan 25, 2021
-
-
Antoine R. Dumont authored
-
Vincent Sellier authored
Related to T2984
-
Antoine Lambert authored
The PaginatedListedOriginList model has been updated in rDSCHb93aa5be2c2d5dc2130e1027698f3e1255052d8d and the origins field has been renamed to results.
-
Tenma authored
-
Tenma authored
-
Tenma authored
The lister is stateless and has full listing capability. It can request the Gitea API using HTTP token authentication. Rate-limiting was not encountered but is handled generically. Added support for getting repo last update date through API.
-
Tenma authored
-
Antoine R. Dumont authored
Related to T2987
-
Vincent Sellier authored
Related to T2984
-
Antoine Lambert authored
Use random credentials from the list in configuration and improve related logging messages.
-
Antoine R. Dumont authored
Related to T2987
-
Antoine R. Dumont authored
It's used in multiple module tests now.
-
Antoine R. Dumont authored
-
Antoine R. Dumont authored
Related to T2987
-
Antoine R. Dumont authored
Related to T2987
-
Antoine R. Dumont authored
Note that the current implementation will start back the new visit from the last next_page link seen (that's what is stored in the lister state to avoid computing back the url). This means that this page will be seen at least 2 times, on the first visit and on the next. This should not pose any problems as the listing is idempotent. Related to T2987
-
Antoine R. Dumont authored
Related to T2987
-
Antoine R. Dumont authored
Related to T2987
-
- Jan 22, 2021
-
-
Antoine Lambert authored
Port npm lister to `swh.lister.pattern.Lister` API. As before, the lister can be run in full or incremental mode. When using incremental mode, only new and modified packages will be returned since the last incremental listing process. Otherwise, all packages will be listed in lexicographical order. One major improvement to be noted, latest package update date is now retrieved when available and sent to scheduler database. Closes T2972
-
- Jan 21, 2021
- Jan 20, 2021
-
-
Tenma authored
The new lister has only full listing capability. It scrapes pypi.org list of packages. Rate-limiting was not encountered but is handled generically.
-