GNU Lister (!369) · Merge requests · Platform / Development / swh-lister

Closed Archit Agrawal requested to merge generated-differential-D1482-source into generated-differential-D1482-target 5 years ago

Jun 11, 2019

swh.lister.gnu: Add function to check for file extension. · 7c6245e6

Archit Agrawal authored 5 years ago

Added a function which will derive the extension from filename
and check if the fie extension match the type of file that is to be
archived.

7c6245e6

Jun 08, 2019

swh.lister.gnu: Add functionality to list all the tarballs for a package. · 709ba8a6

Archit Agrawal authored 5 years ago

As discussed in T1389 to ingest all packages using base loader, it need
a list of all the tarballs for a pakage.
Hence modifified lister to recursively list all the tarballs for a
package with their last updated time.

709ba8a6

swh.lister.gnu : Change download method of tree.json file to request · ebdb9598

Archit Agrawal authored 5 years ago

Previously gnu lister was using same code as that of tarball loader
to download, unzip and read tree.json file.
To make the code consise the downloading method is changed to
requests library.

ebdb9598

swh.lister.gnu · 151f6cd2

Archit Agrawal authored 5 years ago

Implement first pass of gnu lister to list all the
packages present in https://ftp.gnu.org/
Add GNU lister in README and cli.py

Closes T1722

151f6cd2

GNU Lister

Merge request reports

Activity