npm lister

We want to be able to list all available packages on npm in order to load their content into the archive.

---

Migrated from T1380 (view on Phabricator)

mentioned in issue #1398 (closed)

mentioned in merge request !365 (closed)

marked this issue as related to swh/meta#1378 (closed)

added Npm Lister Origin-npm priority:Normal labels

The npm registry is a CouchDB database located at https://replicate.npmjs.com.

The following endpoint enables to list all registered packages: https://replicate.npmjs.com/_all_docs?limit=100

We should be able to use the SWHIndexingHttpLister [1] using the recommended CouchDB pagination method [2]

• [1] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/core/indexing_lister.py

• [2] http://docs.couchdb.org/en/stable/ddocs/views/pagination.html#paging-alternate-method

mentioned in commit 605a67f5

changed the description

assigned to @anlambert

closed