Collecting ecosyste.ms like package metadata
There are two ways in which we want to have ecosyste.ms like metadata.
-
Extracting dependencies using bibliothecary: ecosyste.ms uses bibliothecary to get dependency metadata from any file. swh-indexer already runs on such files so we will call the bibliothecary lib inside swh-indexer and translate the dependencies in the swhpkg format (!484 (merged)).
-
Collect extrinsic metadata by reusing ecosyste.ms package repo: ecosyste.ms package repo has an ecosystem module that contains crawlers for different package managers like npm, pypi, hex, etc. The ecosystem module is written in Ruby language. So we'll have to provide alternative classes/functions to emulate external functions (like Redis client). This metadata collection code should be attached to relevant loaders of swh-loader-core so that the scheduling is taken care of.