Publish status of existing listers and loaders

This is a meta-task to track activities related to exposing the status of existing listers and loaders. The goal is to, on one side, make public our progress, and on the other side, to make it easier to external contributors to see what has already been done and what is missing.

This involves the following:

• design web page
• process to maintain up to date
• make this information clearly visible and link to Sloan subgrants

---

Migrated from T3117 (view on Phabricator)

mentioned in merge request !220 (closed)

mentioned in merge request !221 (closed)

mentioned in merge request swh-web!813 (closed)

mentioned in merge request swh-web!814 (closed)

mentioned in merge request swh-web!815 (closed)

mentioned in merge request swh-web!816 (closed)

mentioned in merge request swh-web!817 (closed)

mentioned in merge request swh-web!818 (closed)

mentioned in merge request swh-web!820 (closed)

mentioned in merge request !241 (closed)

mentioned in merge request !244 (closed)

mentioned in merge request !249 (closed)

added Community Building Documentation Roadmap 2021 Roadmap 2022 meta-task labels

changed the description

added priority:Normal label

assigned to @anlambert

added priority:High label and removed priority:Normal label

As that documentation of loaders and listers currently available for the archive will target a general audience and not a technical one,
it will be added in the Software Heritage User documentation.

Two new sections Software Heritage loaders and Software Heritage listers will be added. One subsection per loader/lister will document it in a high-level overview way. At the top of each new section, a grid of logos related to the implemented loaders (VCS, source packages, ...) and listers (forges, package managers, ...) will be displayed linking to the related subsections.

For a loader, the following content should be found in its documentation:

• description of the source of software artifacts to load into the archive (VCS tool introduction for instance)
• link to source code repository and mention of related grants and external contributors if any
• link to current related issues on our forge
• high-level overview of how the loader process data to load into the archive
• how the found software artifacts are mapped to the SWH data model
• noticeable implemented features (for instance dump protocol support in git loader or externals processing in svn loader)
• reference mapping of external ids to SWHIDs if any
• description of extracted intrinsic metadata if any
• description of loader parameters
• example of invocation in the docker environment for quick user testing
• set of listers producing tasks that will execute the loader
• current deployment state (in production ? in staging ?)

For a lister, the following content should be found in its documentation:

• description of the source of software origins listing (forge, package manager, ...)
• link to source code repository and mention of related grants and external contributors if any
• link to current related issues on our forge
• high-level overview of the process to discover origins
• description of last update date for an origin if any
• description of lister parameters
• description of lister credentials (required ? recommended ?)
• example of invocation in the docker environment for quick user testing
• set of loaders that will be used to ingest listed origins into the archive
• current deployment state (in production ? in staging ?)

marked this issue as related to #4150

marked this issue as related to #4151