Introduce bioconductor lister
The way this lister works is that it scrapes origins from pages like:
- https://bioconductor.org/packages/json/3.17/bioc/packages.json
- https://bioconductor.org/packages/json/3.17/workflows/packages.json
- https://bioconductor.org/packages/json/3.18/bioc/packages.json
Note the pattern https://bioconductor.org/packages/json/<release-version>/<category>/packages.json
Here, I've coined the term "category" to represent the 4 possible values: bioc, workflows, data/annotation, and data/experiment.
This packages.json file is a dictionary of all the packages. We iterate through them and send them to the scheduler.
Related to #4699
Refs. swh/meta#5051
Merge request reports
Activity
Jenkins job DLS/gitlab-builds #156 failed .
See Console Output and Coverage Report for more details.-
Jenkins job DLS/gitlab-builds #180 failed .
See Console Output and Coverage Report for more details. -
Jenkins job DLS/gitlab-builds #181 failed .
See Console Output and Coverage Report for more details. added 1 commit
- f5214879 - Use default_factory to use empty dict in lister state
-
Jenkins job DLS/gitlab-builds #182 failed .
See Console Output and Coverage Report for more details. -
Jenkins job DLS/gitlab-builds #183 succeeded .
See Console Output and Coverage Report for more details. added 1 commit
- 69998aee - List packages of data annotation and experiment category
-
Jenkins job DLS/gitlab-builds #184 failed .
See Console Output and Coverage Report for more details. -
Jenkins job DLS/gitlab-builds #185 succeeded .
See Console Output and Coverage Report for more details. 
swh/lister/bioconductor/lister.py 0 → 1006449 10 import iso8601 11 12 from swh.scheduler.interface import SchedulerInterface 13 from swh.scheduler.model import ListedOrigin 14 15 from ..pattern import StatelessLister 16 17 logger = logging.getLogger(__name__) 18 19 Release = str 20 Category = str 21 BioconductorListerPage = Tuple[Release, Category, Dict[str, Any]] 22 23 24 DEFAULT_BIOCONDUCTOR_RELEASES = [ -
we need a way to detect those automatically. Could you send them a patch to https://github.com/Bioconductor/bioconductor.org/blob/e2eb96c8106e101fc0e47281514b241aae79ead8/lib/data_sources/bioc_views.rb to make it display the versions?
if that's not possible, someone at SWH should be notified when there's a new version. Maybe make the lister query https://bioconductor.org/js/versions.js and tell Icinga/Prometheus/... if the versions advertised there are missing from this file.
changed this line in version 8 of the diff
