Skip to content
Snippets Groups Projects
Forked from Platform / Development / swh-lister
Source project has a limited visibility.
  • Antoine Lambert's avatar
    108816f2
    rubygems: Use gems database dump to improve listing output · 108816f2
    Antoine Lambert authored
    Instead of using an undocumented rubygems HTTP endpoint that only
    gives us the names of the gems, prefer to exploit the daily PostgreSQL
    dump of the rubygems.org database.
    
    It enables to list all gems but also all versions of a gem and its
    release artifacts. For each relase artifact, the following info are
    extracted: version, download URL, sha256 checksum, release date
    plus a couple of extra metadata.
    
    The lister will now set list of artifacts and list of metadata as extra
    loader arguments when sending a listed origin to the scheduler database.
    A last_update date is also computed which should ensure loading tasks
    for rubygems will be scheduled only when new releases are available since
    last loadings.
    
    To be noted, the lister will spawn a temporary postgres instance so this
    require the initdb executable from postgres server installation to be
    available in the execution environment.
    
    Related to T1777
    108816f2
    History
    rubygems: Use gems database dump to improve listing output
    Antoine Lambert authored
    Instead of using an undocumented rubygems HTTP endpoint that only
    gives us the names of the gems, prefer to exploit the daily PostgreSQL
    dump of the rubygems.org database.
    
    It enables to list all gems but also all versions of a gem and its
    release artifacts. For each relase artifact, the following info are
    extracted: version, download URL, sha256 checksum, release date
    plus a couple of extra metadata.
    
    The lister will now set list of artifacts and list of metadata as extra
    loader arguments when sending a listed origin to the scheduler database.
    A last_update date is also computed which should ensure loading tasks
    for rubygems will be scheduled only when new releases are available since
    last loadings.
    
    To be noted, the lister will spawn a temporary postgres instance so this
    require the initdb executable from postgres server installation to be
    available in the execution environment.
    
    Related to T1777