Skip to content
Snippets Groups Projects

Add a non-incremental sourceforge lister

Closed Add a non-incremental sourceforge lister
generated-differential-D5293-source into generated-differential-D5293-target
Closed Raphaël Gomès requested to merge generated-differential-D5293-source into generated-differential-D5293-target

Following @zack's work on T735, this change introduces an actual SWH lister for SourceForge.

SourceForge provides a main sitemap that lists sharded sitemaps, which themselves list pages. Each page belongs to a project (or sub-project, though those are rare), information about which can be found by querying a REST API, which gives us the list of any and all VCS used for said project. Both sitemaps and pages have a "last modified" timestamp that will be used in a future patch to implement incremental listing.

More precise information can be found as inline comments or docstrings.

Migrated from D5293 (view on Phabricator)

Merge request reports

Approved by
vlorentz

Closed by Phabricator Migration userPhabricator Migration user 4 years ago (Mar 23, 2021 5:41pm UTC)

Merge details

  • The changes were not merged into generated-differential-D5293-target.

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build has FAILED

    Patch application report for D5293 (id=18945)

    Rebasing onto df73073a...

    First, rewinding head to replay your work on top of it...
Applying: Add a non-incremental sourceforge lister
    Changes applied before test
    commit 5c77c477538af6e1a7b67c620a578618d08a5774
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Wed Mar 17 17:39:41 2021 +0100

    Add a non-incremental sourceforge lister
    
    Following zack's work on [T735](https://forge.softwareheritage.org/T735 'view original for T735 on Phabricator'), this change introduces an actual SWH lister for
    SourceForge.
    
    SourceForge provides a main sitemap that lists sharded sitemaps, which
    themselves list pages. Each page belongs to a project (or sub-project,
    though those are rare), information about which can be found by querying
    a REST API, which gives us the list of any and all VCS used for said
    project. Both sitemaps and pages have a "last modified" timestamp that
    will be used in a future patch to implement incremental listing.
    
    More precise information can be found as inline comments or docstrings.

    Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/258/ See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/258/console

  • Raphaël Gomès
    Raphaël Gomès @Alphare ·
    Author

    Add python 3.7 compat

  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build is green

    Patch application report for D5293 (id=18953)

    Rebasing onto df73073a...

    First, rewinding head to replay your work on top of it...
Applying: Add a non-incremental sourceforge lister
    Changes applied before test
    commit a085f108debd45ac7c7dc8b2777869bfe3aca822
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Wed Mar 17 17:39:41 2021 +0100

    Add a non-incremental sourceforge lister
    
    Following zack's work on [T735](https://forge.softwareheritage.org/T735 'view original for T735 on Phabricator'), this change introduces an actual SWH lister for
    SourceForge.
    
    SourceForge provides a main sitemap that lists sharded sitemaps, which
    themselves list pages. Each page belongs to a project (or sub-project,
    though those are rare), information about which can be found by querying
    a REST API, which gives us the list of any and all VCS used for said
    project. Both sitemaps and pages have a "last modified" timestamp that
    will be used in a future patch to implement incremental listing.
    
    More precise information can be found as inline comments or docstrings.

    See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/259/ for more details.

  • vlorentz
    vlorentz @vlorentz ·
    Maintainer

    Very nice! My only suggestion would be to add some tests for error conditions (eg. to make sure it retries).

  • Raphaël Gomès
    Raphaël Gomès @Alphare ·
    Author

    Update typing and add retry tests + check other fatal errors

  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build is green

    Patch application report for D5293 (id=18994)

    Rebasing onto 879170a5...

    First, rewinding head to replay your work on top of it...
Applying: Add a non-incremental sourceforge lister
    Changes applied before test
    commit b6bc9bd0d0b00ec731773617871ff6d3c1a53a1d
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Wed Mar 17 17:39:41 2021 +0100

    Add a non-incremental sourceforge lister
    
    Following zack's work on [T735](https://forge.softwareheritage.org/T735 'view original for T735 on Phabricator'), this change introduces an actual SWH lister for
    SourceForge.
    
    SourceForge provides a main sitemap that lists sharded sitemaps, which
    themselves list pages. Each page belongs to a project (or sub-project,
    though those are rare), information about which can be found by querying
    a REST API, which gives us the list of any and all VCS used for said
    project. Both sitemaps and pages have a "last modified" timestamp that
    will be used in a future patch to implement incremental listing.
    
    More precise information can be found as inline comments or docstrings.

    See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/260/ for more details.

  • Raphaël Gomès
    Raphaël Gomès @Alphare ·
    Author

    Record lastmod for origins

  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build is green

    Patch application report for D5293 (id=19015)

    Rebasing onto 879170a5...

    First, rewinding head to replay your work on top of it...
Applying: Add a non-incremental sourceforge lister
    Changes applied before test
    commit 247a8a25fd7e44b37766e2d14094227ecf56d32b
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Wed Mar 17 17:39:41 2021 +0100

    Add a non-incremental sourceforge lister
    
    Following zack's work on [T735](https://forge.softwareheritage.org/T735 'view original for T735 on Phabricator'), this change introduces an actual SWH lister for
    SourceForge.
    
    SourceForge provides a main sitemap that lists sharded sitemaps, which
    themselves list pages. Each page belongs to a project (or sub-project,
    though those are rare), information about which can be found by querying
    a REST API, which gives us the list of any and all VCS used for said
    project. Both sitemaps and pages have a "last modified" timestamp that
    will be used in a future patch to implement incremental listing.
    
    More precise information can be found as inline comments or docstrings.

    See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/261/ for more details.

  • Raphaël Gomès
    Raphaël Gomès @Alphare ·
    Author

    Simplify date tests (sorry for the update spam)

  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build is green

    Patch application report for D5293 (id=19016)

    Rebasing onto 879170a5...

    First, rewinding head to replay your work on top of it...
Applying: Add a non-incremental sourceforge lister
    Changes applied before test
    commit 5903a523aaa8108a007261d6923004cb6d400934
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Wed Mar 17 17:39:41 2021 +0100

    Add a non-incremental sourceforge lister
    
    Following zack's work on [T735](https://forge.softwareheritage.org/T735 'view original for T735 on Phabricator'), this change introduces an actual SWH lister for
    SourceForge.
    
    SourceForge provides a main sitemap that lists sharded sitemaps, which
    themselves list pages. Each page belongs to a project (or sub-project,
    though those are rare), information about which can be found by querying
    a REST API, which gives us the list of any and all VCS used for said
    project. Both sitemaps and pages have a "last modified" timestamp that
    will be used in a future patch to implement incremental listing.
    
    More precise information can be found as inline comments or docstrings.

    See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/262/ for more details.

  • vlorentz
    vlorentz @vlorentz ·
    Maintainer

    Merge request was accepted

  • vlorentz approved this merge request

    approved this merge request

  • Raphaël Gomès
    Raphaël Gomès @Alphare ·
    Author

    Rebase

  • Raphaël Gomès
    Raphaël Gomès @Alphare ·
    Author

    Merge request was merged

  • Raphaël Gomès closed

    closed

  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build is green

    Patch application report for D5293 (id=19067)

    Rebasing onto 879170a5...

    Current branch diff-target is up to date.
    Changes applied before test
    commit f7b27c6930220b4ed3e1141167d61e67416852c9
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Wed Mar 17 17:39:41 2021 +0100

    Add a non-incremental sourceforge lister
    
    Following zack's work on [T735](https://forge.softwareheritage.org/T735 'view original for T735 on Phabricator'), this change introduces an actual SWH lister for
    SourceForge.
    
    SourceForge provides a main sitemap that lists sharded sitemaps, which
    themselves list pages. Each page belongs to a project (or sub-project,
    though those are rare), information about which can be found by querying
    a REST API, which gives us the list of any and all VCS used for said
    project. Both sitemaps and pages have a "last modified" timestamp that
    will be used in a future patch to implement incremental listing.
    
    More precise information can be found as inline comments or docstrings.

    See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/263/ for more details.

Please register or sign in to reply