Skip to content

Make the SourceForge lister incremental

Raphaël Gomès requested to merge generated-differential-D5659-source into generated-differential-D5659-target

SourceForge's sitemaps (1 main one + many sharded) give us a "last modified" date for every subsitemap and project, allowing us to perform an incremental listing.

We store the subsitemaps' "last modified" dates in the lister state, as well as those of the empty projects (projects which don't have any VCS registered), and the rest comes from the already visited origins from the database.

The tests try to cover the possible cases of a subsitemap that has changed, one that hasn't, a project that has change, one that hasn't, and same for an empty project.

Migrated from D5659 (view on Phabricator)

Merge request reports