sourceforge: retry for all retryable exceptions

Since this lister is doing a lot more requests than most other, it makes sense that issues would arise more often. We want the lister to continue even if the website is having issues and not break on the first 500 or closed connection it encounters.

This change introduces a mechanism to retry all exceptions worth retrying and uses it for the SourceForge lister. Other listers might benefit from this, but this is out of scope here.

Tests had to be adjusted to stub the sleep function since retries happened way more often.

Related to swh/infra/sysadm-environment#3310 (closed)

---

Migrated from D5785 (view on Phabricator)

Build is green

Patch application report for D5785 (id=20682)

Rebasing onto 8f3bbacd...

Current branch diff-target is up to date.

Changes applied before test

commit 9ca5295a408b7f1f5be3a39ca7c4e00524dab4ac
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Wed May 26 12:05:39 2021 +0200

    sourceforge: retry for all retryable exceptions
    
    Since this lister is doing a lot more requests than most other, it makes
    sense that issues would arise more often. We want the lister to continue
    even if the website is having issues and not break on the first 500 or
    closed connection it encounters.
    
    This change introduces a mechanism to retry all exceptions worth
    retrying and uses it for the SourceForge lister. Other listers might
    benefit from this, but this is out of scope here.
    
    Tests had to be adjusted to stub the sleep function since retries happened
    way more often.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/295/ for more details.

Awesome!

Thanks.

Merge request was accepted

approved this merge request

Merge request was merged

closed