pipeline storage: Add retry behavior on flushing failures

Currently, wrong "hash collisions" are happening a lot on ingestion [1] [2] [3] The last loading step (flush) is failing on most loaders (git, npm, etc...).

This commits adds the retry behavior to the flush method. Which should decrease the frequency of that error.

The remaining hash collisions which won't subside should be then real hash collisions.

• [1] https://sentry.softwareheritage.org/share/issue/102aace238fe4ba6b49bcc5531f7c2bf/

• [2] https://sentry.softwareheritage.org/share/issue/8e8b48a1d94c465b8109e76311ecdbe7/

• [3] https://sentry.softwareheritage.org/share/issue/d4f1208b7eec4b43b11e38494ff039cc/

Test Plan

tox

---

Migrated from D2603 (view on Phabricator)