Make the retry proxy work on all functions.

The metadata migration script kept crashing otherwise.

This also remove tests that are now redundant, as they all test the exact same code in the same scenario

---

Migrated from D4434 (view on Phabricator)