Make sure schema changes get committed when doing the AlteringSchemaTests

This restores some sanity when trying to handle concurrent connections to the databases

Test Plan

make test keeps working

---

Migrated from D321 (view on Phabricator)