settings/tests: Fix tests when using pytest-xdist
This fixes the tests when using the parallelization feature of pytest-xdist
.
We gain at least 10 seconds on the overall tests execution time using 4 workers.
Migrated from D2574 (view on Phabricator)