Add random directory sampling policy

This makes use of the new discovery algorithm introduced in swh-loader-core, which should help speed up large (think Linux kernel or way larger) scans.

Most of the time is spend walking the on-disk directory and hashing, which is where the new optimizations in swh-model==6.5.0 should come in handy. Python is close to its limit in that regard, some future endeavor should look into setting up SWH for native extensions.

---

Migrated from D8539 (view on Phabricator)

swh/scanner/tests/conftest.py

@@ -144,12 +144,21 @@ def test_swhids_sample(tmp_path):
 
 @pytest.fixture(scope="session")
 def tmp_requests(tmpdir_factory):
+    """Logs each SWHID per line in every request made to the `known` endpoint"""
     requests_file = tmpdir_factory.mktemp("data").join("requests.json")
     return requests_file
 
 
 @pytest.fixture(scope="session")
-def app(tmp_requests):
+def tmp_accesses(tmpdir_factory):
+    """Logs each request made to the `known` endpoint, writing the number
+    of SWHIDs queried, one per line."""
+    requests_file = tmpdir_factory.mktemp("data").join("accesses.json")
+    return requests_file
+
+
+@pytest.fixture(scope="session")
+def app(tmp_requests, tmp_accesses):
     """Flask backend API (used by live_server)."""
-    app = create_app(tmp_requests)
+    app = create_app(tmp_requests, tmp_accesses)
     return app