tests: Use production backends within the indexer tests
This detected some paper cuts within the db.py module (through some cli tests no longer using the memory storage).
The main goal is to decrease friction when actually deploying indexer related services (backend, indexers, ...).
The pg backend tests should still be reasonably fast as it's using the swh.core.db.pytest_plugin (which truncate tables in between tests).
Indeed, a rapid look in the jenkins diff ui [1] shows that the overall execution time is reasonably as fast as before.
One of the next steps would be to improve the current journal client tests in the indexer [2]
Related to T2821
Test Plan
tox
Migrated from D4638 (view on Phabricator)
Merge request reports
Activity
Build is green
Patch application report for D4638 (id=16459)
Rebasing onto 28ae49da...
Current branch diff-target is up to date.Changes applied before test
commit 394576b95688b867a86a5bcb4da30534640a0577 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Tue Dec 1 15:40:01 2020 +0100 tests: Use production backends within the indexer tests This detected some paper cuts within cli tests for example. The main goal is to decrease friction when actually deploying indexer related services (backend, indexers, ...). The pg backends tests should still be reasonably fast as it's using the swh.core.db.pytest_plugin (which truncate tables in between tests). Related to [T2821](https://forge.softwareheritage.org/T2821 'view original for T2821 on Phabricator')See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/131/ for more details.
mentioned in commit f775fea1
16 16 # loading) 17 17 collect_ignore = ["swh/indexer/storage/api/wsgi.py"] 18 18 19 # we use the swh_scheduler fixture 20 pytest_plugins = ["swh.scheduler.pytest_plugin"] 19 # we use the various swh fixtures 20 pytest_plugins = [ 21 "swh.scheduler.pytest_plugin", 22 "swh.storage.pytest_plugin", 23 "swh.core.db.pytest_plugin", 24 ] 487 487 args.append(last) 488 488 if mappings is not None: 489 489 where.append("oim.mappings && %s") 490 args.append(mappings) 490 args.append(list(mappings)) 491 491 if tool_ids is not None: 492 492 where.append("oim.indexer_configuration_id = ANY(%s)") 493 args.append(tool_ids) 493 args.append(list(tool_ids)) 92 ) as config_fd: 93 config_fd.write(CLI_CONFIG) 94 config_fd.seek(0) 95 get_scheduler_mock.return_value = scheduler 96 result = runner.invoke(indexer_cli_group, ["-C" + config_fd.name] + args) 97 if not catch_exceptions and result.exception: 98 print(result.output) 99 raise result.exception 100 return result 101 102 103 def test_mapping_list(indexer_scheduler): 104 result = invoke(indexer_scheduler, False, ["mapping", "list",]) 75 @pytest.fixture 76 def cli_runner(): 77 return CliRunner() 14 17 from swh.model.model import Origin 15 18 from swh.storage.interface import StorageInterface 16 19 17 from .test_metadata import REVISION_METADATA_CONFIG 20 from .test_metadata import TRANSLATOR_TOOL 18 21 from .utils import REVISION, YARN_PARSER_METADATA 19 22 20 23 24 @pytest.fixture 25 def swh_indexer_config(swh_indexer_config): 26 """Override the default configuration to override the tools entry 27 28 """ 29 cfg = copy.deepcopy(swh_indexer_config) 30 cfg["tools"] = TRANSLATOR_TOOL -
This looks alright, thanks.
The fixture stacking between indexer_config and storage/scheduler/objstorage feels a bit backwards, but I can understand why you'd want to keep the full configuration in a single fixture instead of breaking it down into a bunch of them.
I've made a few comments inline about stuff I'm not sure about.
mentioned in merge request !460 (closed)
Some references in the commit message have been migrated:
- T2821 is now swh/meta#2821 (closed)
-
Build is green
Patch application report for D4638 (id=16502)
Rebasing onto 28ae49da...
Current branch diff-target is up to date.Changes applied before test
commit 2e73f3be6a1cc9d66c88a37e817e042aa60db0f5 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Tue Dec 1 15:40:01 2020 +0100 tests: Use production backends within the indexer tests This detected some paper cuts within cli tests for example. The main goal is to decrease friction when actually deploying indexer related services (backend, indexers, ...). The pg backends tests should still be reasonably fast as it's using the swh.core.db.pytest_plugin (which truncate tables in between tests). Related to [T2821](https://forge.softwareheritage.org/T2821 'view original for T2821 on Phabricator')See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/134/ for more details.
