Refactor data model to use config_id instead of datastore in xxx_object tables
These tables used to reference the datastore the invalid/missing object was found in, but not keeping the config entry, i.e. the checking session during wich the invalid/missing object was found, which can be an issue when more than one checking session is executed on a given datastore. This replaces the `datastore` field of tables `corrupt_object`, `missing_object` and `missing_object_reference` tables by `config_id`. Adapt all the code accordingly. Note that it changes a bit the cli usage: the kafka checker now needs a config entry, thus a kafka checking session can ony target a given object type (i.e. one kafka topic), The migration script will fill the config_id column for corrupt_object using the check_config entry that matches the oject_type (of corrupt_object) and datastore. For missing_object and missing_object_reference, it will use this later table to idenify the check_config entry corresponding object type for the reference_id and datastore, since it is a checking session on this object type that will generate a missing object entry (which is generaaly not of the same type). For the missing_object table, the config_id will use the one extracted from the missing_object_reference (joining on the missing_id column). Note that the migration script will fail if there are rows in one of these tables for which there exists more than one possible config_entry (i.e. with the same object_type and datastore).
Showing
- swh/scrubber/cli.py 62 additions, 19 deletionsswh/scrubber/cli.py
- swh/scrubber/db.py 134 additions, 81 deletionsswh/scrubber/db.py
- swh/scrubber/journal_checker.py 51 additions, 26 deletionsswh/scrubber/journal_checker.py
- swh/scrubber/sql/30-schema.sql 9 additions, 9 deletionsswh/scrubber/sql/30-schema.sql
- swh/scrubber/sql/60-indexes.sql 10 additions, 10 deletionsswh/scrubber/sql/60-indexes.sql
- swh/scrubber/sql/upgrades/7.sql 139 additions, 0 deletionsswh/scrubber/sql/upgrades/7.sql
- swh/scrubber/storage_checker.py 13 additions, 17 deletionsswh/scrubber/storage_checker.py
- swh/scrubber/tests/conftest.py 70 additions, 2 deletionsswh/scrubber/tests/conftest.py
- swh/scrubber/tests/data/cli/sql/10-init-all.sql 401 additions, 0 deletionsswh/scrubber/tests/data/cli/sql/10-init-all.sql
- swh/scrubber/tests/data/cli/sql/20-data.sql 77 additions, 0 deletionsswh/scrubber/tests/data/cli/sql/20-data.sql
- swh/scrubber/tests/data/cli/sql/upgrades/006.sql 1 addition, 0 deletionsswh/scrubber/tests/data/cli/sql/upgrades/006.sql
- swh/scrubber/tests/data/cli/sql/upgrades/007.sql 1 addition, 0 deletionsswh/scrubber/tests/data/cli/sql/upgrades/007.sql
- swh/scrubber/tests/storage_checker_tests.py 2 additions, 2 deletionsswh/scrubber/tests/storage_checker_tests.py
- swh/scrubber/tests/test_cli.py 32 additions, 2 deletionsswh/scrubber/tests/test_cli.py
- swh/scrubber/tests/test_fixer.py 42 additions, 77 deletionsswh/scrubber/tests/test_fixer.py
- swh/scrubber/tests/test_journal_kafka.py 59 additions, 16 deletionsswh/scrubber/tests/test_journal_kafka.py
- swh/scrubber/tests/test_migration.py 258 additions, 0 deletionsswh/scrubber/tests/test_migration.py
- swh/scrubber/tests/test_origin_locator.py 30 additions, 39 deletionsswh/scrubber/tests/test_origin_locator.py
- swh/scrubber/utils.py 6 additions, 4 deletionsswh/scrubber/utils.py
Loading
Please register or sign in to comment