Add a documentation section for swh.core.db
Closed
requested to merge generated-differential-D7159-source into generated-differential-D7159-target
Merge request reports
Activity
Build has FAILED
Patch application report for D7159 (id=25953)
Could not rebase; Attempt merge onto 3139d334...
Updating 3139d33..e6fbe24 Fast-forward MANIFEST.in | 2 + docs/db.rst | 181 ++++++++++++++++ docs/index.rst | 1 + swh/core/cli/db.py | 179 +++++++++++++++- swh/core/db/db_utils.py | 228 ++++++++++++++++++++- swh/core/db/pytest_plugin.py | 213 +++++++++++++------ swh/core/db/sql/35-dbmetadata.sql | 28 +++ swh/core/db/tests/conftest.py | 42 ++++ .../tests/data/cli/{1-schema.sql => 30-schema.sql} | 2 +- .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} | 0 .../db/tests/data/cli/{4-data.sql => 50-data.sql} | 2 +- .../db/tests/data/cli_new/0-superuser-init.sql | 1 + swh/core/db/tests/data/cli_new/30-schema.sql | 6 + swh/core/db/tests/data/cli_new/40-funcs.sql | 6 + swh/core/db/tests/data/cli_new/50-data.sql | 2 + swh/core/db/tests/data/cli_new/upgrades/001.sql | 5 + swh/core/db/tests/data/cli_new/upgrades/002.sql | 4 + swh/core/db/tests/data/cli_new/upgrades/003.sql | 4 + swh/core/db/tests/data/cli_new/upgrades/004.sql | 4 + swh/core/db/tests/data/cli_new/upgrades/005.sql | 4 + swh/core/db/tests/data/cli_new/upgrades/006.sql | 7 + .../db/tests/pytest_plugin/test_pytest_plugin.py | 26 ++- swh/core/db/tests/test_cli.py | 197 +++++++++++++----- swh/core/db/tests/test_db_utils.py | 117 +++++++++++ swh/core/tests/test_utils.py | 5 + swh/core/utils.py | 5 + 26 files changed, 1135 insertions(+), 136 deletions(-) create mode 100644 docs/db.rst create mode 100644 swh/core/db/sql/35-dbmetadata.sql rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%) rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%) rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%) create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql create mode 100644 swh/core/db/tests/data/cli_new/upgrades/001.sql create mode 100644 swh/core/db/tests/data/cli_new/upgrades/002.sql create mode 100644 swh/core/db/tests/data/cli_new/upgrades/003.sql create mode 100644 swh/core/db/tests/data/cli_new/upgrades/004.sql create mode 100644 swh/core/db/tests/data/cli_new/upgrades/005.sql create mode 100644 swh/core/db/tests/data/cli_new/upgrades/006.sql create mode 100644 swh/core/db/tests/test_db_utils.pyChanges applied before test
commit e6fbe24e0b8456ced4afb2c717f9efcb05d98cfd Author: David Douard <david.douard@sdfa3.org> Date: Fri Feb 11 15:54:25 2022 +0100 Add a documentation section for swh.core.db this is far from completion. commit e71baa97f684c75f530456fcddef6755e5de9bde Author: David Douard <david.douard@sdfa3.org> Date: Mon Feb 7 15:46:48 2022 +0100 Refactor swh.core.db.pytest_plugin update the postgresql_fact fixture to be able to use new-ish pytest_postgresql ability to recreate databases between tests from a template database. The ides is to stop using the fragile and confusing "reset" of the database between tests, which means de facto keep a stateful database between tests, which make tests result depend on de facto order of execution of these tests. For example, test_smoke_test_people_db_up_and_reset cannot be executed alone, since it depends on the fact test_smoke_test_people_db_up has been executed before so that the people's db has been reset as a result of the test setup done by the fixture, which will happen if and only if the database already exists (thus if another test using this db has been executed before in this test session). The old table truncate based behavior has been kept (using a no_db_drop flag argument) in this diff for bw compatibility reasons, and also because the new drop-and-create-from-template beahvior remains a bit slower, which might be a problem for some test cases. Tests for dependencies (swh-storage, etc.) should still pass with this refactoring, but will be significantly slower. A series of revisions for these dependencies is about to be proposed to adapt them to the new db metadata (version, module, flavor) handling feature coming with swh.core 2, as well as updating their tests to take advantage of the template-based db creation in tests, which should bring back test execution time to almost the same as before this refactoring. This is needed to be able to implement and test support for the new 'swh db upgrade' all this is actually about. commit 5712a5d290966b13242d3a790bd29e6a442fa87b Author: David Douard <david.douard@sdfa3.org> Date: Tue Feb 1 15:43:09 2022 +0100 Add a 'swh db upgrade' command this new command allows to handle the database migration from the current version to the latest version advertized but the backend datastore. Thie feature is only working for "new style" swh.core.db based datastores. commit cfe1e6059e85bbe8d1ad39a68f9953b478dcb294 Author: David Douard <david.douard@sdfa3.org> Date: Tue Feb 1 14:51:43 2022 +0100 Make 'swh db init' fill the dbversion table for this feature to work properly, the backend datastore package must provide the 'get_datastore' factory function in its root namespace and the datastore instance must provide a 'get_current_version()' method. This also means the canonical cli usage for the `swh db init` is now using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`) instead of giving only the db cnx string (`--db-name`), so that the backend datastore class can be instanciated using the `get_datastore()` factory function. However a '--initial-version' cli option has been added to 'swh db init' to make it easier, especially for unit tests. commit 2e184856ac6ff8af359d6112441ccbf837e6fed8 Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 17:54:01 2022 +0100 Use a standard postgresql db fixture in test_cli instead of a custom 'test_db' (which did not truncate dbversion and origin tables, not sure what the intent was for this). commit 943a813e318248e0c51a35d7866dbb145290c82a Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 15:02:51 2022 +0100 Add support for dbversion and dbmodule handling in swh db init for now, managing the dbversion table (storing the db schema upgrade history) is partially the responsibility of the swh modules implementing a datastore (swh-storage etc.), and the actual swh module was not stored in the database. This adds support for both the dbversion and dbmodule tables management in swh.core.db. The `swh db init` command now creates both these tables and initialize their values if possible. For this, it introduces a new common "API" for swh.core.db based datastores: - a swh module implementing a datastore is expected to have a `get_datastore` function in its top namespace; this function is typically the actual `get_storage`, `get_scheduler` etc. functions - this factory function is expected to use the "postgresql" cls for the datastore based on swh.core.db, - the datastore instance (eg. instance of the swh.storage.postgresql.Storage class) is expected to have a `get_current_version()` method (returning an int). This revision also provides a new `swh db version` command displaying the current (code) version (if available), the db version (possibly the whole version history) and flavor (if any) for the datastore given as argument. It should be backward compatible with existing swh datastore modules (i.e. not yet adapted to this new API). commit b1e449fd91ebcc403966a79fd62d449921fcf550 Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 16:00:43 2022 +0100 Refactor the mock_package_sql fixture to allow multiple sql script set will be used to implement several test scenarios in following commits. Also rename existing sql scripts with a "legit" numbering scheme.Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/309/ See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/309/console
-
Build is green
Patch application report for D7159 (id=25958)
Could not rebase; Attempt merge onto 3139d334...
Updating 3139d33..7065059 Fast-forward MANIFEST.in | 2 + docs/db.rst | 181 ++++++++++++++++ docs/index.rst | 1 + swh/core/cli/db.py | 179 +++++++++++++++- swh/core/db/db_utils.py | 228 ++++++++++++++++++++- swh/core/db/pytest_plugin.py | 213 +++++++++++++------ swh/core/db/sql/35-dbmetadata.sql | 28 +++ swh/core/db/tests/conftest.py | 42 ++++ .../tests/data/cli/{1-schema.sql => 30-schema.sql} | 2 +- .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} | 0 .../db/tests/data/cli/{4-data.sql => 50-data.sql} | 2 +- .../db/tests/data/cli_new/0-superuser-init.sql | 1 + swh/core/db/tests/data/cli_new/30-schema.sql | 6 + swh/core/db/tests/data/cli_new/40-funcs.sql | 6 + swh/core/db/tests/data/cli_new/50-data.sql | 2 + swh/core/db/tests/data/cli_new/upgrades/001.sql | 5 + swh/core/db/tests/data/cli_new/upgrades/002.sql | 4 + swh/core/db/tests/data/cli_new/upgrades/003.sql | 4 + swh/core/db/tests/data/cli_new/upgrades/004.sql | 4 + swh/core/db/tests/data/cli_new/upgrades/005.sql | 4 + swh/core/db/tests/data/cli_new/upgrades/006.sql | 7 + .../db/tests/pytest_plugin/test_pytest_plugin.py | 26 ++- swh/core/db/tests/test_cli.py | 197 +++++++++++++----- swh/core/db/tests/test_db_utils.py | 117 +++++++++++ swh/core/tests/test_utils.py | 5 + swh/core/utils.py | 5 + 26 files changed, 1135 insertions(+), 136 deletions(-) create mode 100644 docs/db.rst create mode 100644 swh/core/db/sql/35-dbmetadata.sql rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%) rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%) rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%) create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql create mode 100644 swh/core/db/tests/data/cli_new/upgrades/001.sql create mode 100644 swh/core/db/tests/data/cli_new/upgrades/002.sql create mode 100644 swh/core/db/tests/data/cli_new/upgrades/003.sql create mode 100644 swh/core/db/tests/data/cli_new/upgrades/004.sql create mode 100644 swh/core/db/tests/data/cli_new/upgrades/005.sql create mode 100644 swh/core/db/tests/data/cli_new/upgrades/006.sql create mode 100644 swh/core/db/tests/test_db_utils.pyChanges applied before test
commit 706505961fef233a781022007b860df06ff7ee10 Author: David Douard <david.douard@sdfa3.org> Date: Fri Feb 11 15:54:25 2022 +0100 Add a documentation section for swh.core.db this is far from completion. commit 2981fa08f812d020485603ca4303a17c53a718d0 Author: David Douard <david.douard@sdfa3.org> Date: Mon Feb 7 15:46:48 2022 +0100 Refactor swh.core.db.pytest_plugin update the postgresql_fact fixture to be able to use new-ish pytest_postgresql ability to recreate databases between tests from a template database. The ides is to stop using the fragile and confusing "reset" of the database between tests, which means de facto keep a stateful database between tests, which make tests result depend on de facto order of execution of these tests. For example, test_smoke_test_people_db_up_and_reset cannot be executed alone, since it depends on the fact test_smoke_test_people_db_up has been executed before so that the people's db has been reset as a result of the test setup done by the fixture, which will happen if and only if the database already exists (thus if another test using this db has been executed before in this test session). The old table truncate based behavior has been kept (using a no_db_drop flag argument) in this diff for bw compatibility reasons, and also because the new drop-and-create-from-template beahvior remains a bit slower, which might be a problem for some test cases. Tests for dependencies (swh-storage, etc.) should still pass with this refactoring, but will be significantly slower. A series of revisions for these dependencies is about to be proposed to adapt them to the new db metadata (version, module, flavor) handling feature coming with swh.core 2, as well as updating their tests to take advantage of the template-based db creation in tests, which should bring back test execution time to almost the same as before this refactoring. This is needed to be able to implement and test support for the new 'swh db upgrade' all this is actually about. commit fc451f2b86eff00fb5dea99c919d5187e5e3c5b5 Author: David Douard <david.douard@sdfa3.org> Date: Tue Feb 1 15:43:09 2022 +0100 Add a 'swh db upgrade' command this new command allows to handle the database migration from the current version to the latest version advertized but the backend datastore. Thie feature is only working for "new style" swh.core.db based datastores. commit 330a9d31f5f188dbc457f6cda349bcafa09e396e Author: David Douard <david.douard@sdfa3.org> Date: Tue Feb 1 14:51:43 2022 +0100 Make 'swh db init' fill the dbversion table for this feature to work properly, the backend datastore package must provide the 'get_datastore' factory function in its root namespace and the datastore instance must provide a 'get_current_version()' method. This also means the canonical cli usage for the `swh db init` is now using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`) instead of giving only the db cnx string (`--db-name`), so that the backend datastore class can be instanciated using the `get_datastore()` factory function. However a '--initial-version' cli option has been added to 'swh db init' to make it easier, especially for unit tests. commit 2e184856ac6ff8af359d6112441ccbf837e6fed8 Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 17:54:01 2022 +0100 Use a standard postgresql db fixture in test_cli instead of a custom 'test_db' (which did not truncate dbversion and origin tables, not sure what the intent was for this). commit 943a813e318248e0c51a35d7866dbb145290c82a Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 15:02:51 2022 +0100 Add support for dbversion and dbmodule handling in swh db init for now, managing the dbversion table (storing the db schema upgrade history) is partially the responsibility of the swh modules implementing a datastore (swh-storage etc.), and the actual swh module was not stored in the database. This adds support for both the dbversion and dbmodule tables management in swh.core.db. The `swh db init` command now creates both these tables and initialize their values if possible. For this, it introduces a new common "API" for swh.core.db based datastores: - a swh module implementing a datastore is expected to have a `get_datastore` function in its top namespace; this function is typically the actual `get_storage`, `get_scheduler` etc. functions - this factory function is expected to use the "postgresql" cls for the datastore based on swh.core.db, - the datastore instance (eg. instance of the swh.storage.postgresql.Storage class) is expected to have a `get_current_version()` method (returning an int). This revision also provides a new `swh db version` command displaying the current (code) version (if available), the db version (possibly the whole version history) and flavor (if any) for the datastore given as argument. It should be backward compatible with existing swh datastore modules (i.e. not yet adapted to this new API). commit b1e449fd91ebcc403966a79fd62d449921fcf550 Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 16:00:43 2022 +0100 Refactor the mock_package_sql fixture to allow multiple sql script set will be used to implement several test scenarios in following commits. Also rename existing sql scripts with a "legit" numbering scheme.See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/313/ for more details.
-
Build has FAILED
Patch application report for D7159 (id=25987)
Could not rebase; Attempt merge onto 3139d334...
Updating 3139d33..4c28b1c Fast-forward MANIFEST.in | 2 + docs/db.rst | 181 ++++++++++++++++ docs/index.rst | 1 + swh/core/cli/db.py | 179 +++++++++++++++- swh/core/db/db_utils.py | 228 ++++++++++++++++++++- swh/core/db/pytest_plugin.py | 213 +++++++++++++------ swh/core/db/sql/35-dbmetadata.sql | 28 +++ swh/core/db/tests/conftest.py | 42 ++++ .../tests/data/cli/{1-schema.sql => 30-schema.sql} | 2 +- .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} | 0 .../db/tests/data/cli/{4-data.sql => 50-data.sql} | 2 +- .../db/tests/data/cli_new/0-superuser-init.sql | 1 + swh/core/db/tests/data/cli_new/30-schema.sql | 6 + swh/core/db/tests/data/cli_new/40-funcs.sql | 6 + swh/core/db/tests/data/cli_new/50-data.sql | 2 + swh/core/db/tests/data/cli_new/upgrades/001.sql | 5 + swh/core/db/tests/data/cli_new/upgrades/002.sql | 4 + swh/core/db/tests/data/cli_new/upgrades/003.sql | 4 + swh/core/db/tests/data/cli_new/upgrades/004.sql | 4 + swh/core/db/tests/data/cli_new/upgrades/005.sql | 4 + swh/core/db/tests/data/cli_new/upgrades/006.sql | 7 + .../db/tests/pytest_plugin/test_pytest_plugin.py | 26 ++- swh/core/db/tests/test_cli.py | 197 +++++++++++++----- swh/core/db/tests/test_db_utils.py | 118 +++++++++++ swh/core/tests/test_utils.py | 5 + swh/core/utils.py | 5 + 26 files changed, 1136 insertions(+), 136 deletions(-) create mode 100644 docs/db.rst create mode 100644 swh/core/db/sql/35-dbmetadata.sql rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%) rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%) rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%) create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql create mode 100644 swh/core/db/tests/data/cli_new/upgrades/001.sql create mode 100644 swh/core/db/tests/data/cli_new/upgrades/002.sql create mode 100644 swh/core/db/tests/data/cli_new/upgrades/003.sql create mode 100644 swh/core/db/tests/data/cli_new/upgrades/004.sql create mode 100644 swh/core/db/tests/data/cli_new/upgrades/005.sql create mode 100644 swh/core/db/tests/data/cli_new/upgrades/006.sql create mode 100644 swh/core/db/tests/test_db_utils.pyChanges applied before test
commit 4c28b1c7b64f976c81fd19592d5ef44013cb8306 Author: David Douard <david.douard@sdfa3.org> Date: Fri Feb 11 15:54:25 2022 +0100 Add a documentation section for swh.core.db this is far from completion. commit ddce7a1485f39277da9773c4aed494459ffd12eb Author: David Douard <david.douard@sdfa3.org> Date: Mon Feb 7 15:46:48 2022 +0100 Refactor swh.core.db.pytest_plugin update the postgresql_fact fixture to be able to use new-ish pytest_postgresql ability to recreate databases between tests from a template database. The ides is to stop using the fragile and confusing "reset" of the database between tests, which means de facto keep a stateful database between tests, which make tests result depend on de facto order of execution of these tests. For example, test_smoke_test_people_db_up_and_reset cannot be executed alone, since it depends on the fact test_smoke_test_people_db_up has been executed before so that the people's db has been reset as a result of the test setup done by the fixture, which will happen if and only if the database already exists (thus if another test using this db has been executed before in this test session). The old table truncate based behavior has been kept (using a no_db_drop flag argument) in this diff for bw compatibility reasons, and also because the new drop-and-create-from-template beahvior remains a bit slower, which might be a problem for some test cases. Tests for dependencies (swh-storage, etc.) should still pass with this refactoring, but will be significantly slower. A series of revisions for these dependencies is about to be proposed to adapt them to the new db metadata (version, module, flavor) handling feature coming with swh.core 2, as well as updating their tests to take advantage of the template-based db creation in tests, which should bring back test execution time to almost the same as before this refactoring. This is needed to be able to implement and test support for the new 'swh db upgrade' all this is actually about. commit 528371a0f84aff02604ea51c6e24fe97ebd3de7a Author: David Douard <david.douard@sdfa3.org> Date: Tue Feb 1 15:43:09 2022 +0100 Add a 'swh db upgrade' command this new command allows to handle the database migration from the current version to the latest version advertized but the backend datastore. Thie feature is only working for "new style" swh.core.db based datastores. commit 544e184045403c735c33e6029f8bb7fa1a4df5cb Author: David Douard <david.douard@sdfa3.org> Date: Tue Feb 1 14:51:43 2022 +0100 Make 'swh db init' fill the dbversion table for this feature to work properly, the backend datastore package must provide the 'get_datastore' factory function in its root namespace and the datastore instance must provide a 'get_current_version()' method. This also means the canonical cli usage for the `swh db init` is now using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`) instead of giving only the db cnx string (`--db-name`), so that the backend datastore class can be instanciated using the `get_datastore()` factory function. However a '--initial-version' cli option has been added to 'swh db init' to make it easier, especially for unit tests. commit adeb54f770209284f0924a7e4d5bd5977a567527 Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 17:54:01 2022 +0100 Use a standard postgresql db fixture in test_cli instead of a custom 'test_db' (which did not truncate dbversion and origin tables, not sure what the intent was for this). commit d50695c3feb57b1e21d93e78364a943232fe91ff Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 15:02:51 2022 +0100 Add support for dbversion and dbmodule handling in swh db init for now, managing the dbversion table (storing the db schema upgrade history) is partially the responsibility of the swh modules implementing a datastore (swh-storage etc.), and the actual swh module was not stored in the database. This adds support for both the dbversion and dbmodule tables management in swh.core.db. The `swh db init` command now creates both these tables and initialize their values if possible. For this, it introduces a new common "API" for swh.core.db based datastores: - a swh module implementing a datastore is expected to have a `get_datastore` function in its top namespace; this function is typically the actual `get_storage`, `get_scheduler` etc. functions - this factory function is expected to use the "postgresql" cls for the datastore based on swh.core.db, - the datastore instance (eg. instance of the swh.storage.postgresql.Storage class) is expected to have a `get_current_version()` method (returning an int). This revision also provides a new `swh db version` command displaying the current (code) version (if available), the db version (possibly the whole version history) and flavor (if any) for the datastore given as argument. It should be backward compatible with existing swh datastore modules (i.e. not yet adapted to this new API). commit b1e449fd91ebcc403966a79fd62d449921fcf550 Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 16:00:43 2022 +0100 Refactor the mock_package_sql fixture to allow multiple sql script set will be used to implement several test scenarios in following commits. Also rename existing sql scripts with a "legit" numbering scheme.Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/319/ See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/319/console
-
Build is green
Patch application report for D7159 (id=26036)
Could not rebase; Attempt merge onto 3139d334...
Updating 3139d33..33ae358 Fast-forward MANIFEST.in | 2 + docs/db.rst | 181 +++++++++++++ docs/index.rst | 1 + swh/core/cli/db.py | 200 +++++++++++++- swh/core/db/db_utils.py | 293 +++++++++++++++++++-- swh/core/db/pytest_plugin.py | 213 ++++++++++----- swh/core/db/sql/35-dbversion.sql | 18 ++ swh/core/db/sql/36-dbmodule.sql | 15 ++ swh/core/db/tests/conftest.py | 50 ++++ .../tests/data/cli/{ => sql}/0-superuser-init.sql | 0 .../data/cli/{1-schema.sql => sql/30-schema.sql} | 2 +- .../data/cli/{3-func.sql => sql/40-funcs.sql} | 0 .../tests/data/cli/{4-data.sql => sql/50-data.sql} | 2 +- .../db/tests/data/cli_new/sql/0-superuser-init.sql | 1 + swh/core/db/tests/data/cli_new/sql/30-schema.sql | 6 + swh/core/db/tests/data/cli_new/sql/40-funcs.sql | 6 + swh/core/db/tests/data/cli_new/sql/50-data.sql | 2 + .../db/tests/data/cli_new/sql/upgrades/001.sql | 5 + .../db/tests/data/cli_new/sql/upgrades/002.sql | 4 + .../db/tests/data/cli_new/sql/upgrades/003.sql | 4 + .../db/tests/data/cli_new/sql/upgrades/004.sql | 4 + .../db/tests/data/cli_new/sql/upgrades/005.sql | 4 + .../db/tests/data/cli_new/sql/upgrades/006.sql | 7 + .../db/tests/pytest_plugin/test_pytest_plugin.py | 26 +- swh/core/db/tests/test_cli.py | 214 +++++++++++---- swh/core/db/tests/test_db_utils.py | 181 +++++++++++++ swh/core/tests/test_utils.py | 5 + swh/core/utils.py | 5 + 28 files changed, 1305 insertions(+), 146 deletions(-) create mode 100644 docs/db.rst create mode 100644 swh/core/db/sql/35-dbversion.sql create mode 100644 swh/core/db/sql/36-dbmodule.sql rename swh/core/db/tests/data/cli/{ => sql}/0-superuser-init.sql (100%) rename swh/core/db/tests/data/cli/{1-schema.sql => sql/30-schema.sql} (87%) rename swh/core/db/tests/data/cli/{3-func.sql => sql/40-funcs.sql} (100%) rename swh/core/db/tests/data/cli/{4-data.sql => sql/50-data.sql} (73%) create mode 100644 swh/core/db/tests/data/cli_new/sql/0-superuser-init.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/30-schema.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/40-funcs.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/50-data.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/001.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/002.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/003.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/004.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/005.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/006.sql create mode 100644 swh/core/db/tests/test_db_utils.pyChanges applied before test
commit 33ae35845b07e2651cb407a1430802288dd56ad1 Author: David Douard <david.douard@sdfa3.org> Date: Fri Feb 11 15:54:25 2022 +0100 Add a documentation section for swh.core.db this is far from completion. commit a06d92f6b608df9b057f27e2c59fc1b9785779a0 Author: David Douard <david.douard@sdfa3.org> Date: Wed Feb 16 12:37:07 2022 +0100 Make get_sql_for_package() return pathlib.Path objects and replace mock_package_sql by a mock_import_swhmodule fixture in favor of using the stock get_sql_for_package() function (on the mock module returned by mock_import_swhmodule). This also comes with a change in expected test data structure for sql script; the `tests/data/<mod>` is now expected to store the sql script files in a dedicated `sql/` sub directory instead of having them directly in the `<mod>` directory so that the get_sql_for_package() can be used in tests unmocked. commit d187e7e4574bb190a3af8254ec116a89a59e34bf Author: David Douard <david.douard@sdfa3.org> Date: Mon Feb 7 15:46:48 2022 +0100 Refactor swh.core.db.pytest_plugin update the postgresql_fact fixture to be able to use new-ish pytest_postgresql ability to recreate databases between tests from a template database. The ides is to stop using the fragile and confusing "reset" of the database between tests, which means de facto keep a stateful database between tests, which make tests result depend on de facto order of execution of these tests. For example, test_smoke_test_people_db_up_and_reset cannot be executed alone, since it depends on the fact test_smoke_test_people_db_up has been executed before so that the people's db has been reset as a result of the test setup done by the fixture, which will happen if and only if the database already exists (thus if another test using this db has been executed before in this test session). The old table truncate based behavior has been kept (using a no_db_drop flag argument) in this diff for bw compatibility reasons, and also because the new drop-and-create-from-template beahvior remains a bit slower, which might be a problem for some test cases. Tests for dependencies (swh-storage, etc.) should still pass with this refactoring, but will be significantly slower. A series of revisions for these dependencies is about to be proposed to adapt them to the new db metadata (version, module, flavor) handling feature coming with swh.core 2, as well as updating their tests to take advantage of the template-based db creation in tests, which should bring back test execution time to almost the same as before this refactoring. This is needed to be able to implement and test support for the new 'swh db upgrade' all this is actually about. commit dfa01610264a925ba02e33c6f2401d34fd34ec18 Author: David Douard <david.douard@sdfa3.org> Date: Tue Feb 1 15:43:09 2022 +0100 Add a 'swh db upgrade' command this new command allows to handle the database migration from the current version to the latest version advertized but the backend datastore. Thie feature is only working for "new style" swh.core.db based datastores. commit 544e184045403c735c33e6029f8bb7fa1a4df5cb Author: David Douard <david.douard@sdfa3.org> Date: Tue Feb 1 14:51:43 2022 +0100 Make 'swh db init' fill the dbversion table for this feature to work properly, the backend datastore package must provide the 'get_datastore' factory function in its root namespace and the datastore instance must provide a 'get_current_version()' method. This also means the canonical cli usage for the `swh db init` is now using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`) instead of giving only the db cnx string (`--db-name`), so that the backend datastore class can be instanciated using the `get_datastore()` factory function. However a '--initial-version' cli option has been added to 'swh db init' to make it easier, especially for unit tests. commit adeb54f770209284f0924a7e4d5bd5977a567527 Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 17:54:01 2022 +0100 Use a standard postgresql db fixture in test_cli instead of a custom 'test_db' (which did not truncate dbversion and origin tables, not sure what the intent was for this). commit d50695c3feb57b1e21d93e78364a943232fe91ff Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 15:02:51 2022 +0100 Add support for dbversion and dbmodule handling in swh db init for now, managing the dbversion table (storing the db schema upgrade history) is partially the responsibility of the swh modules implementing a datastore (swh-storage etc.), and the actual swh module was not stored in the database. This adds support for both the dbversion and dbmodule tables management in swh.core.db. The `swh db init` command now creates both these tables and initialize their values if possible. For this, it introduces a new common "API" for swh.core.db based datastores: - a swh module implementing a datastore is expected to have a `get_datastore` function in its top namespace; this function is typically the actual `get_storage`, `get_scheduler` etc. functions - this factory function is expected to use the "postgresql" cls for the datastore based on swh.core.db, - the datastore instance (eg. instance of the swh.storage.postgresql.Storage class) is expected to have a `get_current_version()` method (returning an int). This revision also provides a new `swh db version` command displaying the current (code) version (if available), the db version (possibly the whole version history) and flavor (if any) for the datastore given as argument. It should be backward compatible with existing swh datastore modules (i.e. not yet adapted to this new API). commit b1e449fd91ebcc403966a79fd62d449921fcf550 Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 16:00:43 2022 +0100 Refactor the mock_package_sql fixture to allow multiple sql script set will be used to implement several test scenarios in following commits. Also rename existing sql scripts with a "legit" numbering scheme.See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/327/ for more details.
-
Build is green
Patch application report for D7159 (id=26038)
Could not rebase; Attempt merge onto 3139d334...
Updating 3139d33..bb77dc0 Fast-forward MANIFEST.in | 2 + docs/db.rst | 176 +++++++++++++ docs/index.rst | 1 + swh/core/cli/db.py | 200 +++++++++++++- swh/core/db/db_utils.py | 293 +++++++++++++++++++-- swh/core/db/pytest_plugin.py | 213 ++++++++++----- swh/core/db/sql/35-dbversion.sql | 18 ++ swh/core/db/sql/36-dbmodule.sql | 15 ++ swh/core/db/tests/conftest.py | 50 ++++ .../tests/data/cli/{ => sql}/0-superuser-init.sql | 0 .../data/cli/{1-schema.sql => sql/30-schema.sql} | 2 +- .../data/cli/{3-func.sql => sql/40-funcs.sql} | 0 .../tests/data/cli/{4-data.sql => sql/50-data.sql} | 2 +- .../db/tests/data/cli_new/sql/0-superuser-init.sql | 1 + swh/core/db/tests/data/cli_new/sql/30-schema.sql | 6 + swh/core/db/tests/data/cli_new/sql/40-funcs.sql | 6 + swh/core/db/tests/data/cli_new/sql/50-data.sql | 2 + .../db/tests/data/cli_new/sql/upgrades/001.sql | 5 + .../db/tests/data/cli_new/sql/upgrades/002.sql | 4 + .../db/tests/data/cli_new/sql/upgrades/003.sql | 4 + .../db/tests/data/cli_new/sql/upgrades/004.sql | 4 + .../db/tests/data/cli_new/sql/upgrades/005.sql | 4 + .../db/tests/data/cli_new/sql/upgrades/006.sql | 7 + .../db/tests/pytest_plugin/test_pytest_plugin.py | 26 +- swh/core/db/tests/test_cli.py | 214 +++++++++++---- swh/core/db/tests/test_db_utils.py | 181 +++++++++++++ swh/core/tests/test_utils.py | 5 + swh/core/utils.py | 5 + 28 files changed, 1300 insertions(+), 146 deletions(-) create mode 100644 docs/db.rst create mode 100644 swh/core/db/sql/35-dbversion.sql create mode 100644 swh/core/db/sql/36-dbmodule.sql rename swh/core/db/tests/data/cli/{ => sql}/0-superuser-init.sql (100%) rename swh/core/db/tests/data/cli/{1-schema.sql => sql/30-schema.sql} (87%) rename swh/core/db/tests/data/cli/{3-func.sql => sql/40-funcs.sql} (100%) rename swh/core/db/tests/data/cli/{4-data.sql => sql/50-data.sql} (73%) create mode 100644 swh/core/db/tests/data/cli_new/sql/0-superuser-init.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/30-schema.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/40-funcs.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/50-data.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/001.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/002.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/003.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/004.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/005.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/006.sql create mode 100644 swh/core/db/tests/test_db_utils.pyChanges applied before test
commit bb77dc02d7cbefc989561e292c88e53aecfa8100 Author: David Douard <david.douard@sdfa3.org> Date: Fri Feb 11 15:54:25 2022 +0100 Add a documentation section for swh.core.db this is far from completion. commit a06d92f6b608df9b057f27e2c59fc1b9785779a0 Author: David Douard <david.douard@sdfa3.org> Date: Wed Feb 16 12:37:07 2022 +0100 Make get_sql_for_package() return pathlib.Path objects and replace mock_package_sql by a mock_import_swhmodule fixture in favor of using the stock get_sql_for_package() function (on the mock module returned by mock_import_swhmodule). This also comes with a change in expected test data structure for sql script; the `tests/data/<mod>` is now expected to store the sql script files in a dedicated `sql/` sub directory instead of having them directly in the `<mod>` directory so that the get_sql_for_package() can be used in tests unmocked. commit d187e7e4574bb190a3af8254ec116a89a59e34bf Author: David Douard <david.douard@sdfa3.org> Date: Mon Feb 7 15:46:48 2022 +0100 Refactor swh.core.db.pytest_plugin update the postgresql_fact fixture to be able to use new-ish pytest_postgresql ability to recreate databases between tests from a template database. The ides is to stop using the fragile and confusing "reset" of the database between tests, which means de facto keep a stateful database between tests, which make tests result depend on de facto order of execution of these tests. For example, test_smoke_test_people_db_up_and_reset cannot be executed alone, since it depends on the fact test_smoke_test_people_db_up has been executed before so that the people's db has been reset as a result of the test setup done by the fixture, which will happen if and only if the database already exists (thus if another test using this db has been executed before in this test session). The old table truncate based behavior has been kept (using a no_db_drop flag argument) in this diff for bw compatibility reasons, and also because the new drop-and-create-from-template beahvior remains a bit slower, which might be a problem for some test cases. Tests for dependencies (swh-storage, etc.) should still pass with this refactoring, but will be significantly slower. A series of revisions for these dependencies is about to be proposed to adapt them to the new db metadata (version, module, flavor) handling feature coming with swh.core 2, as well as updating their tests to take advantage of the template-based db creation in tests, which should bring back test execution time to almost the same as before this refactoring. This is needed to be able to implement and test support for the new 'swh db upgrade' all this is actually about. commit dfa01610264a925ba02e33c6f2401d34fd34ec18 Author: David Douard <david.douard@sdfa3.org> Date: Tue Feb 1 15:43:09 2022 +0100 Add a 'swh db upgrade' command this new command allows to handle the database migration from the current version to the latest version advertized but the backend datastore. Thie feature is only working for "new style" swh.core.db based datastores. commit 544e184045403c735c33e6029f8bb7fa1a4df5cb Author: David Douard <david.douard@sdfa3.org> Date: Tue Feb 1 14:51:43 2022 +0100 Make 'swh db init' fill the dbversion table for this feature to work properly, the backend datastore package must provide the 'get_datastore' factory function in its root namespace and the datastore instance must provide a 'get_current_version()' method. This also means the canonical cli usage for the `swh db init` is now using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`) instead of giving only the db cnx string (`--db-name`), so that the backend datastore class can be instanciated using the `get_datastore()` factory function. However a '--initial-version' cli option has been added to 'swh db init' to make it easier, especially for unit tests. commit adeb54f770209284f0924a7e4d5bd5977a567527 Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 17:54:01 2022 +0100 Use a standard postgresql db fixture in test_cli instead of a custom 'test_db' (which did not truncate dbversion and origin tables, not sure what the intent was for this). commit d50695c3feb57b1e21d93e78364a943232fe91ff Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 15:02:51 2022 +0100 Add support for dbversion and dbmodule handling in swh db init for now, managing the dbversion table (storing the db schema upgrade history) is partially the responsibility of the swh modules implementing a datastore (swh-storage etc.), and the actual swh module was not stored in the database. This adds support for both the dbversion and dbmodule tables management in swh.core.db. The `swh db init` command now creates both these tables and initialize their values if possible. For this, it introduces a new common "API" for swh.core.db based datastores: - a swh module implementing a datastore is expected to have a `get_datastore` function in its top namespace; this function is typically the actual `get_storage`, `get_scheduler` etc. functions - this factory function is expected to use the "postgresql" cls for the datastore based on swh.core.db, - the datastore instance (eg. instance of the swh.storage.postgresql.Storage class) is expected to have a `get_current_version()` method (returning an int). This revision also provides a new `swh db version` command displaying the current (code) version (if available), the db version (possibly the whole version history) and flavor (if any) for the datastore given as argument. It should be backward compatible with existing swh datastore modules (i.e. not yet adapted to this new API). commit b1e449fd91ebcc403966a79fd62d449921fcf550 Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 16:00:43 2022 +0100 Refactor the mock_package_sql fixture to allow multiple sql script set will be used to implement several test scenarios in following commits. Also rename existing sql scripts with a "legit" numbering scheme.See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/329/ for more details.
-
Build is green
Patch application report for D7159 (id=26043)
Could not rebase; Attempt merge onto 3139d334...
Updating 3139d33..8646bc1 Fast-forward MANIFEST.in | 2 + docs/db.rst | 176 ++++++++++++ docs/index.rst | 1 + swh/core/cli/db.py | 200 +++++++++++++- swh/core/db/db_utils.py | 298 +++++++++++++++++++-- swh/core/db/pytest_plugin.py | 213 ++++++++++----- swh/core/db/sql/35-dbversion.sql | 18 ++ swh/core/db/sql/36-dbmodule.sql | 15 ++ swh/core/db/tests/conftest.py | 50 ++++ .../tests/data/cli/{ => sql}/0-superuser-init.sql | 0 .../data/cli/{1-schema.sql => sql/30-schema.sql} | 2 +- .../data/cli/{3-func.sql => sql/40-funcs.sql} | 0 .../tests/data/cli/{4-data.sql => sql/50-data.sql} | 2 +- .../db/tests/data/cli_new/sql/0-superuser-init.sql | 1 + swh/core/db/tests/data/cli_new/sql/30-schema.sql | 6 + swh/core/db/tests/data/cli_new/sql/40-funcs.sql | 6 + swh/core/db/tests/data/cli_new/sql/50-data.sql | 2 + .../db/tests/data/cli_new/sql/upgrades/001.sql | 5 + .../db/tests/data/cli_new/sql/upgrades/002.sql | 4 + .../db/tests/data/cli_new/sql/upgrades/003.sql | 4 + .../db/tests/data/cli_new/sql/upgrades/004.sql | 4 + .../db/tests/data/cli_new/sql/upgrades/005.sql | 4 + .../db/tests/data/cli_new/sql/upgrades/006.sql | 7 + .../db/tests/pytest_plugin/test_pytest_plugin.py | 26 +- swh/core/db/tests/test_cli.py | 214 +++++++++++---- swh/core/db/tests/test_db_utils.py | 181 +++++++++++++ swh/core/tests/test_utils.py | 5 + swh/core/utils.py | 5 + 28 files changed, 1305 insertions(+), 146 deletions(-) create mode 100644 docs/db.rst create mode 100644 swh/core/db/sql/35-dbversion.sql create mode 100644 swh/core/db/sql/36-dbmodule.sql rename swh/core/db/tests/data/cli/{ => sql}/0-superuser-init.sql (100%) rename swh/core/db/tests/data/cli/{1-schema.sql => sql/30-schema.sql} (87%) rename swh/core/db/tests/data/cli/{3-func.sql => sql/40-funcs.sql} (100%) rename swh/core/db/tests/data/cli/{4-data.sql => sql/50-data.sql} (73%) create mode 100644 swh/core/db/tests/data/cli_new/sql/0-superuser-init.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/30-schema.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/40-funcs.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/50-data.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/001.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/002.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/003.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/004.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/005.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/006.sql create mode 100644 swh/core/db/tests/test_db_utils.pyChanges applied before test
commit 8646bc13577331bae62c6536dae05cd470be0295 Author: David Douard <david.douard@sdfa3.org> Date: Fri Feb 11 15:54:25 2022 +0100 Add a documentation section for swh.core.db this is far from completion. commit ef48c771fe9ef7f797ecc49672e4aa470b1e3a1f Author: David Douard <david.douard@sdfa3.org> Date: Wed Feb 16 12:37:07 2022 +0100 Make get_sql_for_package() return pathlib.Path objects and replace mock_package_sql by a mock_import_swhmodule fixture in favor of using the stock get_sql_for_package() function (on the mock module returned by mock_import_swhmodule). This also comes with a change in expected test data structure for sql script; the `tests/data/<mod>` is now expected to store the sql script files in a dedicated `sql/` sub directory instead of having them directly in the `<mod>` directory so that the get_sql_for_package() can be used in tests unmocked. commit 9e599f03076f85b44ac0eb5d001ff6367a930276 Author: David Douard <david.douard@sdfa3.org> Date: Mon Feb 7 15:46:48 2022 +0100 Refactor swh.core.db.pytest_plugin update the postgresql_fact fixture to be able to use new-ish pytest_postgresql ability to recreate databases between tests from a template database. The ides is to stop using the fragile and confusing "reset" of the database between tests, which means de facto keep a stateful database between tests, which make tests result depend on de facto order of execution of these tests. For example, test_smoke_test_people_db_up_and_reset cannot be executed alone, since it depends on the fact test_smoke_test_people_db_up has been executed before so that the people's db has been reset as a result of the test setup done by the fixture, which will happen if and only if the database already exists (thus if another test using this db has been executed before in this test session). The old table truncate based behavior has been kept (using a no_db_drop flag argument) in this diff for bw compatibility reasons, and also because the new drop-and-create-from-template beahvior remains a bit slower, which might be a problem for some test cases. Tests for dependencies (swh-storage, etc.) should still pass with this refactoring, but will be significantly slower. A series of revisions for these dependencies is about to be proposed to adapt them to the new db metadata (version, module, flavor) handling feature coming with swh.core 2, as well as updating their tests to take advantage of the template-based db creation in tests, which should bring back test execution time to almost the same as before this refactoring. This is needed to be able to implement and test support for the new 'swh db upgrade' all this is actually about. commit 1a187ca7787503d41ac13b2678b29f332d8fd200 Author: David Douard <david.douard@sdfa3.org> Date: Tue Feb 1 15:43:09 2022 +0100 Add a 'swh db upgrade' command this new command allows to handle the database migration from the current version to the latest version advertized but the backend datastore. Thie feature is only working for "new style" swh.core.db based datastores. commit 544e184045403c735c33e6029f8bb7fa1a4df5cb Author: David Douard <david.douard@sdfa3.org> Date: Tue Feb 1 14:51:43 2022 +0100 Make 'swh db init' fill the dbversion table for this feature to work properly, the backend datastore package must provide the 'get_datastore' factory function in its root namespace and the datastore instance must provide a 'get_current_version()' method. This also means the canonical cli usage for the `swh db init` is now using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`) instead of giving only the db cnx string (`--db-name`), so that the backend datastore class can be instanciated using the `get_datastore()` factory function. However a '--initial-version' cli option has been added to 'swh db init' to make it easier, especially for unit tests. commit adeb54f770209284f0924a7e4d5bd5977a567527 Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 17:54:01 2022 +0100 Use a standard postgresql db fixture in test_cli instead of a custom 'test_db' (which did not truncate dbversion and origin tables, not sure what the intent was for this). commit d50695c3feb57b1e21d93e78364a943232fe91ff Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 15:02:51 2022 +0100 Add support for dbversion and dbmodule handling in swh db init for now, managing the dbversion table (storing the db schema upgrade history) is partially the responsibility of the swh modules implementing a datastore (swh-storage etc.), and the actual swh module was not stored in the database. This adds support for both the dbversion and dbmodule tables management in swh.core.db. The `swh db init` command now creates both these tables and initialize their values if possible. For this, it introduces a new common "API" for swh.core.db based datastores: - a swh module implementing a datastore is expected to have a `get_datastore` function in its top namespace; this function is typically the actual `get_storage`, `get_scheduler` etc. functions - this factory function is expected to use the "postgresql" cls for the datastore based on swh.core.db, - the datastore instance (eg. instance of the swh.storage.postgresql.Storage class) is expected to have a `get_current_version()` method (returning an int). This revision also provides a new `swh db version` command displaying the current (code) version (if available), the db version (possibly the whole version history) and flavor (if any) for the datastore given as argument. It should be backward compatible with existing swh datastore modules (i.e. not yet adapted to this new API). commit b1e449fd91ebcc403966a79fd62d449921fcf550 Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 16:00:43 2022 +0100 Refactor the mock_package_sql fixture to allow multiple sql script set will be used to implement several test scenarios in following commits. Also rename existing sql scripts with a "legit" numbering scheme.See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/334/ for more details.
-
Build is green
Patch application report for D7159 (id=26047)
Could not rebase; Attempt merge onto 3139d334...
Updating 3139d33..3cc57bb Fast-forward MANIFEST.in | 2 + docs/db.rst | 176 ++++++++++++ docs/index.rst | 1 + swh/core/cli/db.py | 200 +++++++++++++- swh/core/db/db_utils.py | 298 +++++++++++++++++++-- swh/core/db/pytest_plugin.py | 230 +++++++++++----- swh/core/db/sql/35-dbversion.sql | 18 ++ swh/core/db/sql/36-dbmodule.sql | 15 ++ swh/core/db/tests/conftest.py | 50 ++++ .../tests/data/cli/{ => sql}/0-superuser-init.sql | 0 .../data/cli/{1-schema.sql => sql/30-schema.sql} | 2 +- .../data/cli/{3-func.sql => sql/40-funcs.sql} | 0 .../tests/data/cli/{4-data.sql => sql/50-data.sql} | 2 +- .../db/tests/data/cli_new/sql/0-superuser-init.sql | 1 + swh/core/db/tests/data/cli_new/sql/30-schema.sql | 6 + swh/core/db/tests/data/cli_new/sql/40-funcs.sql | 6 + swh/core/db/tests/data/cli_new/sql/50-data.sql | 2 + .../db/tests/data/cli_new/sql/upgrades/001.sql | 5 + .../db/tests/data/cli_new/sql/upgrades/002.sql | 4 + .../db/tests/data/cli_new/sql/upgrades/003.sql | 4 + .../db/tests/data/cli_new/sql/upgrades/004.sql | 4 + .../db/tests/data/cli_new/sql/upgrades/005.sql | 4 + .../db/tests/data/cli_new/sql/upgrades/006.sql | 7 + .../db/tests/pytest_plugin/test_pytest_plugin.py | 26 +- swh/core/db/tests/test_cli.py | 214 +++++++++++---- swh/core/db/tests/test_db_utils.py | 181 +++++++++++++ swh/core/tests/test_utils.py | 5 + swh/core/utils.py | 5 + 28 files changed, 1321 insertions(+), 147 deletions(-) create mode 100644 docs/db.rst create mode 100644 swh/core/db/sql/35-dbversion.sql create mode 100644 swh/core/db/sql/36-dbmodule.sql rename swh/core/db/tests/data/cli/{ => sql}/0-superuser-init.sql (100%) rename swh/core/db/tests/data/cli/{1-schema.sql => sql/30-schema.sql} (87%) rename swh/core/db/tests/data/cli/{3-func.sql => sql/40-funcs.sql} (100%) rename swh/core/db/tests/data/cli/{4-data.sql => sql/50-data.sql} (73%) create mode 100644 swh/core/db/tests/data/cli_new/sql/0-superuser-init.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/30-schema.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/40-funcs.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/50-data.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/001.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/002.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/003.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/004.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/005.sql create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/006.sql create mode 100644 swh/core/db/tests/test_db_utils.pyChanges applied before test
commit 3cc57bb4e1d2dbacdc8a7958281a55cd469d01ca Author: David Douard <david.douard@sdfa3.org> Date: Fri Feb 11 15:54:25 2022 +0100 Add a documentation section for swh.core.db this is far from completion. commit 9e6f8fd036c86602be5f7c7d7df6c3836006aff5 Author: David Douard <david.douard@sdfa3.org> Date: Wed Feb 16 12:37:07 2022 +0100 Make get_sql_for_package() return pathlib.Path objects and replace mock_package_sql by a mock_import_swhmodule fixture in favor of using the stock get_sql_for_package() function (on the mock module returned by mock_import_swhmodule). This also comes with a change in expected test data structure for sql script; the `tests/data/<mod>` is now expected to store the sql script files in a dedicated `sql/` sub directory instead of having them directly in the `<mod>` directory so that the get_sql_for_package() can be used in tests unmocked. commit 447027525f976aec9ef117a4502e6495da0628eb Author: David Douard <david.douard@sdfa3.org> Date: Mon Feb 7 15:46:48 2022 +0100 Refactor swh.core.db.pytest_plugin update the postgresql_fact fixture to be able to use new-ish pytest_postgresql ability to recreate databases between tests from a template database. The ides is to stop using the fragile and confusing "reset" of the database between tests, which means de facto keep a stateful database between tests, which make tests result depend on de facto order of execution of these tests. For example, test_smoke_test_people_db_up_and_reset cannot be executed alone, since it depends on the fact test_smoke_test_people_db_up has been executed before so that the people's db has been reset as a result of the test setup done by the fixture, which will happen if and only if the database already exists (thus if another test using this db has been executed before in this test session). The old table truncate based behavior has been kept (using a no_db_drop flag argument) in this diff for bw compatibility reasons, and also because the new drop-and-create-from-template beahvior remains a bit slower, which might be a problem for some test cases. Tests for dependencies (swh-storage, etc.) should still pass with this refactoring, but will be significantly slower. A series of revisions for these dependencies is about to be proposed to adapt them to the new db metadata (version, module, flavor) handling feature coming with swh.core 2, as well as updating their tests to take advantage of the template-based db creation in tests, which should bring back test execution time to almost the same as before this refactoring. This is needed to be able to implement and test support for the new 'swh db upgrade' all this is actually about. commit 1a187ca7787503d41ac13b2678b29f332d8fd200 Author: David Douard <david.douard@sdfa3.org> Date: Tue Feb 1 15:43:09 2022 +0100 Add a 'swh db upgrade' command this new command allows to handle the database migration from the current version to the latest version advertized but the backend datastore. Thie feature is only working for "new style" swh.core.db based datastores. commit 544e184045403c735c33e6029f8bb7fa1a4df5cb Author: David Douard <david.douard@sdfa3.org> Date: Tue Feb 1 14:51:43 2022 +0100 Make 'swh db init' fill the dbversion table for this feature to work properly, the backend datastore package must provide the 'get_datastore' factory function in its root namespace and the datastore instance must provide a 'get_current_version()' method. This also means the canonical cli usage for the `swh db init` is now using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`) instead of giving only the db cnx string (`--db-name`), so that the backend datastore class can be instanciated using the `get_datastore()` factory function. However a '--initial-version' cli option has been added to 'swh db init' to make it easier, especially for unit tests. commit adeb54f770209284f0924a7e4d5bd5977a567527 Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 17:54:01 2022 +0100 Use a standard postgresql db fixture in test_cli instead of a custom 'test_db' (which did not truncate dbversion and origin tables, not sure what the intent was for this). commit d50695c3feb57b1e21d93e78364a943232fe91ff Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 15:02:51 2022 +0100 Add support for dbversion and dbmodule handling in swh db init for now, managing the dbversion table (storing the db schema upgrade history) is partially the responsibility of the swh modules implementing a datastore (swh-storage etc.), and the actual swh module was not stored in the database. This adds support for both the dbversion and dbmodule tables management in swh.core.db. The `swh db init` command now creates both these tables and initialize their values if possible. For this, it introduces a new common "API" for swh.core.db based datastores: - a swh module implementing a datastore is expected to have a `get_datastore` function in its top namespace; this function is typically the actual `get_storage`, `get_scheduler` etc. functions - this factory function is expected to use the "postgresql" cls for the datastore based on swh.core.db, - the datastore instance (eg. instance of the swh.storage.postgresql.Storage class) is expected to have a `get_current_version()` method (returning an int). This revision also provides a new `swh db version` command displaying the current (code) version (if available), the db version (possibly the whole version history) and flavor (if any) for the datastore given as argument. It should be backward compatible with existing swh datastore modules (i.e. not yet adapted to this new API). commit b1e449fd91ebcc403966a79fd62d449921fcf550 Author: David Douard <david.douard@sdfa3.org> Date: Fri Jan 28 16:00:43 2022 +0100 Refactor the mock_package_sql fixture to allow multiple sql script set will be used to implement several test scenarios in following commits. Also rename existing sql scripts with a "legit" numbering scheme.See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/338/ for more details.
mentioned in merge request !321 (closed)
Please register or sign in to reply