Skip to content
Snippets Groups Projects

[RFC] Add basic migration tests for postgresql

Open [RFC] Add basic migration tests for postgresql
generated-differential-D5014-source into master
Open vlorentz requested to merge generated-differential-D5014-source into master

This adds two test files:

  • test_postgresql_migrated.py applies an old schema definition, runs the migrations, then runs all the usual tests
  • test_postgresql_migration.py applies an old schema definition, inserts data, runs the migrations, and checks the data is still available

test_postgresql_migration.py will probably break in some releases as it uses the old SQL with the new Python to insert, but it should be good enough, and we can disable it in some releases when needed.

What do you think?

Test Plan

tests fail because they don't work in tox (I'll look into later if we want to pursue this way of testing migrations)

Migrated from D5014 (view on Phabricator)

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build has FAILED

    Patch application report for D5014 (id=17885)

    Could not rebase; Attempt merge onto 9a9f234e...

    Updating 9a9f234e..27440f98
Fast-forward
 swh/storage/cassandra/common.py                    |   5 -
 swh/storage/cassandra/converters.py                |   2 +-
 swh/storage/cassandra/cql.py                       |   3 +-
 swh/storage/cassandra/storage.py                   |   3 +-
 swh/storage/postgresql/db.py                       |   9 +-
 .../tests/data/sql-v0.18.0/10-superuser-init.sql   |  27 +
 swh/storage/tests/data/sql-v0.18.0/15-flavor.sql   |  22 +
 swh/storage/tests/data/sql-v0.18.0/20-enums.sql    |  23 +
 swh/storage/tests/data/sql-v0.18.0/30-schema.sql   | 499 +++++++++++
 swh/storage/tests/data/sql-v0.18.0/40-funcs.sql    | 960 +++++++++++++++++++++
 swh/storage/tests/data/sql-v0.18.0/60-indexes.sql  | 283 ++++++
 .../logical_replication/replication_source.sql     |  25 +
 swh/storage/tests/storage_tests.py                 |  59 +-
 swh/storage/tests/test_postgresql_migrated.py      |  63 ++
 swh/storage/tests/test_postgresql_migration.py     | 194 +++++
 swh/storage/utils.py                               |   5 +
 16 files changed, 2158 insertions(+), 24 deletions(-)
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/10-superuser-init.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/15-flavor.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/20-enums.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/30-schema.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/40-funcs.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/60-indexes.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/logical_replication/replication_source.sql
 create mode 100644 swh/storage/tests/test_postgresql_migrated.py
 create mode 100644 swh/storage/tests/test_postgresql_migration.py
    Changes applied before test
    commit 27440f9862b7df4e36b400a4fb74f1679d9ec6f4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 4 14:36:50 2021 +0100

    Add basic migration tests for postgresql
    
    This adds two test files:
    
    * `test_postgresql_migrated.py` applies an old schema definition, runs the
      migrations, then runs all the usual tests
    * `test_postgresql_migration.py` applies an old schema definition, inserts
      data, runs the migrations, and checks the data is still available
    
    `test_postgresql_migration.py` will probably break in some releases as
    it uses the old SQL with the new Python to insert, but it should be good
    enough, and we can disable it in some releases when needed.

commit e9441fef13c11c3eb500403275ba7337ed0c77e1
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 4 14:29:38 2021 +0100

    postgresql: Fix dbversion() to return the max version instead of a random one.

commit 508399ce2abf21f813acc9c56422cbbccca0ae3d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 4 13:59:09 2021 +0100

    storage_tests: recompute ids when evolving RawExtrinsicMetadata objects.
    
    For now this does nothing as RawExtrinsicMetadata has no 'id' field,
    but the equality assertions will become errors when the next version
    of swh.model is released.

    Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1122/ See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1122/console

  • vlorentz
    vlorentz @vlorentz ·
    Author Maintainer

    rebase

  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build has FAILED

    Patch application report for D5014 (id=17970)

    Could not rebase; Attempt merge onto b0383833...

    Updating b0383833..ebe4dab0
Fast-forward
 swh/storage/cassandra/common.py                    |   5 -
 swh/storage/cassandra/converters.py                |   2 +-
 swh/storage/cassandra/cql.py                       |   3 +-
 swh/storage/cassandra/storage.py                   |   3 +-
 .../tests/data/sql-v0.18.0/10-superuser-init.sql   |  27 +
 swh/storage/tests/data/sql-v0.18.0/15-flavor.sql   |  22 +
 swh/storage/tests/data/sql-v0.18.0/20-enums.sql    |  23 +
 swh/storage/tests/data/sql-v0.18.0/30-schema.sql   | 499 +++++++++++
 swh/storage/tests/data/sql-v0.18.0/40-funcs.sql    | 960 +++++++++++++++++++++
 swh/storage/tests/data/sql-v0.18.0/60-indexes.sql  | 283 ++++++
 .../logical_replication/replication_source.sql     |  25 +
 swh/storage/tests/storage_tests.py                 |  59 +-
 swh/storage/tests/test_postgresql_migrated.py      |  63 ++
 swh/storage/tests/test_postgresql_migration.py     | 194 +++++
 swh/storage/utils.py                               |   5 +
 15 files changed, 2150 insertions(+), 23 deletions(-)
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/10-superuser-init.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/15-flavor.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/20-enums.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/30-schema.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/40-funcs.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/60-indexes.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/logical_replication/replication_source.sql
 create mode 100644 swh/storage/tests/test_postgresql_migrated.py
 create mode 100644 swh/storage/tests/test_postgresql_migration.py
    Changes applied before test
    commit ebe4dab07feed7eb8ae03cd407980ba7bea78b37
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 4 14:36:50 2021 +0100

    Add basic migration tests for postgresql
    
    This adds two test files:
    
    * `test_postgresql_migrated.py` applies an old schema definition, runs the
      migrations, then runs all the usual tests
    * `test_postgresql_migration.py` applies an old schema definition, inserts
      data, runs the migrations, and checks the data is still available
    
    `test_postgresql_migration.py` will probably break in some releases as
    it uses the old SQL with the new Python to insert, but it should be good
    enough, and we can disable it in some releases when needed.

commit 75939e4f9b02e130224a90929eb3ed2ba0730592
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 4 13:59:09 2021 +0100

    storage_tests: recompute ids when evolving RawExtrinsicMetadata objects.
    
    For now this does nothing as RawExtrinsicMetadata has no 'id' field,
    but the equality assertions will become errors when the next version
    of swh.model is released.

    Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1143/ See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1143/console

  • vlorentz
    vlorentz @vlorentz ·
    Author Maintainer

    rebase

  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build has FAILED

    Patch application report for D5014 (id=18605)

    Rebasing onto 88ff2c2f...

    Current branch diff-target is up to date.
    Changes applied before test
    commit e5fc49846c1d9ca9961b0b7edf7e0a993401daf6
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 4 14:36:50 2021 +0100

    Add basic migration tests for postgresql
    
    This adds two test files:
    
    * `test_postgresql_migrated.py` applies an old schema definition, runs the
      migrations, then runs all the usual tests
    * `test_postgresql_migration.py` applies an old schema definition, inserts
      data, runs the migrations, and checks the data is still available
    
    `test_postgresql_migration.py` will probably break in some releases as
    it uses the old SQL with the new Python to insert, but it should be good
    enough, and we can disable it in some releases when needed.

    Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1177/ See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1177/console

  • vlorentz
    vlorentz @vlorentz ·
    Author Maintainer

    rebase

  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build has FAILED

    Patch application report for D5014 (id=18608)

    Could not rebase; Attempt merge onto 88ff2c2f...

    Updating 88ff2c2f..e583c3cb
Fast-forward
 sql/upgrades/169.sql                               |  26 +
 swh/storage/postgresql/db.py                       |   5 +-
 swh/storage/postgresql/storage.py                  |   1 +
 swh/storage/sql/30-schema.sql                      |   4 +-
 swh/storage/sql/60-indexes.sql                     |   4 +
 .../tests/data/sql-v0.18.0/10-superuser-init.sql   |  27 +
 swh/storage/tests/data/sql-v0.18.0/15-flavor.sql   |  22 +
 swh/storage/tests/data/sql-v0.18.0/20-enums.sql    |  23 +
 swh/storage/tests/data/sql-v0.18.0/30-schema.sql   | 499 +++++++++++
 swh/storage/tests/data/sql-v0.18.0/40-funcs.sql    | 960 +++++++++++++++++++++
 swh/storage/tests/data/sql-v0.18.0/60-indexes.sql  | 283 ++++++
 .../logical_replication/replication_source.sql     |  25 +
 swh/storage/tests/test_postgresql_migrated.py      |  63 ++
 swh/storage/tests/test_postgresql_migration.py     | 194 +++++
 14 files changed, 2134 insertions(+), 2 deletions(-)
 create mode 100644 sql/upgrades/169.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/10-superuser-init.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/15-flavor.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/20-enums.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/30-schema.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/40-funcs.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/60-indexes.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/logical_replication/replication_source.sql
 create mode 100644 swh/storage/tests/test_postgresql_migrated.py
 create mode 100644 swh/storage/tests/test_postgresql_migration.py
    Changes applied before test
    commit e583c3cb0efb0805af0b88547bc67b7841e00e2b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 4 14:36:50 2021 +0100

    Add basic migration tests for postgresql
    
    This adds two test files:
    
    * `test_postgresql_migrated.py` applies an old schema definition, runs the
      migrations, then runs all the usual tests
    * `test_postgresql_migration.py` applies an old schema definition, inserts
      data, runs the migrations, and checks the data is still available
    
    `test_postgresql_migration.py` will probably break in some releases as
    it uses the old SQL with the new Python to insert, but it should be good
    enough, and we can disable it in some releases when needed.

commit 7ed181fe084152ebb2ad5b1792a8d2d9e5a1c429
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Feb 5 13:56:15 2021 +0100

    Add raw_extrinsic_metadata.id column in postgresql.
    
    For now, this has absolutely no effect on the API users,
    as rows are already deduplicated based on a subset of the
    fields hashed by the id.

    Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1180/ See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1180/console

  • Phabricator Migration user mentioned in merge request swh-core!206 (closed)

    mentioned in merge request swh-core!206 (closed)

  • Jérémy Bobbio (Lunar) added mr-reviewed-fall-2023 label

    added mr-reviewed-fall-2023 label

  • Jenkins Bot
    Jenkins Bot @jenkins ·
    Maintainer

    Jenkins job DSTO/gitlab-builds #400 failed .
    See Console Output and Coverage Report for more details.

Please register or sign in to reply