cassandra: Add framework to manage migrations

based on this design: https://gitlab.softwareheritage.org/product-management/core-platform/-/blob/main/specifications/cassandra_migrations/specification.md?ref_type=heads TODO:

• write more tests, because only the happy path is tested for now.
• write documentation

cc @olasd

Jenkins job DSTO/gitlab-builds #765 failed in 13 min.
See Console Output, Blue Ocean and Coverage Report for more details.

changed the description

added 3 commits

• 31604920 - sys.exit -> ctx.exit
• abe3d8eb - Move Cassandra initialization from test_cli to test_cli_cassandra
• 163baffb - Fix test_schema_matches_model

Compare with previous version

added 1 commit

• 8cf2e9e0 - Add tests

Compare with previous version

Jenkins job DSTO/gitlab-builds #767 failed in 14 min.
See Console Output, Blue Ocean and Coverage Report for more details.

added 1 commit

• e467a417 - Fix test_object_delete

Compare with previous version

marked this merge request as ready

Jenkins job DSTO/gitlab-builds #768 succeeded in 19 min.
See Console Output, Blue Ocean and Coverage Report for more details.

marked the checklist item write more tests, because only the happy path is tested for now. as completed

added 1 commit

• 55cbb7e0 - cli: Mark all migrations as applied when running 'create-keyspace'

Compare with previous version

added 1 commit

• 4d5bd31e - cli: Mark all migrations as applied when running 'create-keyspace'

Compare with previous version

Jenkins job DSTO/gitlab-builds #783 failed in 17 min.
See Console Output, Blue Ocean and Coverage Report for more details.

Jenkins job DSTO/gitlab-builds #782 failed in 18 min.
See Console Output, Blue Ocean and Coverage Report for more details.

added 1 commit

• 22898519 - Add documentation

Compare with previous version

Jenkins job DSTO/gitlab-builds #784 succeeded in 15 min.
See Console Output, Blue Ocean and Coverage Report for more details.

marked the checklist item write documentation as completed

added 6 commits

• 22898519...c2fd0f01 - 5 commits from branch master
• 71c70250 - Merge branch 'master' into migrations-framework

Compare with previous version

Jenkins job DSTO/gitlab-builds #799 succeeded in 15 min.
See Console Output, Blue Ocean and Coverage Report for more details.

Thanks, this looks really good.

The previous versions of swh.storage would have us use swh storage create-keyspace to apply table-creation-only migrations, so there was an expectation that the command was idempotent.

In the new implementation, those table creation migrations are expected to be actual declared migrations, correct?

The new implementation of swh storage cassandra init always marks all migrations as executed after creating the schema, which is fine, I think. However, I believe that this means that it also marks the migrations as executed when it /updates/ an existing schema, for instance when creating a new table, which doesn't seem all that good in general (even though it'd probably be fine for this one release). I'm thinking it should now just bail out if the keyspace isn't empty, and defer to the migration commands.

Finally, we should clarify what steps one should follow to bootstrap the migrations table on an existing migration-less schema?

Suppose a version was skipped, running (the current version of) swh storage cassandra init would actually ignore the migrations that still need to happen after adding the migrations table. I'm thinking the addition of the migrations table should also be implemented as the first migration, to be run somehow even if the migration table doesn't exist.

In the new implementation, those table creation migrations are expected to be actual declared migrations, correct?

correct

I'm thinking it should now just bail out if the keyspace isn't empty, and defer to the migration commands.

agreed

Finally, we should clarify what steps one should follow to bootstrap the migrations table on an existing migration-less schema?

see below

Suppose a version was skipped, [...]

indeed

I'm thinking the addition of the migrations table should also be implemented as the first migration, to be run somehow even if the migration table doesn't exist.

Sounds complicated. What if I add a new command specifically to add the migrations table?

I was thinking that the migration command could check + create the migrations table if it doesn't exist (even if that table creation is not strictly implemented as a migration), but I guess having the additional command to create the migrations table could also allow checking that the schema looks similar to what is expected at version 3.0.0-minus-one, and advise people to downgrade and run create-keyspace to adjust the schema one last time if it doesn't. Either way is fine with me.

I'm thinking the addition of the migrations table should also be implemented as the first migration, to be run somehow even if the migration table doesn't exist.

I went with this design in the end, it turned out to be simpler than i thought.

Yeah, seems fine to me.

Looks like the test failure is hypothesis_strategies missing an update for the restrictions in timestamp values?

added 1 commit

• 1ea27563 - Automatically create 'migration' table if missing

Compare with previous version