Make 'swh db init' fill the dbversion table

for this feature to work properly, the backend datastore package must provide the 'get_datastore' factory function in its root namespace and the datastore instance must provide a 'get_current_version()' method.

This also means the canonical cli usage for the swh db init is now using the config file (via --config-file or SWH_CONFIG_FILENAME) instead of giving only the db cnx string (--db-name), so that the backend datastore class can be instanciated using the get_datastore() factory function.

However a '--initial-version' cli option has been added to 'swh db init' to make it easier, especially for unit tests.

Also use a standard postgresql db fixture in test_cli instead of a custom 'test_db' (which did not truncate dbversion and origin tables, not sure what the intent was for this).

Depends on !315 (closed) Related to #3894 (closed)

---

Migrated from D7063 (view on Phabricator)

Build has FAILED

Patch application report for D7063 (id=25613)

Could not rebase; Attempt merge onto aba5c807...

Updating aba5c80..a65510a
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 264 ++++++++---------
 swh/core/db/db_utils.py                            | 330 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 swh/core/db/tests/test_cli.py                      | 123 +++++---
 swh/core/db/tests/test_db_utils.py                 |  73 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 11 files changed, 659 insertions(+), 239 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit a65510a5e1cbc9aca48457bbb413c83906fa1e05
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 9f5b365b013c44b9219eddc4d4f92c06906c1d73
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit 886c8c545fa424db720b4ec3cd1c8ec2de4e3952
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit cbc634d04ee9e57cdc42ebab2e087e1eac0031b5
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit 4f41cc905d805953363688316a088bc84100e181
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit 6c665e38e6579a02839eef6a8a6b60dc332be51c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit e4117fcf51a256c5b708f909c818f452c794c0f8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/268/ See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/268/console

rebase + fix

Build has FAILED

Patch application report for D7063 (id=25620)

Could not rebase; Attempt merge onto aba5c807...

Updating aba5c80..a4e3c6f
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 264 ++++++++---------
 swh/core/db/db_utils.py                            | 329 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 swh/core/db/tests/test_cli.py                      | 123 +++++---
 swh/core/db/tests/test_db_utils.py                 |  73 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 11 files changed, 658 insertions(+), 239 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit a4e3c6ff54903248fe2c581a2c920146208ad348
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 5ad23667d2f1acfef602236ebfe7e18ff72695d6
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit 54c53dbdc2574e5f8d9f699505a3217708bf8b54
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit c75374bafd77dfff82904985b935bbd035147cbc
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit 5c5b981020c4b3e38f89f112480b7297cceded5c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit d3b7b081fd3eda460d11d1d6d70f2b52fa3e0ef3
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit e4117fcf51a256c5b708f909c818f452c794c0f8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/273/ See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/273/console

rebase

Build has FAILED

Patch application report for D7063 (id=25623)

Could not rebase; Attempt merge onto aba5c807...

Updating aba5c80..18aceb1
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 264 ++++++++---------
 swh/core/db/db_utils.py                            | 329 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 swh/core/db/tests/test_cli.py                      | 123 +++++---
 swh/core/db/tests/test_db_utils.py                 |  73 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 11 files changed, 658 insertions(+), 239 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit 18aceb1ebdbd670d52efb0caf5d292a3fce6ba77
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 981ae1d754edeea5cda2ff5e5eac957d7c376dfb
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit f375b231aa73d934fb44c7d1071c20d4121a6186
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit c75374bafd77dfff82904985b935bbd035147cbc
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit 5c5b981020c4b3e38f89f112480b7297cceded5c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit d3b7b081fd3eda460d11d1d6d70f2b52fa3e0ef3
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit e4117fcf51a256c5b708f909c818f452c794c0f8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/276/ See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/276/console

fix

Build has FAILED

Patch application report for D7063 (id=25625)

Could not rebase; Attempt merge onto aba5c807...

Updating aba5c80..23eecb8
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 264 ++++++++---------
 swh/core/db/db_utils.py                            | 329 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 swh/core/db/tests/test_cli.py                      | 123 +++++---
 swh/core/db/tests/test_db_utils.py                 |  72 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 11 files changed, 657 insertions(+), 239 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit 23eecb8b0c95c57143abdeedc8fa7c12d5b715a8
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 981ae1d754edeea5cda2ff5e5eac957d7c376dfb
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit f375b231aa73d934fb44c7d1071c20d4121a6186
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit c75374bafd77dfff82904985b935bbd035147cbc
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit 5c5b981020c4b3e38f89f112480b7297cceded5c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit d3b7b081fd3eda460d11d1d6d70f2b52fa3e0ef3
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit e4117fcf51a256c5b708f909c818f452c794c0f8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/278/ See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/278/console

Build has FAILED

Patch application report for D7063 (id=25625)

Could not rebase; Attempt merge onto aba5c807...

Updating aba5c80..23eecb8
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 264 ++++++++---------
 swh/core/db/db_utils.py                            | 329 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 swh/core/db/tests/test_cli.py                      | 123 +++++---
 swh/core/db/tests/test_db_utils.py                 |  72 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 11 files changed, 657 insertions(+), 239 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit 23eecb8b0c95c57143abdeedc8fa7c12d5b715a8
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 981ae1d754edeea5cda2ff5e5eac957d7c376dfb
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit f375b231aa73d934fb44c7d1071c20d4121a6186
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit c75374bafd77dfff82904985b935bbd035147cbc
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit 5c5b981020c4b3e38f89f112480b7297cceded5c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit d3b7b081fd3eda460d11d1d6d70f2b52fa3e0ef3
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit e4117fcf51a256c5b708f909c818f452c794c0f8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/280/ See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/280/console

add forgotten data files

Build is green

Patch application report for D7063 (id=25628)

Could not rebase; Attempt merge onto aba5c807...

Updating aba5c80..471e411
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 264 ++++++++---------
 swh/core/db/db_utils.py                            | 329 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   5 +
 swh/core/db/tests/test_cli.py                      | 123 +++++---
 swh/core/db/tests/test_db_utils.py                 |  72 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 15 files changed, 675 insertions(+), 239 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit 471e41184ef3e8f983fa55043d8c3daebbf87e8c
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 981ae1d754edeea5cda2ff5e5eac957d7c376dfb
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit f375b231aa73d934fb44c7d1071c20d4121a6186
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit c75374bafd77dfff82904985b935bbd035147cbc
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit 5c5b981020c4b3e38f89f112480b7297cceded5c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit d3b7b081fd3eda460d11d1d6d70f2b52fa3e0ef3
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit e4117fcf51a256c5b708f909c818f452c794c0f8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/281/ for more details.

small cleanup in cli_new/50-data.sql

Build is green

Patch application report for D7063 (id=25630)

Could not rebase; Attempt merge onto aba5c807...

Updating aba5c80..3fbb230
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 264 ++++++++---------
 swh/core/db/db_utils.py                            | 329 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/test_cli.py                      | 123 +++++---
 swh/core/db/tests/test_db_utils.py                 |  72 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 15 files changed, 672 insertions(+), 239 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit 3fbb2305a22a6ef447b3634c09bf743eaba0a25e
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 981ae1d754edeea5cda2ff5e5eac957d7c376dfb
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit f375b231aa73d934fb44c7d1071c20d4121a6186
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit c75374bafd77dfff82904985b935bbd035147cbc
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit 5c5b981020c4b3e38f89f112480b7297cceded5c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit d3b7b081fd3eda460d11d1d6d70f2b52fa3e0ef3
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit e4117fcf51a256c5b708f909c818f452c794c0f8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/283/ for more details.

rebase

Build is green

Patch application report for D7063 (id=25635)

Could not rebase; Attempt merge onto aba5c807...

Updating aba5c80..1a4ab4e
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 264 ++++++++---------
 swh/core/db/db_utils.py                            | 326 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/test_cli.py                      | 123 ++++----
 swh/core/db/tests/test_db_utils.py                 |  72 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 15 files changed, 669 insertions(+), 239 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit 1a4ab4e856b2d5b7c97bc955c8d0373a95a0fd66
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 58e4f342eeeab1638e57d090d584435c91cf506d
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit 2f2b047ffd906acf0d64d8f0d94e192e49759480
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit 7774e8e504afe02b46dc075e3ce9cc82bcc59ab8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit 36bc18267c911abb85b6a1cfc3321942e0359369
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit de1c894e3fa88b3e1e2ba1e7c7dbfc77f7ea4b51
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit e4117fcf51a256c5b708f909c818f452c794c0f8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/288/ for more details.

rebase

Build is green

Patch application report for D7063 (id=25639)

Could not rebase; Attempt merge onto aba5c807...

Updating aba5c80..a8a8214
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 289 +++++++++---------
 swh/core/db/db_utils.py                            | 326 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/test_cli.py                      | 123 ++++----
 swh/core/db/tests/test_db_utils.py                 |  72 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 15 files changed, 684 insertions(+), 249 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit a8a821485532a09a4b2fecbb897d70aeb6d1cc1d
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit ba7318d74ef473a57c2791fad0ae32c55a74e6cf
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit 7036f2d8cf38f8d5b5cb9a2df02235012e12ddcd
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit d1ab8a3ab5cfef97918b5168c0fa15f5c26c0e18
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit b11933f997e32db1b59be095a4dfe5d0c3be912b
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit de1c894e3fa88b3e1e2ba1e7c7dbfc77f7ea4b51
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit e4117fcf51a256c5b708f909c818f452c794c0f8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/292/ for more details.

I'm not sure I like this change. What is the advantage of this over keeping an insert into dbversion ... statement in the SQL files, which would be used when the database is initialized?

rebase

Build is green

Patch application report for D7063 (id=25934)

Could not rebase; Attempt merge onto 5046a95f...

Updating 5046a95..20e63a8
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 289 +++++++++---------
 swh/core/db/db_utils.py                            | 326 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/test_cli.py                      | 123 ++++----
 swh/core/db/tests/test_db_utils.py                 |  72 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 swh/core/tests/test_utils.py                       | 193 ++++++------
 swh/core/utils.py                                  |  16 +-
 17 files changed, 793 insertions(+), 349 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit 20e63a8c5105972d402264beb8d0fb84704142cd
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit ec245207af61185298b087955bc609a0704a126c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit 93c9c701d28bf38b37bcf8ba5607fad3116c0087
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit 3139d3344e232db83edacc95884604e47f0461cb
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Feb 11 12:10:43 2022 +0100

    Fix a bug in numfile_sortkey and add tests for it

commit 68dc6811967ff0f6fb1296f9687d22d7f0fb684b
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Feb 11 11:59:56 2022 +0100

    Convert test_utils to pytest

commit 3e43730df6150168ca99cb083bd2e60feb4a2771
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit 459da96c1e6b6044deb874e0e668606c3da3e43c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit 932eed4863baedf011b4a59d0227331d37db1dfd
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/301/ for more details.

rebase