Skip to content
Snippets Groups Projects

Add a --check-config option for cli commands

Closed Add a --check-config option for cli commands
generated-differential-D3834-source into generated-differential-D3834-target
Closed David Douard requested to merge generated-differential-D3834-source into generated-differential-D3834-target

this allows to specify on command line wether to check the configuration and read or write access to the storage at startup time (especially for rpc-server and replay commands.)

Warning: this option defaults to "read" for the backfill command and "write" for rpc-serve and replay commands, so now the creation of the Storage instance used in cli commands will be checked.

Also remove the deprecated config-path option from swh storage rpc-serve command and include the validation of the presence of a "storage" config section in the main storage click.group, where the config-file is actually parsed.

Depends on !550 (closed).

Migrated from D3834 (view on Phabricator)

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • David Douard
    David Douard @douardda ·
    Author Maintainer

    rebase + add closes #2525 (closed) in the ci msg

  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build is green

    Patch application report for D3834 (id=13496)

    Could not rebase; Attempt merge onto f570f933...

    Merge made by the 'recursive' strategy.
 swh/storage/__init__.py              |  12 ++-
 swh/storage/cli.py                   |  59 +++++++----
 swh/storage/postgresql/db.py         |  12 +++
 swh/storage/postgresql/storage.py    |   3 +
 swh/storage/pytest_plugin.py         |   3 +-
 swh/storage/tests/test_init.py       | 189 +++++++++++++++++++++++++++++------
 swh/storage/tests/test_postgresql.py |  19 ++++
 7 files changed, 243 insertions(+), 54 deletions(-)
    Changes applied before test
    commit 0fc87dcdc697634d5357d03a997436f1aa7d95ce
Merge: f570f933 c8fe81c3
Author: Jenkins user <jenkins@localhost>
Date:   Mon Aug 24 15:12:14 2020 +0000

    Merge branch 'diff-target' into HEAD

commit c8fe81c350256721162ed56573dbf1b8ee666be5
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Aug 24 16:14:47 2020 +0200

    Add a --check-config option for cli commands
    
    this allows to specify on command line wether to check the configuration
    and read or write access to the storage at startup time (especially for
    `rpc-server` and `replay` commands.)
    
    Warning: this option defaults to "read" for the backfill command and
    "write" for rpc-serve and replay commands, so now the creation of
    the Storage instance used in cli commands *will be checked*.

commit b42f3bac0afd08b5bbfd2605b6de077bbed71d6e
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Aug 24 16:10:30 2020 +0200

    Remove the deprecated config-path option from `swh storage rpc-serve` command
    
    include the validation of the presence of a "storage" config section
    in the main `storage` click.group, where the config-file is actually
    parsed.

commit ee1ed57625cc6db56efaff1e1051a9e412ccf135
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 17:39:59 2020 +0200

    Add support for a new "check_config" config option in get_storage()
    
    if "check_config" is present in get_storage()'s kwargs, call the
    storage.check_config() method and raise an exception is the result
    fails. The storage.check_config() method is called with the content
    of the "check_config" config value (so expected value for this
    currently is `check_config={"check_write": <bool>}`).

commit f7434a8cb9c1ecd72bc716ed6dd3cf5f05ad6893
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 15:43:35 2020 +0200

    Check for db version mismatch in PgStorage.check_config()

commit d0052506f4deb165dd06d28f415787e2cd31c8ee
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 12:00:37 2020 +0200

    Add a check_dbversion() method to the Db class
    
    This method compares the version stored in the database in the `dbversion`
    table with the currently declared version.
    
    This current version is declared as a simple `current_version` attribute
    on the Db class. **It must be updated jointly in 30-swh-schema.sql** when
    modifying the db schema. Note that if this is forgotten, the added test
    `test_dbversion` should fail.
    
    Related to #2525.

commit 346ad9c4197e57d377fed351b2353e14af486995
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 11:55:56 2020 +0200

    Fix pytest_plugin's database janitor: do not truncate the dbversion table

    See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/858/ for more details.

  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build is green

    Patch application report for D3834 (id=13501)

    Rebasing onto cc33dd35...

    Current branch diff-target is up to date.
    Changes applied before test
    commit bc4f8ef0f647969d51481d77a8bcc8bbce70b416
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Aug 24 16:14:47 2020 +0200

    Add a --check-config option for cli commands
    
    this allows to specify on command line wether to check the configuration
    and read or write access to the storage at startup time (especially for
    `rpc-server` and `replay` commands.)
    
    Warning: this option defaults to "read" for the backfill command and
    "write" for rpc-serve and replay commands, so now the creation of
    the Storage instance used in cli commands *will be checked*.
    
    Closes #2525.

commit f0fc833e8ef2ab12fec0340b2f5e4198feb1492b
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Aug 24 16:10:30 2020 +0200

    Remove the deprecated config-path option from `swh storage rpc-serve` command
    
    include the validation of the presence of a "storage" config section
    in the main `storage` click.group, where the config-file is actually
    parsed.

    See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/862/ for more details.

  • Antoine R. Dumont
    Antoine R. Dumont @ardumont ·
    Maintainer

    missing types on helper function but looks good

    thanks.

  • Antoine R. Dumont
    Antoine R. Dumont @ardumont ·
    Maintainer

    Merge request was accepted

  • Antoine R. Dumont approved this merge request

    approved this merge request

  • David Douard
    David Douard @douardda ·
    Author Maintainer

    type annotate the new ensure_check_config helper function

    and add a docstring.

  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build is green

    Patch application report for D3834 (id=13510)

    Rebasing onto f3870dcd...

    First, rewinding head to replay your work on top of it...
Applying: Remove the deprecated config-path option from `swh storage rpc-serve` command
Applying: Add a --check-config option for cli commands
    Changes applied before test
    commit 47dae5cf9eef08816fa64c743bcfb322d36a9b6d
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Aug 24 16:14:47 2020 +0200

    Add a --check-config option for cli commands
    
    this allows to specify on command line wether to check the configuration
    and read or write access to the storage at startup time (especially for
    `rpc-server` and `replay` commands.)
    
    Warning: this option defaults to "read" for the backfill command and
    "write" for rpc-serve and replay commands, so now the creation of
    the Storage instance used in cli commands *will be checked*.
    
    Closes #2525.

commit d3f8e24e535ded09a451fb661acf637f161bc65b
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Aug 24 16:10:30 2020 +0200

    Remove the deprecated config-path option from `swh storage rpc-serve` command
    
    include the validation of the presence of a "storage" config section
    in the main `storage` click.group, where the config-file is actually
    parsed.

    See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/866/ for more details.

  • David Douard
    David Douard @douardda ·
    Author Maintainer

    rebase

  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build is green

    Patch application report for D3834 (id=13515)

    Rebasing onto e8b1b216...

    Current branch diff-target is up to date.
    Changes applied before test
    commit 4532a4dc64e7a1967e501303d00e6dfbce739ca7
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Aug 24 16:14:47 2020 +0200

    Add a --check-config option for cli commands
    
    this allows to specify on command line wether to check the configuration
    and read or write access to the storage at startup time (especially for
    `rpc-server` and `replay` commands.)
    
    Warning: this option defaults to "read" for the backfill command and
    "write" for rpc-serve and replay commands, so now the creation of
    the Storage instance used in cli commands *will be checked*.
    
    Closes #2525.

commit 2a35c0bb231c177824f93c56c9d11bc24215a6ce
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Aug 24 16:10:30 2020 +0200

    Remove the deprecated config-path option from `swh storage rpc-serve` command
    
    include the validation of the presence of a "storage" config section
    in the main `storage` click.group, where the config-file is actually
    parsed.

    See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/868/ for more details.

  • David Douard
    David Douard @douardda ·
    Author Maintainer

    Merge request was merged

  • David Douard closed

    closed

Please register or sign in to reply