config: Deprecate SWHConfig in favor of load_from_envvar function
That new function declares exactly what happens today during our configuration loading (for loader/lister/indexer/... and rpc services in general not for the CLIs which will be dealt with later).
Loads and parses the yaml configuration file out of the SWH_CONFIG_FILENAME
environment variable. Allowing eventual dict enrichment with default
configuration if provided.
Everything docker/staging/production related is setting the SWH_CONFIG_FILENAME in the environment [1-5] (including the tests now as well).
This currently impacts (which I will deal with soon):
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/indexer.py:from swh.core.config import SWHConfig
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/indexer.py:class BaseIndexer(SWHConfig, metaclass=abc.ABCMeta):
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/rehash.py:from swh.core.config import SWHConfig
$SWH_ENVIRONMENT_HOME/swh-indexer/swh/indexer/rehash.py:class RecomputeChecksums(SWHConfig):
$SWH_ENVIRONMENT_HOME/swh-lister/swh/lister/core/lister_base.py:class ListerBase(abc.ABC, config.SWHConfig):
$SWH_ENVIRONMENT_HOME/swh-loader-core/swh/loader/core/loader.py:class BaseLoader(config.SWHConfig, metaclass=ABCMeta):
$SWH_ENVIRONMENT_HOME/swh-loader-core/swh/loader/package/loader.py:from swh.core.config import SWHConfig
$SWH_ENVIRONMENT_HOME/swh-loader-core/swh/loader/package/loader.py: self.config = SWHConfig.parse_config_file()
- [1] prod/staging: https://forge.softwareheritage.org/source/puppet-swh-site/browse/production/?grep=SWH_CONFIG_FILENAME
- [2] docker: https://forge.softwareheritage.org/source/swh-environment/browse/master/?grep=SWH_CONFIG_FILENAME
As sample:
- [3] loader/lister: https://forge.softwareheritage.org/source/puppet-swh-site/browse/production/site-modules/profile/templates/swh/deploy/worker/swh-worker@.service.erb$10
- [4] scheduler: https://forge.softwareheritage.org/source/swh-environment/browse/master/docker/docker-compose.yml$109
- [5] api/rpc servers: https://forge.softwareheritage.org/source/puppet-swh-site/browse/production/site-modules/profile/manifests/swh/deploy/rpc_server.pp$104
Related to #1532 (closed) Related to #1410
Test Plan
tox
Migrated from D3965 (view on Phabricator)
Merge request reports
Activity
Build is green
Patch application report for D3965 (id=13957)
Could not rebase; Attempt merge onto 60c535dd...
Updating 60c535d..5c05fa7 Fast-forward swh/core/config.py | 135 ++++++++++++++---------------------------- swh/core/tests/test_config.py | 97 +++++++++++++++++++++++------- 2 files changed, 122 insertions(+), 110 deletions(-)Changes applied before test
commit 5c05fa7170486abeb1f9e23a4f076e34db75a98c Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Sep 16 15:23:16 2020 +0200 swh.core.config: Simplify SWHConfig mixin to the actual use Related to #1532 commit 66ac6ed3aa31a63da3226f2857755f7398721d31 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Sep 16 14:32:19 2020 +0200 swh.core.config: Drop support for ini configuration file This is no longer used. This also adds some types on impacted changed functions. Related to #1532See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/45/ for more details.
-
Build is green
Patch application report for D3965 (id=13958)
Could not rebase; Attempt merge onto 60c535dd...
Updating 60c535d..29414b5 Fast-forward swh/core/config.py | 119 +++++++++++++----------------------------- swh/core/tests/test_config.py | 97 ++++++++++++++++++++++++++-------- 2 files changed, 113 insertions(+), 103 deletions(-)Changes applied before test
commit 29414b50cc6f009f7413792d0658c47eaca04a1e Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Sep 16 15:25:52 2020 +0200 swh.core.config: Simplify SWHConfig mixin to the actual use Related to #1532 commit 66ac6ed3aa31a63da3226f2857755f7398721d31 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Sep 16 14:32:19 2020 +0200 swh.core.config: Drop support for ini configuration file This is no longer used. This also adds some types on impacted changed functions. Related to #1532See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/46/ for more details.
-
Build is green
Patch application report for D3965 (id=13960)
Could not rebase; Attempt merge onto 60c535dd...
Updating 60c535d..7e3ed44 Fast-forward swh/core/config.py | 122 ++++++++++++++---------------------------- swh/core/tests/test_config.py | 97 +++++++++++++++++++++++++-------- 2 files changed, 116 insertions(+), 103 deletions(-)Changes applied before test
commit 7e3ed44614cf5978fc57f3b5cd6894ab6d3212e7 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Sep 16 15:25:52 2020 +0200 swh.core.config: Simplify SWHConfig mixin to the actual use Related to #1532 commit 66ac6ed3aa31a63da3226f2857755f7398721d31 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Sep 16 14:32:19 2020 +0200 swh.core.config: Drop support for ini configuration file This is no longer used. This also adds some types on impacted changed functions. Related to #1532See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/48/ for more details.
-
Build is green
Patch application report for D3965 (id=13963)
Could not rebase; Attempt merge onto 60c535dd...
Updating 60c535d..e7b3048 Fast-forward swh/core/config.py | 122 ++++++++++++++---------------------------- swh/core/tests/test_config.py | 97 +++++++++++++++++++++++++-------- 2 files changed, 116 insertions(+), 103 deletions(-)Changes applied before test
commit e7b3048844622fe74b83c54682aaa464e34f09b1 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Sep 16 15:25:52 2020 +0200 swh.core.config: Simplify SWHConfig mixin to the actual use Related to #1532 commit 66ac6ed3aa31a63da3226f2857755f7398721d31 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Sep 16 14:32:19 2020 +0200 swh.core.config: Drop support for ini configuration file This is no longer used. This also adds some types on impacted changed functions. Related to #1532See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/51/ for more details.
mentioned in merge request swh-loader-core!162 (closed)
mentioned in merge request swh-loader-svn!47 (closed)
mentioned in merge request swh-loader-mercurial!32 (closed)
mentioned in merge request swh-loader-git!51 (closed)
mentioned in merge request swh-indexer!252 (closed)
mentioned in merge request swh-lister!160 (closed)
mentioned in merge request swh-loader-core!335 (closed)
mentioned in issue #1410
-
Build is green
Patch application report for D3965 (id=13967)
Rebasing onto 66ac6ed3...
Current branch diff-target is up to date.Changes applied before test
commit 2934097d81da51a6f9b9423bf710c9a9f3116ae2 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Sep 16 15:25:52 2020 +0200 swh.core.config: Simplify SWHConfig mixin to its actual use Related to #1532See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/52/ for more details.
