core.loader: Merge Loader into BaseLoader
Closed
requested to merge generated-differential-D5088-source into generated-differential-D5088-target
This simplifies the docstring to a more relevant docstring.
This drops the spurious {*args, **kwargs} no longer used in the core loaders.
(it's required because PackageLoader's load signature is without (kw)args)
Depends on !341 (closed)
Migrated from D5088 (view on Phabricator)
Merge request reports
Activity
Build has FAILED
Patch application report for D5088 (id=18163)
Could not rebase; Attempt merge onto 7ea69e2b...
Updating 7ea69e2..593641a Fast-forward swh/loader/cli.py | 33 ++++- swh/loader/core/loader.py | 133 ++++++++++--------- swh/loader/core/tests/test_loader.py | 47 ++++--- swh/loader/package/archive/loader.py | 7 +- swh/loader/package/archive/tasks.py | 5 +- swh/loader/package/archive/tests/test_archive.py | 74 ++++++----- swh/loader/package/archive/tests/test_tasks.py | 12 +- swh/loader/package/cran/loader.py | 13 +- swh/loader/package/cran/tasks.py | 4 +- swh/loader/package/cran/tests/test_cran.py | 42 +++--- swh/loader/package/cran/tests/test_tasks.py | 17 ++- swh/loader/package/debian/loader.py | 14 +- swh/loader/package/debian/tasks.py | 3 +- swh/loader/package/debian/tests/test_debian.py | 65 +++++---- swh/loader/package/debian/tests/test_tasks.py | 12 +- swh/loader/package/deposit/loader.py | 35 ++++- swh/loader/package/deposit/tasks.py | 4 +- swh/loader/package/deposit/tests/conftest.py | 9 +- swh/loader/package/deposit/tests/test_deposit.py | 45 +++++-- swh/loader/package/deposit/tests/test_tasks.py | 15 ++- swh/loader/package/loader.py | 39 ++---- swh/loader/package/nixguix/loader.py | 16 ++- swh/loader/package/nixguix/tasks.py | 4 +- swh/loader/package/nixguix/tests/test_nixguix.py | 114 ++++++++-------- swh/loader/package/nixguix/tests/test_tasks.py | 20 +-- swh/loader/package/npm/loader.py | 10 +- swh/loader/package/npm/tasks.py | 4 +- swh/loader/package/npm/tests/test_npm.py | 89 ++++++------- swh/loader/package/npm/tests/test_tasks.py | 10 +- swh/loader/package/pypi/loader.py | 10 +- swh/loader/package/pypi/tasks.py | 4 +- swh/loader/package/pypi/tests/test_pypi.py | 160 ++++++++++------------- swh/loader/package/pypi/tests/test_tasks.py | 10 +- swh/loader/package/tests/test_loader.py | 8 +- swh/loader/package/tests/test_loader_metadata.py | 47 ++----- swh/loader/pytest_plugin.py | 28 ++-- swh/loader/tests/conftest.py | 4 +- swh/loader/tests/test_cli.py | 65 +++++---- swh/loader/tests/test_init.py | 11 +- 39 files changed, 674 insertions(+), 568 deletions(-)Changes applied before test
commit 593641af0eaab910cb9a82c7180b8d08dbb1eb45 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Tue Feb 16 18:48:42 2021 +0100 core.loader: Merge Loader into BaseLoader This also simplifies the docstring to a more relevant docstring. This also drops the spurious {*args, **kwargs} no longer used in the core loaders. commit 0c36690f10b26a9916150a5606d3de2ad2d97cb6 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Mon Feb 8 09:04:17 2021 +0100 Unify loader instantiation This unifies and centralizes the instantiation the same way the lister does. This introduces a new base class `swh.loader.pattern.Loader` for all loaders whose only concern for now is to instantiate loaders from either a configuration dict or a configuration file. This simplify instantiation in celery task code and avoids duplicating the configuration load in each loader constructor. The end goal is to simplify the future refactoring on configuration. Now, we will only have to adapt the Loader class when we finally start simplifying uniformly the configuration.Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/376/ See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/376/console
-
Build is green
Patch application report for D5088 (id=18164)
Could not rebase; Attempt merge onto 7ea69e2b...
Updating 7ea69e2..3bb2cfd Fast-forward swh/loader/cli.py | 33 ++++- swh/loader/core/loader.py | 138 ++++++++++--------- swh/loader/core/tests/test_loader.py | 47 ++++--- swh/loader/package/archive/loader.py | 7 +- swh/loader/package/archive/tasks.py | 5 +- swh/loader/package/archive/tests/test_archive.py | 74 ++++++----- swh/loader/package/archive/tests/test_tasks.py | 12 +- swh/loader/package/cran/loader.py | 13 +- swh/loader/package/cran/tasks.py | 4 +- swh/loader/package/cran/tests/test_cran.py | 42 +++--- swh/loader/package/cran/tests/test_tasks.py | 17 ++- swh/loader/package/debian/loader.py | 14 +- swh/loader/package/debian/tasks.py | 3 +- swh/loader/package/debian/tests/test_debian.py | 65 +++++---- swh/loader/package/debian/tests/test_tasks.py | 12 +- swh/loader/package/deposit/loader.py | 35 ++++- swh/loader/package/deposit/tasks.py | 4 +- swh/loader/package/deposit/tests/conftest.py | 9 +- swh/loader/package/deposit/tests/test_deposit.py | 45 +++++-- swh/loader/package/deposit/tests/test_tasks.py | 15 ++- swh/loader/package/loader.py | 39 ++---- swh/loader/package/nixguix/loader.py | 16 ++- swh/loader/package/nixguix/tasks.py | 4 +- swh/loader/package/nixguix/tests/test_nixguix.py | 114 ++++++++-------- swh/loader/package/nixguix/tests/test_tasks.py | 20 +-- swh/loader/package/npm/loader.py | 10 +- swh/loader/package/npm/tasks.py | 4 +- swh/loader/package/npm/tests/test_npm.py | 89 ++++++------- swh/loader/package/npm/tests/test_tasks.py | 10 +- swh/loader/package/pypi/loader.py | 10 +- swh/loader/package/pypi/tasks.py | 4 +- swh/loader/package/pypi/tests/test_pypi.py | 160 ++++++++++------------- swh/loader/package/pypi/tests/test_tasks.py | 10 +- swh/loader/package/tests/test_loader.py | 8 +- swh/loader/package/tests/test_loader_metadata.py | 47 ++----- swh/loader/pytest_plugin.py | 28 ++-- swh/loader/tests/conftest.py | 4 +- swh/loader/tests/test_cli.py | 65 +++++---- swh/loader/tests/test_init.py | 11 +- 39 files changed, 679 insertions(+), 568 deletions(-)Changes applied before test
commit 3bb2cfda4ee5ad78a928159f0dc994d9307395fe Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Tue Feb 16 18:48:42 2021 +0100 core.loader: Merge Loader into BaseLoader This also simplifies the docstring to a more relevant docstring. This also drops the spurious {*args, **kwargs} no longer used in the core loaders. commit 0c36690f10b26a9916150a5606d3de2ad2d97cb6 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Mon Feb 8 09:04:17 2021 +0100 Unify loader instantiation This unifies and centralizes the instantiation the same way the lister does. This introduces a new base class `swh.loader.pattern.Loader` for all loaders whose only concern for now is to instantiate loaders from either a configuration dict or a configuration file. This simplify instantiation in celery task code and avoids duplicating the configuration load in each loader constructor. The end goal is to simplify the future refactoring on configuration. Now, we will only have to adapt the Loader class when we finally start simplifying uniformly the configuration.See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/377/ for more details.
-
Build is green
Patch application report for D5088 (id=18168)
Could not rebase; Attempt merge onto 7ea69e2b...
Updating 7ea69e2..7c776a4 Fast-forward swh/loader/cli.py | 33 ++++- swh/loader/core/loader.py | 142 +++++++++++--------- swh/loader/core/tests/test_loader.py | 47 ++++--- swh/loader/package/archive/loader.py | 7 +- swh/loader/package/archive/tasks.py | 5 +- swh/loader/package/archive/tests/test_archive.py | 74 ++++++----- swh/loader/package/archive/tests/test_tasks.py | 12 +- swh/loader/package/cran/loader.py | 13 +- swh/loader/package/cran/tasks.py | 4 +- swh/loader/package/cran/tests/test_cran.py | 42 +++--- swh/loader/package/cran/tests/test_tasks.py | 17 ++- swh/loader/package/debian/loader.py | 14 +- swh/loader/package/debian/tasks.py | 3 +- swh/loader/package/debian/tests/test_debian.py | 65 +++++---- swh/loader/package/debian/tests/test_tasks.py | 12 +- swh/loader/package/deposit/loader.py | 35 ++++- swh/loader/package/deposit/tasks.py | 4 +- swh/loader/package/deposit/tests/conftest.py | 9 +- swh/loader/package/deposit/tests/test_deposit.py | 45 +++++-- swh/loader/package/deposit/tests/test_tasks.py | 15 ++- swh/loader/package/loader.py | 39 ++---- swh/loader/package/nixguix/loader.py | 16 ++- swh/loader/package/nixguix/tasks.py | 4 +- swh/loader/package/nixguix/tests/test_nixguix.py | 114 ++++++++-------- swh/loader/package/nixguix/tests/test_tasks.py | 20 +-- swh/loader/package/npm/loader.py | 10 +- swh/loader/package/npm/tasks.py | 4 +- swh/loader/package/npm/tests/test_npm.py | 89 ++++++------- swh/loader/package/npm/tests/test_tasks.py | 10 +- swh/loader/package/pypi/loader.py | 10 +- swh/loader/package/pypi/tasks.py | 4 +- swh/loader/package/pypi/tests/test_pypi.py | 160 ++++++++++------------- swh/loader/package/pypi/tests/test_tasks.py | 10 +- swh/loader/package/tests/test_loader.py | 8 +- swh/loader/package/tests/test_loader_metadata.py | 47 ++----- swh/loader/pytest_plugin.py | 28 ++-- swh/loader/tests/conftest.py | 4 +- swh/loader/tests/test_cli.py | 65 +++++---- swh/loader/tests/test_init.py | 11 +- 39 files changed, 682 insertions(+), 569 deletions(-)Changes applied before test
commit 7c776a487c648d8e0a804783408f6d954b4f84ef Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Tue Feb 16 18:48:42 2021 +0100 core.loader: Merge Loader into BaseLoader This also simplifies the docstring to a more relevant docstring. This also drops the spurious {*args, **kwargs} no longer used in the core loaders. commit 7116bb75897ae878f68679e86aaa5aec6cc507be Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Mon Feb 8 09:04:17 2021 +0100 Unify loader instantiation This unifies and centralizes the instantiation the same way the lister does. This introduces a new base class swh.loader.core.loader.Loader for all loaders whose only concern for now is to instantiate loaders from either a configuration dict or a configuration file. This simplifies instantiation in celery task code and avoids duplicating the configuration load in each loader constructor. The end goal is to simplify the future refactoring on configuration. With the following, we will only have to adapt the Loader class when we start simplifying uniformly the configuration. Also note that I mostly reused the equivalent `swh.lister.pattern.Lister.from_config*`. I did not refactor the common behavior (to avoid throwing another dependency in the mix). That could always be refactored later. (inspired by both the work on listers and the configuration system work) Related to swh/devel/swh-core#1410See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/379/ for more details.
mentioned in merge request !341 (closed)
mentioned in merge request swh-loader-svn!51 (closed)
mentioned in merge request swh-loader-git!115 (closed)
mentioned in merge request swh-loader-mercurial!47 (closed)
We need to make sure that other loaders really don't use the *args, **kwargs, but yeah, that looks fine.
I checked git and svn so far. I'll check mercurial prior to land it.
Did you run mypy on other swh-loader-* modules against this change?
nope, not yet. Just mostly read the code.
Please register or sign in to reply