staging: Activate metadata loader for git loader
This will install the extra python3-swh.loader.metadata package for staging workers.
By default, this is off for production and on for staging.
When on, it's also installing the extra set of credentials in the loader_git's configuration to allow the use of such credentials when communicating with the forges.
Related to T4193
Test Plan
Expectedly:
- octo-diff on worker.production does nothing
- octo-diff on worker.staging wants to do as described.
$SWH_PUPPET_ENVIRONMENT_HOME/bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging worker0.internal.staging.swh.network
Found host worker0.internal.staging.swh.network
...
*** Running octocatalog-diff on host worker0.internal.staging.swh.network
I, [2022-04-26T16:22:23.543299 #881891] INFO -- : Catalogs compiled for worker0.internal.staging.swh.network
I, [2022-04-26T16:22:23.943674 #881891] INFO -- : Diffs computed for worker0.internal.staging.swh.network
diff origin/production/worker0.internal.staging.swh.network current/worker0.internal.staging.swh.network
*******************************************
File[/etc/softwareheritage/loader_git.yml] =>
parameters =>
content =>
@@ -28,3 +28,105 @@
- swh.loader.git.tasks.LoadDiskGitRepository
- swh.loader.git.tasks.UncompressAndLoadDiskGitRepository
+credentials:
+ github:
+ github:
+ - password: swh-deploy-worker-lister-config-credentials-github-github-0-password
+ username: swh::deploy::worker::lister::config::credentials/github/github/0/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github-1-password
+ username: swh::deploy::worker::lister::config::credentials/github/github/1/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github-2-password
+ username: swh::deploy::worker::lister::config::credentials/github/github/2/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github-3-password
+ username: swh::deploy::worker::lister::config::credentials/github/github/3/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github-4-password
+ username: swh::deploy::worker::lister::config::credentials/github/github/4/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github-5-password
+ username: swh::deploy::worker::lister::config::credentials/github/github/5/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github-6-password
+ username: swh::deploy::worker::lister::config::credentials/github/github/6/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github-7-password
+ username: swh::deploy::worker::lister::config::credentials/github/github/7/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github-8-password
+ username: swh::deploy::worker::lister::config::credentials/github/github/8/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github-9-password
+ username: swh::deploy::worker::lister::config::credentials/github/github/9/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github-10-password
+ username: swh::deploy::worker::lister::config::credentials/github/github/10/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github-11-password
+ username: swh::deploy::worker::lister::config::credentials/github/github/11/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github-12-password
+ username: swh::deploy::worker::lister::config::credentials/github/github/12/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github-13-password
+ username: swh::deploy::worker::lister::config::credentials/github/github/13/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github-14-password
+ username: swh::deploy::worker::lister::config::credentials/github/github/14/username
+ github.com:
+ - password: swh-deploy-worker-lister-config-credentials-github-github.com-0-password
+ username: swh::deploy::worker::lister::config::credentials/github/github.com/0/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github.com-1-password
+ username: swh::deploy::worker::lister::config::credentials/github/github.com/1/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github.com-2-password
+ username: swh::deploy::worker::lister::config::credentials/github/github.com/2/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github.com-3-password
+ username: swh::deploy::worker::lister::config::credentials/github/github.com/3/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github.com-4-password
+ username: swh::deploy::worker::lister::config::credentials/github/github.com/4/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github.com-5-password
+ username: swh::deploy::worker::lister::config::credentials/github/github.com/5/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github.com-6-password
+ username: swh::deploy::worker::lister::config::credentials/github/github.com/6/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github.com-7-password
+ username: swh::deploy::worker::lister::config::credentials/github/github.com/7/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github.com-8-password
+ username: swh::deploy::worker::lister::config::credentials/github/github.com/8/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github.com-9-password
+ username: swh::deploy::worker::lister::config::credentials/github/github.com/9/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github.com-10-password
+ username: swh::deploy::worker::lister::config::credentials/github/github.com/10/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github.com-11-password
+ username: swh::deploy::worker::lister::config::credentials/github/github.com/11/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github.com-12-password
+ username: swh::deploy::worker::lister::config::credentials/github/github.com/12/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github.com-13-password
+ username: swh::deploy::worker::lister::config::credentials/github/github.com/13/username
+ - password: swh-deploy-worker-lister-config-credentials-github-github.com-14-password
+ username: swh::deploy::worker::lister::config::credentials/github/github.com/14/username
+ gitlab:
+ gitlab:
+ - password: swh-deploy-worker-lister-config-credentials-gitlab-gitlab-0-password
+ username: swh::deploy::worker::lister::config::credentials/gitlab/gitlab/0/username
+ gitlab.com:
+ - password: swh-deploy-worker-lister-config-credentials-gitlab-gitlab.com-0-password
+ username: swh::deploy::worker::lister::config::credentials/gitlab/gitlab.com/0/username
+ phabricator:
+ blender:
+ - password: swh-deploy-worker-lister-config-credentials-phabricator-blender-0-password
+ username: swh::deploy::worker::lister::config::credentials/phabricator/blender/0/username
+ developer.blender.org:
+ - password: swh-deploy-worker-lister-config-credentials-phabricator-developer.blender.org-0-password
+ username: swh::deploy::worker::lister::config::credentials/phabricator/developer.blender.org/0/username
+ forge.softwareheritage.org:
+ - password: swh-deploy-worker-lister-config-credentials-phabricator-forge.softwareheritage.org-0-password
+ username: swh::deploy::worker::lister::config::credentials/phabricator/forge.softwareheritage.org/0/username
+ kde:
+ - password: swh-deploy-worker-lister-config-credentials-phabricator-kde-0-password
+ username: swh::deploy::worker::lister::config::credentials/phabricator/kde/0/username
+ llvm:
+ - password: swh-deploy-worker-lister-config-credentials-phabricator-llvm-0-password
+ username: swh::deploy::worker::lister::config::credentials/phabricator/llvm/0/username
+ phabricator.kde.org:
+ - password: swh-deploy-worker-lister-config-credentials-phabricator-phabricator.kde.org-0-password
+ username: swh::deploy::worker::lister::config::credentials/phabricator/phabricator.kde.org/0/username
+ phabricator.wikimedia.org:
+ - password: swh-deploy-worker-lister-config-credentials-phabricator-phabricator.wikimedia.org-0-password
+ username: swh::deploy::worker::lister::config::credentials/phabricator/phabricator.wikimedia.org/0/username
+ reviews.llvm.org:
+ - password: swh-deploy-worker-lister-config-credentials-phabricator-reviews.llvm.org-0-password
+ username: swh::deploy::worker::lister::config::credentials/phabricator/reviews.llvm.org/0/username
+ swh:
+ - password: swh-deploy-worker-lister-config-credentials-phabricator-swh-0-password
+ username: swh::deploy::worker::lister::config::credentials/phabricator/swh/0/username
+ wikimedia:
+ - password: swh-deploy-worker-lister-config-credentials-phabricator-wikimedia-0-password
+ username: swh::deploy::worker::lister::config::credentials/phabricator/wikimedia/0/username
_
*******************************************
+ Package[python3-swh.loader.metadata] =>
parameters =>
"ensure": "installed"
*******************************************
Profile::Swh::Deploy::Worker::Instance[checker_deposit] =>
parameters =>
extra_config =>
+ {}
*******************************************
Profile::Swh::Deploy::Worker::Instance[indexer_content_mimetype] =>
parameters =>
extra_config =>
+ {}
*******************************************
Profile::Swh::Deploy::Worker::Instance[indexer_fossology_license] =>
parameters =>
extra_config =>
+ {}
*******************************************
Profile::Swh::Deploy::Worker::Instance[indexer_origin_intrinsic_metadata] =>
parameters =>
extra_config =>
+ {}
*******************************************
Profile::Swh::Deploy::Worker::Instance[lister] =>
parameters =>
extra_config =>
+ {}
*******************************************
Profile::Swh::Deploy::Worker::Instance[loader_archive] =>
parameters =>
extra_config =>
+ {}
*******************************************
Profile::Swh::Deploy::Worker::Instance[loader_bzr] =>
parameters =>
extra_config =>
+ {}
*******************************************
Profile::Swh::Deploy::Worker::Instance[loader_cran] =>
parameters =>
extra_config =>
+ {}
*******************************************
Profile::Swh::Deploy::Worker::Instance[loader_cvs] =>
parameters =>
extra_config =>
+ {}
*******************************************
Profile::Swh::Deploy::Worker::Instance[loader_debian] =>
parameters =>
extra_config =>
+ {}
*******************************************
Profile::Swh::Deploy::Worker::Instance[loader_deposit] =>
parameters =>
extra_config =>
+ {}
*******************************************
Profile::Swh::Deploy::Worker::Instance[loader_git] =>
parameters =>
extra_config =>
+ {"credentials"=>{"github"=>{"github"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-github-github-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/0/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-1-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/1/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-2-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/2/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-3-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/3/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-4-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/4/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-5-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/5/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-6-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/6/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-7-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/7/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-8-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/8/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-9-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/9/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-10-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/10/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-11-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/11/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-12-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/12/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-13-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/13/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github-14-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github/14/username"}], "github.com"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/0/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-1-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/1/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-2-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/2/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-3-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/3/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-4-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/4/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-5-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/5/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-6-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/6/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-7-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/7/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-8-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/8/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-9-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/9/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-10-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/10/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-11-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/11/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-12-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/12/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-13-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/13/username"}, {"password"=>"swh-deploy-worker-lister-config-credentials-github-github.com-14-password", "username"=>"swh::deploy::worker::lister::config::credentials/github/github.com/14/username"}]}, "gitlab"=>{"gitlab"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-gitlab-gitlab-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/gitlab/gitlab/0/username"}], "gitlab.com"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-gitlab-gitlab.com-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/gitlab/gitlab.com/0/username"}]}, "phabricator"=>{"blender"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-blender-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/blender/0/username"}], "developer.blender.org"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-developer.blender.org-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/developer.blender.org/0/username"}], "forge.softwareheritage.org"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-forge.softwareheritage.org-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/forge.softwareheritage.org/0/username"}], "kde"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-kde-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/kde/0/username"}], "llvm"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-llvm-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/llvm/0/username"}], "phabricator.kde.org"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-phabricator.kde.org-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/phabricator.kde.org/0/username"}], "phabricator.wikimedia.org"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-phabricator.wikimedia.org-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/phabricator.wikimedia.org/0/username"}], "reviews.llvm.org"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-reviews.llvm.org-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/reviews.llvm.org/0/username"}], "swh"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-swh-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/swh/0/username"}], "wikimedia"=>[{"password"=>"swh-deploy-worker-lister-config-credentials-phabricator-wikimedia-0-password", "username"=>"swh::deploy::worker::lister::config::credentials/phabricator/wikimedia/0/username"}]}}}
*******************************************
Profile::Swh::Deploy::Worker::Instance[loader_high_priority] =>
parameters =>
extra_config =>
+ {}
*******************************************
Profile::Swh::Deploy::Worker::Instance[loader_maven] =>
parameters =>
extra_config =>
+ {}
*******************************************
Profile::Swh::Deploy::Worker::Instance[loader_mercurial] =>
parameters =>
extra_config =>
+ {}
*******************************************
Profile::Swh::Deploy::Worker::Instance[loader_nixguix] =>
parameters =>
extra_config =>
+ {}
*******************************************
Profile::Swh::Deploy::Worker::Instance[loader_npm] =>
parameters =>
extra_config =>
+ {}
*******************************************
Profile::Swh::Deploy::Worker::Instance[loader_opam] =>
parameters =>
extra_config =>
+ {}
*******************************************
Profile::Swh::Deploy::Worker::Instance[loader_pypi] =>
parameters =>
extra_config =>
+ {}
*******************************************
Profile::Swh::Deploy::Worker::Instance[loader_svn] =>
parameters =>
extra_config =>
+ {}
*******************************************
Profile::Swh::Deploy::Worker::Instance[vault_cooker] =>
parameters =>
extra_config =>
+ {}
*******************************************
*** End octocatalog-diff on worker0.internal.staging.swh.networkVagrant checks are also ok:
$ vagrant up staging-worker0
...
root@worker0:~# grep -c credentials /etc/softwareheritage/loader_git.yml
85
root@worker0:~# dpkg -l python3-swh.loader.metadata
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===========================-====================-============-=============================================
ii python3-swh.loader.metadata 0.0.1-1~swh2~bpo10+1 all Software Heritage Extrinsic Metadata FetchersMigrated from D7687 (view on Phabricator)