opam: Fix 'opam init' error when relisting an opam instance

mentioned in merge request swh-loader-core!483 (closed)

I'm a bit confused, shouldn't the default be set to the main (presumably, first) opam repository to be added, rather than flipped every time we add another repo?

Well, the error appeared since my recent changes since CalledProcessError is now raised. Passing the --set-default parameter ensures the command succeeds. Nevertheless, that error does not seem critical as I can successfully call opam list --repos <instance> without passing --set-default to opam init.

I did more analysis on the use of the --set-default flag and turned out its use is mandatory in our case to ensure all packages metadata from all opam instances can be queried with the opam show command.

I also noticed that the systemd service used to refresh opam packages database on production workers is using the --all-switches flag when adding a new opam instance so I also tested its use.

Commands below are executed on a production worker.

First we init an opam root directory with the opam.ocaml.org instance.

anlambert@worker08:~$ opam --version
2.0.3

anlambert@worker08:~$ opam init --reinit --bare --no-setup --root ./opam_root opam.ocaml.org https://opam.ocaml.org
[NOTE] Will configure from built-in defaults.
Checking for available remotes: rsync and local, git, mercurial, darcs. Perfect!

<><> Fetching repository information ><><><><><><><><><><><><><><><><><><><><><>
[opam.ocaml.org] Initialised

We can query package metadata from the opam.ocaml.org instance.

anlambert@worker08:~$ opam show agrid --root ./opam_root/

<><> agrid: information on all versions <><><><><><><><><><><><><><><><><><><><>
name         agrid
all-versions 0.1

<><> Version-specific details <><><><><><><><><><><><><><><><><><><><><><><><><>
version       0.1
repository    opam.ocaml.org
url.src:      "https://github.com/OCamlPro/agrid/archive/0.1.tar.gz"
url.checksum: "sha256=ea82546711a6abdd4edf8bc3052041498cae9c2e5a9e147e29820da4eac4beb4"
              "sha512=f53b2c095e3607e53f92d4e7e13848e9e34bd866837335e7d9341dbb468ac46ffbcd2002d1bf1105e2f6060f57871aef7ce8e65594855447fafb72ad32b076b7"
homepage:     "https://github.com/ocamlpro/agrid"
bug-reports:  "https://github.com/ocamlpro/agrid/issues"
dev-repo:     "git+https://github.com/ocamlpro/agrid.git"
authors:      "OCamlPro <contact@ocamlpro.com>"
maintainer:   "OCamlPro <contact@ocamlpro.com>"
license:      "ISC"
depends:      "ocaml" {>= "4.05"}
              "dune" {>= "2.7"}
              "flex-array" {>= "1.2"}
              "bisect_ppx" {with-test & >= "2.6" & dev}
              "odoc" {with-doc}
synopsis      Adjustable grid (two dimensional array) library
description   Adjustable grids are two dimensional arrays whose width/height can be changed by adding or removing row/column at either end (one at a time).

Let's try to add the coq instance without any flag, opam repository gets registered but metadata of its packages cannot be displayed.

anlambert@worker08:~$ opam repository add --root ./opam_root coq.inria.fr https://coq.inria.fr/opam/released
[WARNING] No switch is currently set, perhaps you meant '--set-default'?
[coq.inria.fr] Initialised
[ERROR] No switch is currently set. Please use 'opam switch' to set or install a switch

anlambert@worker08:~$ opam repository --root ./opam_root/ --all
# Repository   # Url                              # Switches(rank)
coq.inria.fr   https://coq.inria.fr/opam/released
opam.ocaml.org https://opam.ocaml.org             <default>(1)

anlambert@worker08:~$ opam show coq-float --root ./opam_root/
[ERROR] No package matching coq-float found

Let's try the same with the --all-switches flag, same result.

anlambert@worker08:~$ opam repository add --root ./opam_root --all-switches coq.inria.fr https://coq.inria.fr/opam/released
[coq.inria.fr] no changes from https://coq.inria.fr/opam/released

anlambert@worker08:~$ opam repository --root ./opam_root/ --all
# Repository   # Url                              # Switches(rank)
coq.inria.fr   https://coq.inria.fr/opam/released
opam.ocaml.org https://opam.ocaml.org             <default>(1)

anlambert@worker08:~$ opam show coq-float --root ./opam_root/
[ERROR] No package matching coq-float found

Finally, let's use the --set-default flag and now we are good.

anlambert@worker08:~$ opam repository add --root ./opam_root --set-default coq.inria.fr https://coq.inria.fr/opam/released
[coq.inria.fr] no changes from https://coq.inria.fr/opam/released

anlambert@worker08:~$ opam repository --root ./opam_root/ --all
# Repository   # Url                              # Switches(rank)
coq.inria.fr   https://coq.inria.fr/opam/released <default>(1)
opam.ocaml.org https://opam.ocaml.org             <default>(2)

anlambert@worker08:~$ opam show coq-float --root ./opam_root/

<><> coq-float: information on all versions <><><><><><><><><><><><><><><><><><>
name         coq-float
all-versions 8.5.0  8.6.0  8.7.0  8.8.0  8.9.0  8.10.0

<><> Version-specific details <><><><><><><><><><><><><><><><><><><><><><><><><>
version       8.10.0
repository    coq.inria.fr
url.src:      "https://github.com/coq-contribs/float/archive/v8.10.0.tar.gz"
url.checksum: "md5=29d80b937683f8384fb505c9a6a396db"
homepage:     "https://github.com/coq-contribs/float"
bug-reports:  "https://github.com/coq-contribs/float/issues"
dev-repo:     "git+https://github.com/coq-contribs/float.git"
authors:      "Laurent Théry" "Sylvie Boldo"
maintainer:   "Hugo.Herbelin@inria.fr"
license:      "LGPL 2.1"
tags:         "keyword: floating-point arithmetic"
              "category: Computer Science/Data Types and Data Structures"
              "date: 2001"
flags:        light-uninstall
depends:      "ocaml" "coq" {>= "8.10" & < "8.11~"}
synopsis      Library for floating-point numbers
description   A library for floating-point numbers.

So I think we should replace the --all-switches flag use by the --set-default one in the systemd service used in production.

This is a bit difficult to follow (due to me not being completely sure of what being a switch is ;).

So i got back to basics, the kind "man"ual to clarify the switch. From the manual (man switch), a switch is an independent installation "repository" prefix with its own compiler and sets of installed and pinned packages. That's a bit clearer.

From the man repository, it mentions the options you are referring to:

       -a, --all-switches
           Act on the selections of all configured switches

       --set-default
           Act on the default repository selection that is used for newly created switches

That's less clear as to their purpose (at least for me). In any case, the loader is only relying on the structure of the folder to determine the list of packages with their versions. So i'm fine with the change of flag in the systemd timer you mention.

Furthermore, from an old discussion (and even a fixme in the opam loader), there was supposed to be an opam option (or command, it's fuzzy now) to improve the code to list versions of a package. I did not find mention on this improvment on the opam running in the container so, moving along.

...

Unless this is the command you just showed... opam show $package --root ./opam_root/ (that'd make sense).

One important question though, what happens for the listing on opam.ocaml.org (after the --set-default is set on coq.inria.fr)?

Does this still work?

$ opam show agrid --root ./opam_root/

Or, since we share the opam root, shall we opam switch set opam.ocaml.org (whatever the real call is, i did not check)?

I'd expect this now ^ but i'm unsure.

Basically --set-default adds an opam repository to the default switch, which is the one we are using. By default opam does not associate the repository to any switches so that's why we need to use it.

Furthermore, from an old discussion (and even a fixme in the opam loader), there was supposed to be an opam option (or command, it's fuzzy now) to improve the code to list versions of a package. I did not find mention on this improvment on the opam running in the container so, moving along.

Indeed when testing the loader with pytest or docker, it was failing as the opam version from debian bookworm is 2.1 and the loader expects a repository layout used by opam < 2.1. After reading opam documentation, there is an easy way to get the list of all versions of a package, see below:

/tmp$ opam show --root ./opam_root ipaddr -f all-versions
0.1.0  0.2.0  0.2.1  0.2.2  0.2.3  1.0.0  2.0.0  2.1.0  2.2.0  2.3.0  2.4.0  2.5.0  2.6.0  2.6.1  2.7.0  2.7.1  2.7.2  2.8.0  2.9.0  3.0.0  3.1.0  4.0.0  5.0.0  5.0.1  5.1.0  5.2.0  5.3.0  5.3.1  5.4.0  5.5.0

One important question though, what happens for the listing on opam.ocaml.org (after the --set-default is set on coq.inria.fr)?

It still works as both repositories are in the default switch.

Or, since we share the opam root, shall we opam switch set opam.ocaml.org (whatever the real call is, i did not check)?

No we do not need to call the switch command explicitly as all repositories are associated to the default switch. See example, once a new repository added to the default switch, we can list its packages and get their metadata.

$ opam repository add --root /tmp/opam_root --set-default synchrone https://www-verimag.imag.fr/DIST-TOOLS/SYNCHRONE/opam-repository/
[synchrone] Initialised

$ opam list --root /tmp/opam_root --repos synchrone
# Name      # Installed # Synopsis
bddrand     --          A simple front-end to the lutin Random toss machinary
ezdl        --          Easy dynamic linking of C functions from ocaml
gbddml      --          The Verimag bdd library
lustre-v6   --          The Lustre V6 Verimag compiler
lutils      --          Tools and libs shared by Verimag/synchronous tools (lustre, lutin, rdbg)
lutin       --          Lutin: modeling stochastic reactive systems
polka       --          Polka: convex polyhedron library by Bertrand Jeannet (now part of apron)
rdbg        --          RDBG: a reactive programs debugger
rdbgui4sasa --          A Graphical User Interface for using rdbg with sasa
salut       --          A Lustre API for SASA
sasa        --          SASA: a Self-stabilizing Algorithms SimulAtor

$ opam show --root /tmp/opam_root salut

<><> salut: information on all versions <><><><><><><><><><><><><><><><><><><><>
name         salut
all-versions v4.8.0

<><> Version-specific details <><><><><><><><><><><><><><><><><><><><><><><><><>
version      v4.8.0
repository   synchrone
url.src      "http://www-verimag.imag.fr/DIST-TOOLS/SYNCHRONE/pool/salut.v4.8.0.tgz"
url.checksum "md5=d49e58b4f72ff3b951ae9fcbf97d3f04"
homepage     "https://gricad-gitlab.univ-grenoble-alpes.fr/verimag/synchrone/sasa"
bug-reports  "https://gricad-gitlab.univ-grenoble-alpes.fr/verimag/synchrone/sasa/issues"
dev-repo     "git+https://gricad-gitlab.univ-grenoble-alpes.fr/verimag/synchrone/sasa.git"
authors      "Erwan Jahier and Gabriel B. Sant'Anna"
maintainer   "erwan.jahier@univ-grenoble-alpes.fr"
license      "CeCILL-2.1"
depends      "sasa" "z3" "kind2"
synopsis     A Lustre API for SASA
description  SASA is a *Self-stabilizing Algorithms SimulAtor*. Algorithms in SASA
             are provided in OCaml.
             With Salut, one can provide them in lustre and perform model-checking
             (with lesar or kind2).

For the record, today I reworked the opam loader code by using proper way to list all versions of a package, adding opam 2.1 support plus other improvements. I battled test the changes in docker and it worked great. I will submit a MR next week.

It still works as both repositories are in the default switch.

Great, now I get what I missed earlier! Thanks.

So yes, we should definitely change to use the --set-default as you proposed in the systemd part.

For the record, today I reworked the opam loader code by using proper way to list all versions of a package, adding opam 2.1 support plus other improvements. I battled test the changes in docker and it worked great. I will submit a MR next week.

\o/

Thanks, I had that in mind too.

So I think we should replace the --all-switches flag use by the --set-default one in the systemd service used in production. So yes, we should definitely change to use the --set-default as you proposed in the systemd part.

That'd be [1]

[1] swh/infra/puppet/puppet-swh-site!629 (merged)

Jenkins job DLS/gitlab-builds #117 succeeded .
See Console Output and Coverage Report for more details.

mentioned in merge request swh/infra/puppet/puppet-swh-site!629 (merged)

I'm a bit confused, shouldn't the default be set to the main (presumably, first) opam repository to be added, rather than flipped every time we add another repo?

So, like me (i think), the confusion is on the flag name which is ambiguous. It does not act on the instance of the repository, it acts on the "switch" (independent installation "repository" prefix with its own compiler and sets of installed and pinned packages.). And by default, the flag --set-default acts only on the "main" switch, the one being created initially by the "opam init" command.

So there is actually no flipping. We just add new opam repository to the only switch we have in the opam root and everything is fine after that.

approved this merge request

merged

mentioned in issue swh/infra/sysadm-environment#4971 (closed)

opam: Fix 'opam init' error when relisting an opam instance

Merge request reports

Activity