Since werkzeug 3.1 release, A CaseInsensitiveDict from requests
library can no longer be initialized with a Headers object from
werkzeug. Explicitly converting a Headers object to a list of
tuples is now required.

Adds a new 'backend' cli group with a 'list' command.

Example:

  $ swh backend list vault

  memory     Stub vault backend, for use in the CLI.
  postgresql Backend for the Software Heritage Vault.
  remote     Client to the Software Heritage vault cache.

  $ swh backend list vault memory
  vault:memory

  Stub vault backend, for use in the CLI.

It makes them more eaily usable.

And use it everywhere possible in swh.core (in particular in
swh/core/tests/test_cli.py itself).

When the `swh db init` command is used with a "full path" module as
argument (like `swh db init storage.proxies.masking`), the db was
initialized with this full path module name instead of the (now
expected) '<package>:<cls>' syntax.

This requires a new helper function to find the registered cls from the
module full name.

Replace mockups of (swh.core) import_swhmodule and get_swh_backend_module
by mockups of lower level functions get_entry_points (actually
backports.entry_points_selectable.entry_points) and importlib.import_module.
This allows to actually test these (previously mocked) functions,
and to clarify a bit the way this loading mechanism works in tests.

Loaded "backends" are now actual python packages located in
<datadir>/test/<cls>.

Actually check they are failing for the expected reason. This needs to
replace usage of subprocess.check_call by subprocess.run with proper
arguments so that the stderr of the process is captured and attached to
the raised CalledProcessError exception.

The cls of a module given as 'package:cls' was wrongly hardwired to
"postgresql", and the `db create` was not properly parsing the given module for the 'package:cls' syntax, like any other commands do.

Rename 'swh/core/test/data/cli' as 'swh/core/test/data/postgresql' so
this directory use the same name as the default db backend
in cli tools (postgresql). This allow to remove a bit of complexity in
mockups for import_swhmodule and get_swh_backend_module.

Also make mock_get_swh_backend_module() depend on
mock_import_swhmodule() since the former need the later to work properly
in most cases.

Make the config entry selection mechanism, used when the
--module-is-path flag is set, target the target config structure instead
of the db connection string within said structure, i.e. use:

  [...] -p storage.steps.1

instead of:

  [...] -p storage.steps.1.db

Add a -p/--module-is-path flag to explicitly activate the search for a
specific config entry in the config file, and replace the ":" path
separator by the (pretty much standard) "."

Make it immune to an existing SWH_CONFIG_FILENAME.

Extract the argument/option handling logic (for init-admin, init,
version, and upgrade commands) in a single handle_cmd_args() function.

Make all db cli commands process given arguments as similar as possible
to each other.

Generalize fallback to using the backend class (declared in the
'swh.<package>.classes' entrypoint) to implement the 'get_datastore()'
if this later is not found in the entry point module.

Also add a message in 'swh db version' if it detects the dbmodule entry
needs to be updated in the database.

aka using backend formatted like '<package>:<cls>' in the dbmodule
table.

i.e. make it aware of modename given as '<packahge>:<cls>', as they are
now stored in the dbmodule table.

Fall back if not entry point is registered for the package.

and use it from populate_database_from_packae() so it's fixed when
running commands like `swh db init` and `swh dn updrade`.

It is IMHO easier to read.

Allow db commands (create, init-admin, init, version, upgrade) get the
db connection string from anywhere in the config file.
For example, it can be used like:

$ cat conf.yml
storage:
  cls: pipeline
  steps:
    - cls: masking
      masking_db: postgresql:///?service=swh-masking-proxy
    - cls: buffer
    - cls: postgresql
      db: postgresql://user:passwd@pghost:5433/swh-storage
      objstorage:
        cls: memory

$ swh db -C conf.yml init storage:steps:0:masking_db

This de facto deprecates the usage of the --module-config-key option.

Also add a -a/--all option to these commands allowing to execute the given
command to all matching configuration entries in the config file; this
should help deployment in testing environments like docker.

For example:

  $ swh db version -a storage

  module: storage.proxies.masking
  current code version: 194
  version: 194

  module: storage
  flavor: default
  current code version: 193
  version: 193

This change also make the module name stored in the dbmodule table an
fully qualified module instead of shortened version, i.e. it will now
store 'swh.storage.postgresql.storage' instead of just 'storage',
so there is less "implicit logic" (like removing "swh.", etc.)
when filling/using this piece of information.

We add a bw compat support in get_sql_for_package to ease the transition
until the dbmodule tables are updated. This bw compat logic will
look for the 'sql/' directory not only in the package directly for the
module, but also in parent directories. This allows to keep the code
compatible with current version of swh.storage (in which the sql/
directory and the actual backend submodule do not match).

Doing this, we also stop defining a hardwritten dbname ('softwareheritage-dev')
as default value for the '--dbname' option in 'swh db create' command.

Warning: for the 'create' and 'init-admin' db commands, looking for the
connection string in the config file is mainly useful for test environments
in which the user used to access the backend database have superuser
permissions on the database. In production-like environments, you most
certainly will prefer using an explicit --dbname argument with a
connection with enough permission to handle the job.

Show the full traceback of the exception that occurred in the cli_runner
execution, if any.

It better captures the semantic of the option and makes the '--all'
option free for a common usage to handle the 'operate on all db backends
in the configuration file' semantic (i.e. replace '--initialize-all' and
'--upgrade-all' options) in the future.

importlib.metadata.entry_points only supports the selection behavior
in python 3.10+.

When running command line tools like `swh alter` or `swh scanner`, there
is rarely a Sentry DSN set. As errors are most often monitored from the
terminal in such configuration, let’s not display a warning that is more
noise than signal.

`.called_once_with` doesn't exist, so it was just returning a mock
object which was boolean True; however pytest-mock is now catching
this (correctly) as a programming error, so handle the call args of the
exception capture properly.

YAML mappings are converted in Python as dict. The order
of the keys in dicts returned by `load_from_envvar()` were
not stable from one Python run to the next.

Since Python 3.6, dicts are ordered by default, and `yaml.safe_load()`
will always return keys in the same order. The culprit was
`merge_configs` which created a set (which are unordered) of keys
to perform the merge.

Latest tenacity release adds some internal changes that broke the
mocking of sleep calls in tests.

Fix it by directly mocking time.sleep (was not working previously).

This avoids having to use two separate configuration files for
initializing and then running a storage instance.

This will ease database introspection from a pod toolbox without the need to configure
extra specific postgresql.

Co-authored-by: Nicolas Dandrimont <olasd@softwareheritage.org>

We now have dotted modules whose section in the configuration file no longer match their
name. The `db upgrade` cli had been evolved to deal with it but not the `db init` cli.

This should mitigate errors related to streamed HTTP responses.

Migration to importlib introduced a bug when loading swh cli
entrypoints: the entry point group name must be given with a named
argument.

There are still situations where the swh root namespace is not properly
handled when running tests from pytest in an editable-mode installed
package situation, so make the 2 concerned tests pass in both cases.

Also replace a few `assert data.items() <= {xxx: xxx}.items()' idioms by
easier to read and understand statements.