Bump development tools: mypy, codespell, isort, ...

Move all tools configuration in pyproject.toml.

Remove no longer needed mypy overrides.

Usage: swh sentry extract-scheduler-tasks [OPTIONS]

  Extract scheduler task parameters from events.

  This command allows to extract scheduler task parameters from Sentry events
  related to a Software Heritage scheduler task and dumps a CSV file to stdout
  that can be consumed by the CLI command:

  $ swh scheduler task schedule --columns type --columns kwargs <csv_file>.

Options:
  -u, --sentry-url TEXT           Sentry URL  [default:
                                  https://sentry.softwareheritage.org]
  -t, --sentry-token TEXT         Bearer token required to communicate with
                                  Sentry API (can also be provided in
                                  SENTRY_TOKEN environment variable)
                                  [required]
  -i, --sentry-issue-number TEXT  Sentry issue number to extract origin URLs
                                  from its events  [required]
  -e, --environment TEXT          Filter on environment: production or
                                  staging, both are selected by default
  -h, --help                      Show this message and exit.

Usage: swh sentry extract-origin-urls [OPTIONS]

  Extract origin URLs from events.

  This command allows to extract origin URLs from Sentry events related to a
  Software Heritage loader and dumps them to stdout.

Options:
  -u, --sentry-url TEXT           Sentry URL  [default:
                                  https://sentry.softwareheritage.org]
  -t, --sentry-token TEXT         Bearer token required to communicate with
                                  Sentry API (can also be provided in
                                  SENTRY_TOKEN environment variable)
                                  [required]
  -i, --sentry-issue-number TEXT  Sentry issue number to extract origin URLs
                                  from its events  [required]
  -e, --environment TEXT          Filter on environment: production or
                                  staging, both are selected by default
  -h, --help                      Show this message and exit.

It allows to avoid side effects related to sentry_sdk calling git
command for fetching release info.

Refs. swh/infra/sysadm-environment#5515

Refs. swh/infra/sysadm-environment#5515

I noticed tests in other SWH packages were really slow to execute
since recently and it turned out that the root cause of these
slowdowns is that pytest fixtures like the swh_storage one end up
calling the get_swh_backend_module from swh.core.config.

As that function reads Python package entry points from disk, this
slows down tests execution as it is called before each test using
a fixture as described above.

So add LRU cache to these functions to restore previous execution
times in SWH package tests.

version 3.6.3

We used to run all the sql init scripts before trying to determine the
backend's version, only to gather this piece of information at time we
want to write it in the dbversion table.

However, the presence of the dbversion is actually used to determine
whether a db has been initialized or not.

So if the configuration for the backend prevent it from being
instantiated, which is done to retrieve the backend version, then this
last initialization step will fail preventing any future 'swh db' handling
operation from working properly.

In this change, we move the backend version identification logic before
attempting any actual db initialization process so this situation does
not occur.

Add a new enable_requests_retry flag to the swh.core.api.RPCClient
class, default to False, allowing to retry requests sent by the
client when encountering specific errors. Default policy is to retry
when connection errors and transient remote exceptions are raised.
Subclasses can change that policy by overriding the retry_policy
method. Such failed requests will be retry at most five times with a
delay of 10 seconds between each.

Add a new enable_requests_retry flag to the swh.core.api.RPCClient
class, default to False, allowing to retry requests sent by the
client when encountering specific errors. Default policy is to retry
when connection errors and transient remote exceptions are raised.
Subclasses can change that policy by overriding the retry_policy
method. Such failed requests will be retry at most five times with a
delay of 10 seconds between each.

When executing a `swh db init` command, the executed sql files for the
package may fail, but the 'swh db' tool would eat the error reported by
sqlsh. Make these errors shown as output of the process if such an error
occurred.

e.g.:

  $ swh db init test:fail

  Error during database setup
  Command '['psql', '--quiet', '--no-psqlrc', '-v', 'ON_ERROR_STOP=1', '-d', 'postgresql://postgres@127.0.0.1:24370/tests', '-f', '/venv/swh-environment/swh-core/swh/core/db/tests/data/test/fail/sql/40-funcs.sql']' returned non-zero exit status 3.
  Process output:
  psql:/venv/swh-environment/swh-core/swh/core/db/tests/data/test/fail/sql/40-funcs.sql:6: ERROR:  function public.digest(text, unknown) does not exist
  LINE 5:     select encode(public.digest($1, 'sha1'), 'hex')
                          ^
  HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

  Aborted!

Since werkzeug 3.1 release, A CaseInsensitiveDict from requests
library can no longer be initialized with a Headers object from
werkzeug. Explicitly converting a Headers object to a list of
tuples is now required.

Adds a new 'backend' cli group with a 'list' command.

Example:

  $ swh backend list vault

  memory     Stub vault backend, for use in the CLI.
  postgresql Backend for the Software Heritage Vault.
  remote     Client to the Software Heritage vault cache.

  $ swh backend list vault memory
  vault:memory

  Stub vault backend, for use in the CLI.

It makes them more eaily usable.

And use it everywhere possible in swh.core (in particular in
swh/core/tests/test_cli.py itself).

When the `swh db init` command is used with a "full path" module as
argument (like `swh db init storage.proxies.masking`), the db was
initialized with this full path module name instead of the (now
expected) '<package>:<cls>' syntax.

This requires a new helper function to find the registered cls from the
module full name.

Replace mockups of (swh.core) import_swhmodule and get_swh_backend_module
by mockups of lower level functions get_entry_points (actually
backports.entry_points_selectable.entry_points) and importlib.import_module.
This allows to actually test these (previously mocked) functions,
and to clarify a bit the way this loading mechanism works in tests.

Loaded "backends" are now actual python packages located in
<datadir>/test/<cls>.

Actually check they are failing for the expected reason. This needs to
replace usage of subprocess.check_call by subprocess.run with proper
arguments so that the stderr of the process is captured and attached to
the raised CalledProcessError exception.

The cls of a module given as 'package:cls' was wrongly hardwired to
"postgresql", and the `db create` was not properly parsing the given module for the 'package:cls' syntax, like any other commands do.

Rename 'swh/core/test/data/cli' as 'swh/core/test/data/postgresql' so
this directory use the same name as the default db backend
in cli tools (postgresql). This allow to remove a bit of complexity in
mockups for import_swhmodule and get_swh_backend_module.

Also make mock_get_swh_backend_module() depend on
mock_import_swhmodule() since the former need the later to work properly
in most cases.

Make the config entry selection mechanism, used when the
--module-is-path flag is set, target the target config structure instead
of the db connection string within said structure, i.e. use:

  [...] -p storage.steps.1

instead of:

  [...] -p storage.steps.1.db

Add a -p/--module-is-path flag to explicitly activate the search for a
specific config entry in the config file, and replace the ":" path
separator by the (pretty much standard) "."

Make it immune to an existing SWH_CONFIG_FILENAME.

Extract the argument/option handling logic (for init-admin, init,
version, and upgrade commands) in a single handle_cmd_args() function.

Make all db cli commands process given arguments as similar as possible
to each other.

Generalize fallback to using the backend class (declared in the
'swh.<package>.classes' entrypoint) to implement the 'get_datastore()'
if this later is not found in the entry point module.

Also add a message in 'swh db version' if it detects the dbmodule entry
needs to be updated in the database.

aka using backend formatted like '<package>:<cls>' in the dbmodule
table.

i.e. make it aware of modename given as '<packahge>:<cls>', as they are
now stored in the dbmodule table.

Fall back if not entry point is registered for the package.

and use it from populate_database_from_packae() so it's fixed when
running commands like `swh db init` and `swh dn updrade`.

It is IMHO easier to read.

Allow db commands (create, init-admin, init, version, upgrade) get the
db connection string from anywhere in the config file.
For example, it can be used like:

$ cat conf.yml
storage:
  cls: pipeline
  steps:
    - cls: masking
      masking_db: postgresql:///?service=swh-masking-proxy
    - cls: buffer
    - cls: postgresql
      db: postgresql://user:passwd@pghost:5433/swh-storage
      objstorage:
        cls: memory

$ swh db -C conf.yml init storage:steps:0:masking_db

This de facto deprecates the usage of the --module-config-key option.

Also add a -a/--all option to these commands allowing to execute the given
command to all matching configuration entries in the config file; this
should help deployment in testing environments like docker.

For example:

  $ swh db version -a storage

  module: storage.proxies.masking
  current code version: 194
  version: 194

  module: storage
  flavor: default
  current code version: 193
  version: 193

This change also make the module name stored in the dbmodule table an
fully qualified module instead of shortened version, i.e. it will now
store 'swh.storage.postgresql.storage' instead of just 'storage',
so there is less "implicit logic" (like removing "swh.", etc.)
when filling/using this piece of information.

We add a bw compat support in get_sql_for_package to ease the transition
until the dbmodule tables are updated. This bw compat logic will
look for the 'sql/' directory not only in the package directly for the
module, but also in parent directories. This allows to keep the code
compatible with current version of swh.storage (in which the sql/
directory and the actual backend submodule do not match).

Doing this, we also stop defining a hardwritten dbname ('softwareheritage-dev')
as default value for the '--dbname' option in 'swh db create' command.

Warning: for the 'create' and 'init-admin' db commands, looking for the
connection string in the config file is mainly useful for test environments
in which the user used to access the backend database have superuser
permissions on the database. In production-like environments, you most
certainly will prefer using an explicit --dbname argument with a
connection with enough permission to handle the job.

Show the full traceback of the exception that occurred in the cli_runner
execution, if any.

It better captures the semantic of the option and makes the '--all'
option free for a common usage to handle the 'operate on all db backends
in the configuration file' semantic (i.e. replace '--initialize-all' and
'--upgrade-all' options) in the future.

importlib.metadata.entry_points only supports the selection behavior
in python 3.10+.