Antoine Lambert authored 1 month ago and vlorentz committed 2 days ago

This was used at the time we were building debian packages for
swh components but we no longer do that.

This replace the previous usage of psycopg2. Beware that MR is various
dependencies needs to be landed and released accordingly.

We upgrade imports and various function and class names

Data type wrapping seems significantly simpler in psycopg3 so we can drop code
related to it.

Adding various typing information to help catch bug and make mypy happy.

We use context manager more to make sure the connection is getting properly
closed (and not just the cursors) as psycopg3 seems more sensitive to this.

We have to jump through quite hacky hoops to get a connection
information with a (reinjected) password in it. I am a bit nervous about
it. Especially as the test does not seems to exercise this case.

We drop the `execute_values_generator` from core.db as psycopg3 has better
capability in that regards that make it unnecessary in more situation. We have
a reimplementation of it compatible with psycopg3 in swh-storage (part of the
series migrating it to psycopg3)

We are dropping call to ...

If some task parameters are of type dict or list, the command was
previously crashing.

It allows to specify the scheduling policy for the tasks configuration
that will be outputted as CSV, default to "oneshot".

Bump development tools: mypy, codespell, isort, ...

Move all tools configuration in pyproject.toml.

Remove no longer needed mypy overrides.

Usage: swh sentry extract-scheduler-tasks [OPTIONS]

  Extract scheduler task parameters from events.

  This command allows to extract scheduler task parameters from Sentry events
  related to a Software Heritage scheduler task and dumps a CSV file to stdout
  that can be consumed by the CLI command:

  $ swh scheduler task schedule --columns type --columns kwargs <csv_file>.

Options:
  -u, --sentry-url TEXT           Sentry URL  [default:
                                  https://sentry.softwareheritage.org]
  -t, --sentry-token TEXT         Bearer token required to communicate with
                                  Sentry API (can also be provided in
                                  SENTRY_TOKEN environment variable)
                                  [required]
  -i, --sentry-issue-number TEXT  Sentry issue number to extract origin URLs
                                  from its events  [required]
  -e, --environment TEXT          Filter on environment: production or
                                  staging, both are selected by default
  -h, --help                      Show this message and exit.

Usage: swh sentry extract-origin-urls [OPTIONS]

  Extract origin URLs from events.

  This command allows to extract origin URLs from Sentry events related to a
  Software Heritage loader and dumps them to stdout.

Options:
  -u, --sentry-url TEXT           Sentry URL  [default:
                                  https://sentry.softwareheritage.org]
  -t, --sentry-token TEXT         Bearer token required to communicate with
                                  Sentry API (can also be provided in
                                  SENTRY_TOKEN environment variable)
                                  [required]
  -i, --sentry-issue-number TEXT  Sentry issue number to extract origin URLs
                                  from its events  [required]
  -e, --environment TEXT          Filter on environment: production or
                                  staging, both are selected by default
  -h, --help                      Show this message and exit.

It allows to avoid side effects related to sentry_sdk calling git
command for fetching release info.

Refs. swh/infra/sysadm-environment#5515

Refs. swh/infra/sysadm-environment#5515

I noticed tests in other SWH packages were really slow to execute
since recently and it turned out that the root cause of these
slowdowns is that pytest fixtures like the swh_storage one end up
calling the get_swh_backend_module from swh.core.config.

As that function reads Python package entry points from disk, this
slows down tests execution as it is called before each test using
a fixture as described above.

So add LRU cache to these functions to restore previous execution
times in SWH package tests.

version 3.6.3

We used to run all the sql init scripts before trying to determine the
backend's version, only to gather this piece of information at time we
want to write it in the dbversion table.

However, the presence of the dbversion is actually used to determine
whether a db has been initialized or not.

So if the configuration for the backend prevent it from being
instantiated, which is done to retrieve the backend version, then this
last initialization step will fail preventing any future 'swh db' handling
operation from working properly.

In this change, we move the backend version identification logic before
attempting any actual db initialization process so this situation does
not occur.

Add a new enable_requests_retry flag to the swh.core.api.RPCClient
class, default to False, allowing to retry requests sent by the
client when encountering specific errors. Default policy is to retry
when connection errors and transient remote exceptions are raised.
Subclasses can change that policy by overriding the retry_policy
method. Such failed requests will be retry at most five times with a
delay of 10 seconds between each.

Add a new enable_requests_retry flag to the swh.core.api.RPCClient
class, default to False, allowing to retry requests sent by the
client when encountering specific errors. Default policy is to retry
when connection errors and transient remote exceptions are raised.
Subclasses can change that policy by overriding the retry_policy
method. Such failed requests will be retry at most five times with a
delay of 10 seconds between each.

When executing a `swh db init` command, the executed sql files for the
package may fail, but the 'swh db' tool would eat the error reported by
sqlsh. Make these errors shown as output of the process if such an error
occurred.

e.g.:

  $ swh db init test:fail

  Error during database setup
  Command '['psql', '--quiet', '--no-psqlrc', '-v', 'ON_ERROR_STOP=1', '-d', 'postgresql://postgres@127.0.0.1:24370/tests', '-f', '/venv/swh-environment/swh-core/swh/core/db/tests/data/test/fail/sql/40-funcs.sql']' returned non-zero exit status 3.
  Process output:
  psql:/venv/swh-environment/swh-core/swh/core/db/tests/data/test/fail/sql/40-funcs.sql:6: ERROR:  function public.digest(text, unknown) does not exist
  LINE 5:     select encode(public.digest($1, 'sha1'), 'hex')
                          ^
  HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

  Aborted!

Since werkzeug 3.1 release, A CaseInsensitiveDict from requests
library can no longer be initialized with a Headers object from
werkzeug. Explicitly converting a Headers object to a list of
tuples is now required.

Adds a new 'backend' cli group with a 'list' command.

Example:

  $ swh backend list vault

  memory     Stub vault backend, for use in the CLI.
  postgresql Backend for the Software Heritage Vault.
  remote     Client to the Software Heritage vault cache.

  $ swh backend list vault memory
  vault:memory

  Stub vault backend, for use in the CLI.

It makes them more eaily usable.

And use it everywhere possible in swh.core (in particular in
swh/core/tests/test_cli.py itself).

When the `swh db init` command is used with a "full path" module as
argument (like `swh db init storage.proxies.masking`), the db was
initialized with this full path module name instead of the (now
expected) '<package>:<cls>' syntax.

This requires a new helper function to find the registered cls from the
module full name.

Replace mockups of (swh.core) import_swhmodule and get_swh_backend_module
by mockups of lower level functions get_entry_points (actually
backports.entry_points_selectable.entry_points) and importlib.import_module.
This allows to actually test these (previously mocked) functions,
and to clarify a bit the way this loading mechanism works in tests.

Loaded "backends" are now actual python packages located in
<datadir>/test/<cls>.

Actually check they are failing for the expected reason. This needs to
replace usage of subprocess.check_call by subprocess.run with proper
arguments so that the stderr of the process is captured and attached to
the raised CalledProcessError exception.

The cls of a module given as 'package:cls' was wrongly hardwired to
"postgresql", and the `db create` was not properly parsing the given module for the 'package:cls' syntax, like any other commands do.

Rename 'swh/core/test/data/cli' as 'swh/core/test/data/postgresql' so
this directory use the same name as the default db backend
in cli tools (postgresql). This allow to remove a bit of complexity in
mockups for import_swhmodule and get_swh_backend_module.

Also make mock_get_swh_backend_module() depend on
mock_import_swhmodule() since the former need the later to work properly
in most cases.

Make the config entry selection mechanism, used when the
--module-is-path flag is set, target the target config structure instead
of the db connection string within said structure, i.e. use:

  [...] -p storage.steps.1

instead of:

  [...] -p storage.steps.1.db

Add a -p/--module-is-path flag to explicitly activate the search for a
specific config entry in the config file, and replace the ":" path
separator by the (pretty much standard) "."

Make it immune to an existing SWH_CONFIG_FILENAME.

Extract the argument/option handling logic (for init-admin, init,
version, and upgrade commands) in a single handle_cmd_args() function.

Make all db cli commands process given arguments as similar as possible
to each other.

Generalize fallback to using the backend class (declared in the
'swh.<package>.classes' entrypoint) to implement the 'get_datastore()'
if this later is not found in the entry point module.

Also add a message in 'swh db version' if it detects the dbmodule entry
needs to be updated in the database.

aka using backend formatted like '<package>:<cls>' in the dbmodule
table.

i.e. make it aware of modename given as '<packahge>:<cls>', as they are
now stored in the dbmodule table.

Fall back if not entry point is registered for the package.

and use it from populate_database_from_packae() so it's fixed when
running commands like `swh db init` and `swh dn updrade`.

It is IMHO easier to read.