This allows to reduce the boilerplate regarding initial connection failures.

types-psycopg2 2.9.12 contains this commit:
https://github.com/python/typeshed/commit/506be4fb0adcf38af77325258af337772abcde77

which removes `Any` as the return type of the last variant, allowing
mypy to type-check the returned value.
However, mypy 0.920 only considers the last variant, even though we use
the first one; causing the return type to be badly inferred, and
erroring every time we use it as context manager.

That hook can be frustrating as it can discard a long commit message
if it finds a typo in it so better removing it.

Instead of only overriding the initial settings when they're left unset,
always override the defaults when the environment variables are set.

This makes the behavior more consistent with the way we usually handle
environment variables. It also allows setting the environment variable
`SWH_SENTRY_DISABLE_LOGGING_EVENTS=false`, to enable events based on the
logging framework in gunicorn backends (where the default has been
flipped to true).

Notably, using logging.config.dictConfig disables all loggers before
setting its own config. There's no simple way of undoing these changes
to the logging config.

Even if we had a way to reset a basic logging config, pytest reuses its
log handler instances across tests, so it's not even that simple to be
able to reset the logging config from scratch and still have pytest able
to capture logs.

Instead of any of that, just add a fixture to reset the root logger
level, and to mock logging.config.dictConfig so that it doesn't actually
get called.

By default, sentry's logging integration will report any error or
exception-level logging as an event.

Now that we explicitly capture exceptions in the RPC server app, we can
disable this automatic event generation, and only use the logging
integration to generate breadcrumbs.

add a configuration option to init_sentry to disable the error reporting
from logging by sentry.

This can be set via the SWH_SENTRY_DISABLE_LOGGING_EVENTS env var.
Default behavior remains unchanged.

Co-Authored-By: David Douard <david.douard@sdfa3.org>

The error handler for RPC server classes is used for both unexpected
exceptions (using a 5xx status code), and exceptions used for the normal
execution flow, which are encoded on the server side, and decoded to be
raised again on the client side (and use a 4xx status code). We only
record sentry errors for the 5xx class exceptions.

This will give us a rough classification of the errors of timed
functions, which will allow us to keep track of trends, without having
to report all errors to sentry for analysis.

Related to T3922

black is considered stable since release 22.1.0 and the version
we are currently using is quite outdated and not compatible with
click 8.1.0, so it is time to bump it to its latest stable release.

Please note that E501 pycodestyle warning related to line length
is replaced by B950 one from flake8-bugbear as recommended by black.
https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length

Related to T3922

pytest-postgresql 3.1.3 and pytest-redis 2.4.0 added support for
pytest >= 7 so we can now drop the pytest pinning.

allowing these to be easily set from a config file.

Since https://github.com/pallets/werkzeug/commit/74601150b559c73a8c8732b4c67bbc6377d1a7df
(published yesterday as part of version 2.1.0), Werkzeug crashes when WERKZEUG_RUN_MAIN
is true but WERKZEUG_SERVER_FD is not provided.

This is legitimate, because applications should not set WERKZEUG_RUN_MAIN.

This commit rewrites the hack hiding the server banner so it still works
with this Werkzeug version.

Due to test modules being copied in subdirectories of the
build directory by setuptools, it makes pytest fail by raising
ImportPathMismatchError exceptions when invoked from root
directory of the module.

So ignore the build folder to discover tests.

so it can be used in a script (e.g. docker entrypoint).

when no migration script is to be executed.

This avoids spamming a deprecation error clients can't do anything about.

swh_set_db_module() takes a db_or_conninfo object, and passes it to execute_sqlfiles,
which only expects a conninfo.

This is detected by mypy when using a recent version of types-psycopg2.

This may be needed depending on import order; but it makes mypy complain when
types-psycopg2 is installed.

It will be used by the swh-objstorage client in 'list_content' to efficiently
chunk the output into sha1 digests without relying on HTTP chunking from the server.

This will be used by the objstorage (and possibly the storage as well)
to publish metrics to statsd.

I want to make swh-objstorage inherit from RPCClient, but its get() and get_stream()
methods would class with RPCClient's.

move the import statement in swhmain() body.
Having the swh.core module loaded with the execution of conftest.py can
have weird side effects, especially on pytest's discovery mechanism in
the context of tox, namely make pytest load test files from the source
directory rather than from the installed package in the tox venv,
potentially leading to ImportPathMismatchError.

this is far from completion.

in favor of using (and thus testing) the stock get_sql_for_package()
function (on the mock module returned by mock_import_swhmodule).

This comes with a change in expected test data structure for sql
script; the `tests/data/<mod>` is now expected to store the sql script
files in a dedicated `sql/` sub directory instead of having them
directly in the `<mod>` directory so that the get_sql_for_package() can
be used in tests unmocked.

update the postgresql_fact fixture to be able to use new-ish
pytest_postgresql ability to recreate databases between tests from a
template database.

The ides is to stop using the fragile and confusing "reset" of the
database between tests, which means de facto keep a stateful database
between tests, which make tests result depend on de facto order of
execution of these tests.

For example, test_smoke_test_people_db_up_and_reset cannot be executed
alone, since it depends on the fact test_smoke_test_people_db_up has
been executed before so that the people's db has been reset as a result
of the test setup done by the fixture, which will happen if and only if
the database already exists (thus if another test using this db has been
executed before in this test session).

The old table truncate based behavior has been kept (using a no_db_drop
flag argument) in this diff for bw compatibility reasons, and also because
the new drop-and-create-from-template beahvior remains a bit slower,
which might be a problem for some test cases.

Tests for dependencies (swh-storage, etc.) should still pass with this
refactoring, but will be significantly slower.

A series of revisions for these dependencies is about to be proposed to
adapt them to the new db metadata (version, module, flavor) handling
feature coming with swh.core 2, as well as updating their tests to take
advantage of the template-based db creation in tests, which should bring
back test execution time to almost the same as before this refactoring.

This is needed to be able to implement and test support for the new 'swh
db upgrade' all this is actually about.

this new command allows to handle the database migration from the
current version to the latest version advertized but the backend
datastore.

This feature is only working for "new style" swh.core.db based
datastores.

for this feature to work properly, the backend datastore package must
provide the 'get_datastore' factory function in its root namespace and
the datastore instance must provide a 'get_current_version()' method.

This also means the canonical cli usage for the `swh db init` is now
using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
instead of giving only the db cnx string (`--db-name`), so that the
backend datastore class can be instanciated using the `get_datastore()`
factory function.

However a '--initial-version' cli option has been added to 'swh db init'
to make it easier, especially for unit tests.

instead of a custom 'test_db' (which did not truncate dbversion and
origin tables, not sure what the intent was for this).

for now, managing the dbversion table (storing the db schema upgrade
history) is partially the responsibility of the swh modules implementing
a datastore (swh-storage etc.), and the actual swh module was not stored
in the database.

This adds support for both the dbversion and dbmodule tables management
in swh.core.db.

The `swh db init` command now creates both these tables and initialize
their values if possible.
For this, it introduces a new common "API" for swh.core.db based datastores:

- a swh module implementing a datastore is expected to have a
  `get_datastore` function in its top namespace; this function is
  typically the actual `get_storage`, `get_scheduler` etc. functions
- this factory function is expected to use the "postgresql" cls for the
  datastore based on swh.core.db,
- the datastore instance (eg. instance of the swh.storage.postgresql.Storage
  class) is expected to have a `get_current_version()` method (returning
  an int).

This revision also provides a new `swh db version` command displaying
the current (code) version (if available), the db version (possibly the whole
version history) and flavor (if any) for the datastore given as argument.

It should be backward compatible with existing swh datastore modules (i.e.
not yet adapted to this new API).

will be used to implement several test scenarios in following commits.
Also rename existing sql scripts with a "legit" numbering scheme.