This will need coordination with other MR migrating other part of swh to
psycopg3, more notably, swh-core.

Updates various imports and names.

Adjusts syntax query and parameter to what psycopg3 expect. The `origins_are_blocked` of the Blocking proxy class needed significant rework for equivalent features.

Dropped timezone related work around that existed for psycopg2 < 2.9.0

Adjust the way we retrieve the connection information. This update as the same caveas as for swh-core and needs some double check.

Use the dedicated `psycopg_pool` package for connection pull.

Adds a small `execute_values_generator` helper to match existing code
expectation better (reducing the complexity of this diff).

Drops a call to the deprecated and noop `app.setup_psycopg_errorhandlers`

Add more type to help mypy and catch errors.

Modify swh_storage_cassandra_cluster to ensure a single instance of
cassandra is spawned when running tests in parallel with pytest-xdist.

Explicitly shutdown clusters create by CqlRunner instances to avoid the
display of traceback related to Python threads when cassandra related
fixtures are tear downed.

These changes significantly reduce the time to execute a tox session
(more than a x2 speedup).

It allows to avoid the display of tracebacks related to Python threads
when cassandra related pytest fixtures are tear downed.

Side effect of adding typing to swh.core.utils.grouper.

This notably removes all the warnings related to comparing a
ModelObjectType enum with a string literal.

As a consequence ModelObjectType enum are used as dict keys
instead of literal strings.

vlorentz authored 1 month ago and vlorentz committed 1 month ago

https://gitlab.softwareheritage.org/swh/devel/swh-loader-metadata/-/blob/c2cdfd5c86d2783ce28c80f84df4930c2eba3cfc/swh/loader/metadata/gitea.py#L121-138

This is used by recent versions of the cassandra Debian package to
override the log directory.

Antoine Lambert authored 1 month ago and Antoine Lambert committed 1 month ago

This was used at the time we were building debian packages for
swh components but we no longer do that.

hypothesis uncovered some edge cases datetime objects where Python will
produce a timestamp offset by one second due to an unexpected rounding,
likely due to float point precision issue during computation.

Set datetime microseconds to 0 before calling the timestamp method to
avoid such side effect and ensure converted date will be the same as
the input one.

Fixes #4720.

Bump development tools: mypy, codespell, isort, ...

Move all tools configuration in pyproject.toml.

Remove no longer needed mypy overrides.

It expected a simple cls cassandra so far but it does not reflect the
production use case (we deploy storage cassandra with pipeline configuration).

It has been observed calling that method can lead to cassandra
query timeout when a snapshot contains a large amount of branches.

The performance bottleneck seems to be the use of a custom cassandra
aggregate function computing the number of branches per target type.

Removing the use of that function and performing the aggregation
client side significantly improves query performance and makes
the timeout issue goes away.

Fixes #4719.

The assertion was failing when executing the code so use a different
one to keep mypy happy.

Also add test to ensure the diagram generation code is covered.

This should speed up a bit tests execution on Jenkins.

Use pytest option --dist=loadfile to group tests by their containing file
in order to avoid spawning too many cassandra or postgres instances.

Recent changes in swh-storage added the PartitionsManagementInterface
but this interface is only implemented on real storage backends, not
proxies.

Any client code overriding the swh_storage_backend_config to return a
storage pipeline (loaders for instance) ends up with error otherwise.

Through a `table_option` dict in the config, mapping table names to what to add to the `WITH` clause of the `CREATE TABLE` clause.

This abstracts this interface from the PostgreSQL-specific implementation,
allowing the object_references table to be sharded in Cassandra.

The storage returned by swh_storage() must be either the same as swh_storage_backend(), or
be a chain of proxies to it. This allows tests to act on the backend using methods
not available on proxies or the RPC

Check these 2 backends can be initialized with `swh db` commands using
both the backend syntax (<package>:<cls>) and the full module name (eg.
storage.proxies.masking).

This requires swh.core 3.6.1 which includes a fix for these.

instead of specific masking_db and blocking_db.
Keep support for these but mark them as deprecated.

Not all codepaths are tested with 3.9, but at least `__init__.py` should work.

This tool can then be used by the sync script filling this db from the
content of the displayname management app in the web app.