Side effect of adding typing to swh.core.utils.grouper.

This notably removes all the warnings related to comparing a
ModelObjectType enum with a string literal.

As a consequence ModelObjectType enum are used as dict keys
instead of literal strings.

vlorentz authored 2 months ago and vlorentz committed 1 month ago

https://gitlab.softwareheritage.org/swh/devel/swh-loader-metadata/-/blob/c2cdfd5c86d2783ce28c80f84df4930c2eba3cfc/swh/loader/metadata/gitea.py#L121-138

This is used by recent versions of the cassandra Debian package to
override the log directory.

Antoine Lambert authored 1 month ago and Antoine Lambert committed 1 month ago

This was used at the time we were building debian packages for
swh components but we no longer do that.

hypothesis uncovered some edge cases datetime objects where Python will
produce a timestamp offset by one second due to an unexpected rounding,
likely due to float point precision issue during computation.

Set datetime microseconds to 0 before calling the timestamp method to
avoid such side effect and ensure converted date will be the same as
the input one.

Fixes #4720.

Bump development tools: mypy, codespell, isort, ...

Move all tools configuration in pyproject.toml.

Remove no longer needed mypy overrides.

It expected a simple cls cassandra so far but it does not reflect the
production use case (we deploy storage cassandra with pipeline configuration).

It has been observed calling that method can lead to cassandra
query timeout when a snapshot contains a large amount of branches.

The performance bottleneck seems to be the use of a custom cassandra
aggregate function computing the number of branches per target type.

Removing the use of that function and performing the aggregation
client side significantly improves query performance and makes
the timeout issue goes away.

Fixes #4719.

The assertion was failing when executing the code so use a different
one to keep mypy happy.

Also add test to ensure the diagram generation code is covered.

This should speed up a bit tests execution on Jenkins.

Use pytest option --dist=loadfile to group tests by their containing file
in order to avoid spawning too many cassandra or postgres instances.

Recent changes in swh-storage added the PartitionsManagementInterface
but this interface is only implemented on real storage backends, not
proxies.

Any client code overriding the swh_storage_backend_config to return a
storage pipeline (loaders for instance) ends up with error otherwise.

Through a `table_option` dict in the config, mapping table names to what to add to the `WITH` clause of the `CREATE TABLE` clause.

This abstracts this interface from the PostgreSQL-specific implementation,
allowing the object_references table to be sharded in Cassandra.

The storage returned by swh_storage() must be either the same as swh_storage_backend(), or
be a chain of proxies to it. This allows tests to act on the backend using methods
not available on proxies or the RPC

Check these 2 backends can be initialized with `swh db` commands using
both the backend syntax (<package>:<cls>) and the full module name (eg.
storage.proxies.masking).

This requires swh.core 3.6.1 which includes a fix for these.

instead of specific masking_db and blocking_db.
Keep support for these but mark them as deprecated.

Not all codepaths are tested with 3.9, but at least `__init__.py` should work.

This tool can then be used by the sync script filling this db from the
content of the displayname management app in the web app.

Fix broken tests due to the introduction of the ModelObjectType enum.

Closes #4712

For consistency with other *_get methods in this interface, set
return type to list instead of iterable, plus it enables to use
the __getitem__ method in client code without mypy errors.

Latest tenacity release adds some internal changes that broke the
mocking of sleep calls in tests.

Fix it by directly mocking time.sleep (was not working previously).

It is fairly easy to overlook calling `storage.flush()` after adding
some objects just before the end of a process. In a direct
configuration, everything will be fine but if a buffering proxy is
configured, this means that the objects still in the buffers are going
to be lost. Let’s add a warning if this is the case.