Side effect of adding typing to swh.core.utils.grouper.

This notably removes all the warnings related to comparing a
ModelObjectType enum with a string literal.

As a consequence ModelObjectType enum are used as dict keys
instead of literal strings.

vlorentz authored 2 months ago and vlorentz committed 1 month ago

https://gitlab.softwareheritage.org/swh/devel/swh-loader-metadata/-/blob/c2cdfd5c86d2783ce28c80f84df4930c2eba3cfc/swh/loader/metadata/gitea.py#L121-138

This is used by recent versions of the cassandra Debian package to
override the log directory.

Antoine Lambert authored 2 months ago and Antoine Lambert committed 2 months ago

This was used at the time we were building debian packages for
swh components but we no longer do that.

hypothesis uncovered some edge cases datetime objects where Python will
produce a timestamp offset by one second due to an unexpected rounding,
likely due to float point precision issue during computation.

Set datetime microseconds to 0 before calling the timestamp method to
avoid such side effect and ensure converted date will be the same as
the input one.

Fixes #4720.

Bump development tools: mypy, codespell, isort, ...

Move all tools configuration in pyproject.toml.

Remove no longer needed mypy overrides.

It expected a simple cls cassandra so far but it does not reflect the
production use case (we deploy storage cassandra with pipeline configuration).

It has been observed calling that method can lead to cassandra
query timeout when a snapshot contains a large amount of branches.

The performance bottleneck seems to be the use of a custom cassandra
aggregate function computing the number of branches per target type.

Removing the use of that function and performing the aggregation
client side significantly improves query performance and makes
the timeout issue goes away.

Fixes #4719.

The assertion was failing when executing the code so use a different
one to keep mypy happy.

Also add test to ensure the diagram generation code is covered.

test_extid_add_hg expects all hg revisions to have a 'node' extra header,
which minimal_revision does not have.

vlorentz authored 4 months ago and vlorentz committed 3 months ago

Cassandra does not support filtering on individual fields of UDTs, as it considers
structures as a single whole value.

However, the infra team needs to filter on author.email and committer.email, hence the need
for separate columns.

This commit reads and writes the new split columns, but keeps reading the UDT as
a fallback. This will be removed after we are done migrating all rows.

Migration plan:

1.
   ```
   ALTER TABLE revision
   ADD (
       author_fullname                 blob,
       author_name                     blob,
       author_email                    blob,
       committer_fullname              blob,
       committer_name                  blob,
       committer_email                 blob
   );
   ALTER TABLE release
   ADD (
       author_fullname                 blob,
       author_name                     blob,
       author_email                    blob
   );
   ```

2. update Python code and restart

3. run a replayer on `revision` and `release` objects without a filtering proxy,
   in order to write the new columns

This should speed up a bit tests execution on Jenkins.

Use pytest option --dist=loadfile to group tests by their containing file
in order to avoid spawning too many cassandra or postgres instances.

Recent changes in swh-storage added the PartitionsManagementInterface
but this interface is only implemented on real storage backends, not
proxies.

Any client code overriding the swh_storage_backend_config to return a
storage pipeline (loaders for instance) ends up with error otherwise.

Through a `table_option` dict in the config, mapping table names to what to add to the `WITH` clause of the `CREATE TABLE` clause.

This abstracts this interface from the PostgreSQL-specific implementation,
allowing the object_references table to be sharded in Cassandra.

The storage returned by swh_storage() must be either the same as swh_storage_backend(), or
be a chain of proxies to it. This allows tests to act on the backend using methods
not available on proxies or the RPC

Check these 2 backends can be initialized with `swh db` commands using
both the backend syntax (<package>:<cls>) and the full module name (eg.
storage.proxies.masking).

This requires swh.core 3.6.1 which includes a fix for these.