It has been observed calling that method can lead to cassandra
query timeout when a snapshot contains a large amount of branches.

The performance bottleneck seems to be the use of a custom cassandra
aggregate function computing the number of branches per target type.

Removing the use of that function and performing the aggregation
client side significantly improves query performance and makes
the timeout issue goes away.

Fixes #4719.

The assertion was failing when executing the code so use a different
one to keep mypy happy.

Also add test to ensure the diagram generation code is covered.

This should speed up a bit tests execution on Jenkins.

Use pytest option --dist=loadfile to group tests by their containing file
in order to avoid spawning too many cassandra or postgres instances.

Recent changes in swh-storage added the PartitionsManagementInterface
but this interface is only implemented on real storage backends, not
proxies.

Any client code overriding the swh_storage_backend_config to return a
storage pipeline (loaders for instance) ends up with error otherwise.

Through a `table_option` dict in the config, mapping table names to what to add to the `WITH` clause of the `CREATE TABLE` clause.

This abstracts this interface from the PostgreSQL-specific implementation,
allowing the object_references table to be sharded in Cassandra.

The storage returned by swh_storage() must be either the same as swh_storage_backend(), or
be a chain of proxies to it. This allows tests to act on the backend using methods
not available on proxies or the RPC

Check these 2 backends can be initialized with `swh db` commands using
both the backend syntax (<package>:<cls>) and the full module name (eg.
storage.proxies.masking).

This requires swh.core 3.6.1 which includes a fix for these.

instead of specific masking_db and blocking_db.
Keep support for these but mark them as deprecated.

Not all codepaths are tested with 3.9, but at least `__init__.py` should work.

This tool can then be used by the sync script filling this db from the
content of the displayname management app in the web app.

Fix broken tests due to the introduction of the ModelObjectType enum.

Closes #4712

For consistency with other *_get methods in this interface, set
return type to list instead of iterable, plus it enables to use
the __getitem__ method in client code without mypy errors.

Latest tenacity release adds some internal changes that broke the
mocking of sleep calls in tests.

Fix it by directly mocking time.sleep (was not working previously).

It is fairly easy to overlook calling `storage.flush()` after adding
some objects just before the end of a process. In a direct
configuration, everything will be fine but if a buffering proxy is
configured, this means that the objects still in the buffers are going
to be lost. Let’s add a warning if this is the case.

When deleting a snapshot, deletion removed the corresponding entries in
the `snapshot_branch` table. These entries are shared between
snapshots (e.g. all snapshots with a `HEAD` pointing to `refs/heads/main`
share the same line in this table), and we don't keep a reverse index to
efficiently remove orphan lines, so we just skip the removal altogether.

Add a test to ensure that the corruption doesn't happen anymore.

These changes are needed to allow `swh alter remove` to remove
RawExtrinsicMetadata and ExtID objects from the PostgreSQL and Cassandra
implementation of the storage.

As RawExtrinsicMetadata objects can be addressed using an
ExtendendSWHID, they can now be removed using `object_delete()`.

For ExtID, a new method `extid_delete_for_target()` has been added to
`ObjectDeletionInterface`. It will remove any ExtID object targeting one
of the object in the a list of SWHID given as an argument.

Address swh-alter!11

The cursor provided by the `@db_transaction` decorator was not
being passed properly.

To be able to use the `swh db` command line, the configurations need to
have the following shape:

<toplevel key>:
  cls: postgresql
  db: <postgresql DSN>

The usage of `blocking_db` and `masking_db` instead of plain `db`, and
lack of overridability in the swh db utilities, made schema migrations
more annoying than necessary.

The old configuration key still works but raises a Deprecation warning.

The configuration of the proxies themselves is unchanged, as having a
differently-named key for the proxy-specific database makes sense to
avoid confusion.

The prefix matching semantics rely on URL patterns not ending with a /.
URLs ending in / will only be used for exact matches. Warn the operator
of this situation.

Instead of doing some collation-dependent, somewhat brittle prefix
matching in PostgreSQL, generate a list of checked prefixes and match
them exactly in the database. This means we can do both exact and prefix
matching in one go.

If that parameter is provided, it enables to get the latest snapshot
produced by a specific visit type.

Related to swh/meta#5092.

vlorentz authored 10 months ago and David Douard committed 10 months ago

This is an alternative to modifying the `person` table, as that
technique only worked on the PostgreSQL backend and not with Cassandra,
as it does not have a `person` table (authors and committers are inlined
in the revision and release tables).

This does not come with cli tooling to manage the display name table
entries for now.

I am trying to use mypy to detect psycopg2 → spycopg3 difference and the
noise is getting in the way.

This let us types various function mypy was warning about.

mypy now understand that BaseModel object have a object_type.