David Douard authored 2 years ago and Nicolas Dandrimont committed 2 years ago

This command allowd to backfill a kafka journal from an existing
Postgresql provenance storage.

The command will run a given number of workers in parallel. The state of
the backfilling process is saved in a leveldb store, so interrupting and
restarting a backfilling process is possible, with limitations: it won't
work properly if the range generation is modified.

This allows to use the journal writing part independently from the
ProvenanceStorage proxy class, eg. for the backfiller mechanism.

This allows answering the earliest occurrence question in a single query
instead of having to go through a join of the revision table.

It seems very unlikely that we'll ever use it considering the efficiency
of zfs compression when running PostgreSQL on it.

For consistency with other modules

For consistency with other modules

This avoids flushing the producer on every message, which will be pretty
slow considering that we've increased the number of messages by a lot.

We forgot to do this change alongside the actual kafka schema change, oops.

Change the keys for relations to allow topics to be compacted.
Entity topics depend on NOT being compacted as outdated data can reach
the topic when multiple writers process the same content.

Update the replayer accordingly fixing a few behavioural bugs.

It prevents us from using the journal proxy as well.

We still need to pre-create the revision entities to avoid massive
add/add conflicts when using the sharded rabbitmq storage backend.

This reverts commit e1da37d4.

this speeds up tests execution significantly.

start the grpc server directly instead of starting the HTTP rpc server,
since this later is actually not used, and the url/port detection was
possibly misleading.

and rename test_utils.py as utils.py to prevent confusion about this
module not being a set of tests.

- pre-commit from 4.1.0 to 4.3.0,
- codespell from 2.2.1 to 2.2.2,
- black from 22.3.0 to 22.10.0 and
- flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).

As usual when kafka is involved in tests, provided new tests are a bit
slow...

instead of depending on the proper behavior of the user of the
ProvenanceStoragePostgresql user.

it's not used, and keeping it makes code unnecessarly complex.

the new ProvenanceStorageJournal is a proxy ProvenanceStorageInterface
that will push added objects in a swh-journal (typ. a kafka).

Journal messages are simple dicts with 2 keys: id (the sharding key) and
value (a serialiazable version of the argument of the xxx_add() method).

Use the 'kafka' pytest marker for all kafka-related tests (especially
used for tox, see tox.ini).

and fix all occurrences of the typo.

make them all accept a Dict[Sha1Git, xxx] as argument, ie:

- remove support for Iterable[bytes] in revision_add, and
- replace Iterable[bytes] by Dict[Sha1Git, bytes] for location_add

Currently, the sha1 of location path in location_add() is not really
used by any backend, so the computation of said hashed is a waste of
resource, but it makes the API of this interface much more consistent
which will be helpful for coming features (like kafka journal).

move ingestion related code for the 3 "layers" in an algos/ submodule.

- move everything (swh)archive related in a archive/ submodule
- move everything provenance storage related in a storage/ submodule
  (which remains a not ideal name, may be confusing with the general
  'storage == swh-storage' acceptance in swh)
- rename rabbitmq's backend from api/ to storage/rabbitmq
- spit interface.py in 3 parts (one for each part, ProvenanceInterface,
  ProvenanceStorageInterface and ArchiveInterface).

so that one can easily run tests for the revision or origin
layer only.

'python' may be Python 2, according to
https://peps.python.org/pep-0394/#for-python-runtime-distributors

This allows us to ignore git bombs and other suspiciously large repos.

replace the (deprecated) HTTP RCP API to access the swh-graph service,
in favor of the grpc server.

To be able to test the (now) grpc-based ArchiveGraph, compressed graph
datasets for all 3 common datasets (cmdbts2, out-of-order and with-merges)
have been generated and included in this revision.

this will be needed for testing grpc swh-graph archive backend.

When the visit_edges response is empty, swh.graph.client generates an
empty tuple, which can't be unpacked. Work around the issue.