Snippets Groups Projects

Forked from Platform / Development / swh-provenance

245 commits behind, 424 commits ahead of the upstream repository.

kafka: improve the message schema for the journal writer

David Douard authored 2 years ago

Change the keys for relations to allow topics to be compacted.
Entity topics depend on NOT being compacted as outdated data can reach
the topic when multiple writers process the same content.

Update the replayer accordingly fixing a few behavioural bugs.

869f9f16

869f9f16 2 years ago

Name	Last commit	Last update
docker
docs
sql/upgrades
swh
.dockerignore
.gitignore
.pre-commit-config.yaml
AUTHORS
CODE_OF_CONDUCT.md
CONTRIBUTORS
Dockerfile
LICENSE
MANIFEST.in
Makefile
README.md
conftest.py
docker-compose.yml
mypy.ini
pyproject.toml
pytest.ini
requirements-swh.txt
requirements-test.txt
requirements.txt
setup.cfg
setup.py
tox.ini

swh-provenance

Provenance DB module to query the provenance of source code artifacts present in the Software Heritage archive.

This project allows to build such a provenance db from the Software Heritage Archive, and query this database.

Building a provenance database

Building the provenance database requires a read access to the Software Heritage archive, either via a direct access to the database (preferred for better performances), or using the RPC API to a Software Heritage Storage instance.

It also need a postgresql database in which the provenance db will be written into.

A configuration file is needed with with the access to both these databases:

archive:
  cls: api
  storage:
      cls: remote
      url: http://uffizi.internal.softwareheritage.org:5002

provenance:
  cls: direct
  db:
    dbname: provenance
    host: localhost

Running in Docker

Build the image

docker build -t swh-provenance .

Run the services

docker-compose up -d
docker-compose logs -f