Skip to content

docker-compose: Add journal stack

Antoine R. Dumont requested to merge generated-differential-D860-source into generated-differential-D860-target

Add:

  • journal (through kafka/zookeeper instance)
  • swh-storage-listener
  • swh-journal-publisher
  • swh-journal-client (dedicated client for the occasion)

Scenario is:

  • sql triggers on newly inserted objects (swh db) events
  • listener picked up those events, published those on the journal on temporary topics
  • publisher who is subscribed to the listener's topics, picked up those events, reified those associated objects, published them to the journal on new public topics
  • client who is subscribed on the origin/origin-visit public topics picked those up and logged them

Depends on swh-storage!138 (closed) Depends on swh-journal!6 (closed)

Related swh-environment#1443 (closed)

Test Plan

docker-compose up

Load a repository, we should see logs about swh-journal-client logging events received (origin, origin-visit).

2018-12-20 13:20:35,344 1 INFO Setting newly assigned partitions {TopicPartition(topic='swh.journal.objects.origin_visit', partition=0), TopicPartition(topic='swh.journal.objects.origin', partition=0)} for group swh.journal.client
2018-12-20 13:20:35,404 1 INFO client received the following messages: defaultdict(<class 'list'>, {'origin_visit': [{b'date': b'2018-12-20 13:03:46.600807+00:00', b'metadata': None, b'origin': 2, b'snapshot': None, b'status': b'ongoing', b'visit': 4}, {b'date': b'2018-12-20 13:03:46.600807+00:00', b'metadata': None, b'origin': 2, b'snapshot': b'|4\x99n(\x8d\x98\xae\xa4\xed\xb2\x9eg\xaeT\x1a\xb2\x8b\xac\xd8', b'status': b'full', b'visit': 4}]})
2018-12-20 13:20:49,681 1 INFO client received the following messages: defaultdict(<class 'list'>, {'origin_visit': [{b'date': b'2018-12-20 13:20:33.234323+00:00', b'metadata': None, b'origin': 2, b'snapshot': None, b'status': b'ongoing', b'visit': 5}, {b'date': b'2018-12-20 13:20:33.234323+00:00', b'metadata': None, b'origin': 2, b'snapshot': b'|4\x99n(\x8d\x98\xae\xa4\xed\xb2\x9eg\xaeT\x1a\xb2\x8b\xac\xd8', b'status': b'full', b'visit': 5}]})

Migrated from D860 (view on Phabricator)

Merge request reports