Skip to content

Manage backfiller configuration in puppet

Currently there is a attempt to configure the backfiller via puppet via the role profile::swh::deploy::journal::backfill applied on storage1.staging

It should be adapted to:

  • use a read-only database/storage configuration (use the guest user)
  • drop storage's journal_writer configuration key
  • use a remote objstorage or even a memory objstorage (backfill does not use at all the objstorage)
  • ~~ perhaps adapt the client id to swh.backfiller.something ~~ (already done)
  • create a logger.yml file like in migrated/migration$924
root@storage1:/etc/softwareheritage/journal# cat backfill.yml 
---
storage:
  cls: local
  args:
    db: host=db1.internal.staging.swh.network port=5432 user=swh dbname=swh password=xxxxxxx
    objstorage:
      cls: pathslicing
      args:
        root: "/srv/softwareheritage/objects"
        slicing: 0:1/1:5
    journal_writer:
      cls: kafka
      args:
        brokers:
        - journal0.internal.staging.swh.network
        prefix: swh.journal.objects
        client_id: swh.storage.journal_writer.storage1
        anonymize: true
        producer_config:
          message.max.bytes: 1000000000
journal_writer:
  cls: kafka
  args:
    brokers:
    - journal0.internal.staging.swh.network
    prefix: swh.journal.objects
    client_id: swh.storage.journal_writer.storage1
    anonymize: true
    producer_config:
      message.max.bytes: 1000000000

Migrated from T3009 (view on Phabricator)

Edited by Antoine R. Dumont