objstorage_checker: Add support for consuming content ids from journal
Add ObjectStorageCheckerJournal class to consume content ids from a kafka topic in order to check their presence in a given object storage but also to check their integrity by fetching their bytes and recomputing checksums. Related to #4694.
Showing
- swh/scrubber/base_checker.py 12 additions, 11 deletionsswh/scrubber/base_checker.py
- swh/scrubber/journal_checker.py 19 additions, 18 deletionsswh/scrubber/journal_checker.py
- swh/scrubber/objstorage_checker.py 110 additions, 35 deletionsswh/scrubber/objstorage_checker.py
- swh/scrubber/tests/conftest.py 25 additions, 0 deletionsswh/scrubber/tests/conftest.py
- swh/scrubber/tests/objstorage_checker_tests.py 128 additions, 21 deletionsswh/scrubber/tests/objstorage_checker_tests.py
- swh/scrubber/tests/test_journal_kafka.py 35 additions, 78 deletionsswh/scrubber/tests/test_journal_kafka.py
Loading
Please register or sign in to comment