The dict form of add_batch argument is being removed as it relies on a
single-hash object identifier.

It has been deprecated for ages now.

Update the scrubber db for better `swh db` compatibility and
update test_migration for swh.core 3.6 (acually 3.6.1 is needed).

Show the actual traceback that occured in the cli command, if any.

Ensure only partition whose scrubbing is running are returned.

Fixes #4704.

Promote use of `swh scrubber check run` instead of deprecated commands.

Add sections about object storage checker and journal checker.

Enable to configure and trigger the scrubbing of an object storage with
swh-scrubber CLI, either using partitions of contents provided by a
storage or by consuming the content kafka topic from a SWH journal
(in that case, the --use-journal flag must be provided to "swh check run"
command).

Related to #4694.

Instead of reinventing the wheel, prefer to use the check method from the
object storage interface for verifying content presence and integrity.

Related to #4694.

Add ObjectStorageCheckerFromJournal class to consume content ids from a kafka
topic in order to check their presence in a given object storage but also
to check their integrity by fetching their bytes and recomputing checksums.

Related to #4694.

Deprecate the former ones.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

The default behavior of Click is to rewrap text based on the width of
the terminal but as a consequence it makes the sample YAML config for
the scrubber displayed in command help quite unreadable as indentation
is lost.

So use \b markers in docstring to ensure proper display of the YAML
config by Click.

Add class ObjectStorageChecker to detect missing and corrupted contents
in an object storage.

It iterates on content objects referenced in a storage instance, check they
are available in a given object storage instance then retrieve their bytes
from it in order to recompute checksums and detect corruptions.

Related to #4694.

Promote use of on_eof parameter instead.

To simplify the future adding of an objstorage checker, extract common
code and features of current checkers in abstract base classes.

Related to #4694.

Remove use of --import-mode=importlib pytest option and use
new option consider_namespace_packages to fix tests execution
with latest pytest release.

The next partition to check, as returned by the
checked_partition_iter_next() iterator should never return a partition
number exceeding the max number of partitions in the config, nor should
it addd this in the database.

Related to swh/meta#5075.