Promote use of `swh scrubber check run` instead of deprecated commands.

Add sections about object storage checker and journal checker.

Enable to configure and trigger the scrubbing of an object storage with
swh-scrubber CLI, either using partitions of contents provided by a
storage or by consuming the content kafka topic from a SWH journal
(in that case, the --use-journal flag must be provided to "swh check run"
command).

Related to #4694.

Instead of reinventing the wheel, prefer to use the check method from the
object storage interface for verifying content presence and integrity.

Related to #4694.

Add ObjectStorageCheckerFromJournal class to consume content ids from a kafka
topic in order to check their presence in a given object storage but also
to check their integrity by fetching their bytes and recomputing checksums.

Related to #4694.

Deprecate the former ones.

Antoine Lambert authored 11 months ago and Antoine Lambert committed 11 months ago

The default behavior of Click is to rewrap text based on the width of
the terminal but as a consequence it makes the sample YAML config for
the scrubber displayed in command help quite unreadable as indentation
is lost.

So use \b markers in docstring to ensure proper display of the YAML
config by Click.

Add class ObjectStorageChecker to detect missing and corrupted contents
in an object storage.

It iterates on content objects referenced in a storage instance, check they
are available in a given object storage instance then retrieve their bytes
from it in order to recompute checksums and detect corruptions.

Related to #4694.

Promote use of on_eof parameter instead.

To simplify the future adding of an objstorage checker, extract common
code and features of current checkers in abstract base classes.

Related to #4694.

Remove use of --import-mode=importlib pytest option and use
new option consider_namespace_packages to fix tests execution
with latest pytest release.

The next partition to check, as returned by the
checked_partition_iter_next() iterator should never return a partition
number exceeding the max number of partitions in the config, nor should
it addd this in the database.

Related to swh/meta#5075.

It now requires a swh-graph server running or connection errors appear.

Use swh-graph NaiveClient to avoid spawning a real graph server during
the tests.

As well as a command to list partitions being checked.

For example:

```
$ swh scrubber check stats snapshot_16 -j
{
  "config": {
    "name": "snapshot_16",
    "datastore": {
      "package": "storage",
      "cls": "postgresql",
      "instance": "postgresql:///?service=swh-storage"
    },
    "object_type": "snapshot",
    "nb_partitions": 65536,
    "check_hashes": true,
    "check_references": true
  },
  "min_duration": 0.002196,
  "max_duration": 0.107398,
  "avg_duration": 0.005969,
  "checked_partition": 65536,
  "running_partition": 0,
  "missing_object": 0,
  "missing_object_reference": 0,
  "corrupt_object": 0
}

$ swh scrubber check running cfg1

Running partitions for cfg1 [id=1, type=snapshot]:
0:	running since today (20 minutes)

```

init` command

These flags allow to configure a checking session including only one of
the 2 possible checks (hash computation and reference validation).

Which allows to remove the dependency on types-pyyaml in [testing]
extra.

These tables used to reference the datastore the invalid/missing object
was found in, but not keeping the config entry, i.e. the checking session
during wich the invalid/missing object was found, which can be an issue
when more than one checking session is executed on a given datastore.

This replaces the `datastore` field of tables `corrupt_object`,
`missing_object` and `missing_object_reference` tables by `config_id`.

Adapt all the code accordingly.

Note that it changes a bit the cli usage: the kafka checker now needs
a config entry, thus a kafka checking session can ony target a given
object type (i.e. one kafka topic),

The migration script will fill the config_id column for corrupt_object
using the check_config entry that matches the oject_type (of
corrupt_object) and datastore. For missing_object and
missing_object_reference, it will use this later table to idenify the
check_config entry corresponding object type for the reference_id and
datastore, since it is a checking session on this object type that will
generate a missing object entry (which is generaaly not of the same
type). For the missing_object table, the config_id will use the one
extracted from the missing_object_reference (joining on the missing_id
column).

Note that the migration script will fail if there are rows in one
of these tables for which there exists more than one possible
config_entry (i.e. with the same object_type and datastore).

was missing the flake8-bugbear dependency, making effectively the
line-too-long check disabled.

Previously, in production, this would retrieve the configuration of the other backend as
those configurations are named the same.

Refs. #4696

To avoid returning only the first one when multiple configuration with the same name
exists for different backend to scrub.

Refs. #4696

It's popping up after having run tests.

This was found while deploying the new version.