This flag reset the partitions identified as stalled by setting
start_date and end_date to NULL.

This should put these reset partition to be selected for checking by a
scrubber worker.

For a given configuration (hence sotrage, object_type and partition scheme)
list partitions that have a start_date but no end_date for a long enough
time.

By default, it will compute the delay for a partition to be considered as
stalled based on the 10 last partitions checked for the given
configuration.

A checker configuration must now be created before being
able to start a checker session. This configuration is stored in the
database and consist in a triplet

  (datastore, object_type, nb_partitions)

Once done, any number of checker can be started for this specific
checker configuration; each checher process will check partitions
one by one, using the status stored in the database to get the next
partition number to check on the next iteration.

This allows to dynamically adapt the number of checker processes.

For example, checking the shapshots splitting the hash space in 4096
partitions using 4 parallel workers could be like:

  $ export SWH_CONFIG_FILENAME=config.yml
  $ swh scrubber check init --object-type snapshot --nb-partitions 4096 --name cfg-snp
  Created configuration cfg-snp [3] for checking shapshot in postgresql storage

  $ for i in {1..4}; do (swh scrubber check storage cfg-snp &); done

This new table stores the "configuration" for a scrubber. A
configuration consists in a set of:

  (datastore, object_type, nb_partitions)

This comes with a migration script; WARNING: this script needs to be
checked before deployment on a productiion-sized big database. Any
activity on the database should be stopped before execution.

This is the first step of a series to make the scrubber easier to deploy
on elastic infrastructure.

It now needs types-click which is indeed a dependency of
swh.core[testing].

This allows overriding the JAVA_HOME to run cassandra with a different
java version (which also happens to be needed in CI, as we force usage
of an old java for cassandra through that envvar).

This avoids reinstalling tox all the time

vlorentz authored 1 year ago and vlorentz committed 1 year ago

and report it as corrupt instead.

It makes more sense to query a range of partition ids with a fixed nb_partition
than a range of nb_partitions with a fix partition id

No migration because the next release will need to scrap the whole table
anyway.

GitLab will display the content of the README file when browsing the
repository. But in case the file is a symlink, it will display the
path pointed by the symlink. There is a 6 year old issue about this:
https://gitlab.com/gitlab-org/gitlab/-/issues/15093

We can workaround the issue by having the content at the root of the
repository and a symlink to this file in the `docs/` directory.

Tested in swh-py-template!27

Related to swh/meta#4960

Related to swh/meta#4959

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

This should avoid pointless reports to Sentry and Icinga on AdminShutdown
exceptions.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Related to T4496

- pre-commit from 4.1.0 to 4.3.0,
- codespell from 2.2.1 to 2.2.2,
- black from 22.3.0 to 22.10.0 and
- flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).

For now this is a naive implementation, which does never rechecks.

For now, this does not use this information to deduplicate work.

It will be used by the storage_checker to 'remember' what ranges
it already checked recently across runs (and crashes), and to
monitor progress.

It will probably be useful to know what part of the check
takes the most time.

This will allow adding more tags easily in future commits

instead of (soon-to-be-deprecated) swh-core's postgresql_fact one.

This alleviates current locking issues where nothing gets written in production.

Related to T4228

Related to T4228

so migration tool can be used

Related to T4284