Skip to content
Snippets Groups Projects
  1. Dec 05, 2023
  2. Dec 03, 2023
  3. Nov 30, 2023
  4. Nov 24, 2023
  5. Oct 17, 2023
    • David Douard's avatar
      Add a cli command to get statistics for a given config entry · c0a4d44b
      David Douard authored
      As well as a command to list partitions being checked.
      
      For example:
      
      ```
      $ swh scrubber check stats snapshot_16 -j
      {
        "config": {
          "name": "snapshot_16",
          "datastore": {
            "package": "storage",
            "cls": "postgresql",
            "instance": "postgresql:///?service=swh-storage"
          },
          "object_type": "snapshot",
          "nb_partitions": 65536,
          "check_hashes": true,
          "check_references": true
        },
        "min_duration": 0.002196,
        "max_duration": 0.107398,
        "avg_duration": 0.005969,
        "checked_partition": 65536,
        "running_partition": 0,
        "missing_object": 0,
        "missing_object_reference": 0,
        "corrupt_object": 0
      }
      
      $ swh scrubber check running cfg1
      
      Running partitions for cfg1 [id=1, type=snapshot]:
      0:	running since today (20 minutes)
      
      ```
      c0a4d44b
  6. Oct 16, 2023
  7. Oct 12, 2023
    • David Douard's avatar
    • David Douard's avatar
      Add support for 2 check config flags in the check_config table · 566db2ac
      David Douard authored
      These flags allow to configure a checking session including only one of
      the 2 possible checks (hash computation and reference validation).
      566db2ac
    • David Douard's avatar
      run mypy with --install-types by default · c08c10b1
      David Douard authored
      Which allows to remove the dependency on types-pyyaml in [testing]
      extra.
      c08c10b1
    • David Douard's avatar
      Refactor data model to use config_id instead of datastore in xxx_object tables · bd8e324c
      David Douard authored
      These tables used to reference the datastore the invalid/missing object
      was found in, but not keeping the config entry, i.e. the checking session
      during wich the invalid/missing object was found, which can be an issue
      when more than one checking session is executed on a given datastore.
      
      This replaces the `datastore` field of tables `corrupt_object`,
      `missing_object` and `missing_object_reference` tables by `config_id`.
      
      Adapt all the code accordingly.
      
      Note that it changes a bit the cli usage: the kafka checker now needs
      a config entry, thus a kafka checking session can ony target a given
      object type (i.e. one kafka topic),
      
      The migration script will fill the config_id column for corrupt_object
      using the check_config entry that matches the oject_type (of
      corrupt_object) and datastore. For missing_object and
      missing_object_reference, it will use this later table to idenify the
      check_config entry corresponding object type for the reference_id and
      datastore, since it is a checking session on this object type that will
      generate a missing object entry (which is generaaly not of the same
      type). For the missing_object table, the config_id will use the one
      extracted from the missing_object_reference (joining on the missing_id
      column).
      
      Note that the migration script will fail if there are rows in one
      of these tables for which there exists more than one possible
      config_entry (i.e. with the same object_type and datastore).
      bd8e324c
  8. Sep 21, 2023
  9. Aug 24, 2023
  10. Jul 26, 2023
  11. Jul 24, 2023
  12. Jul 12, 2023
  13. Jul 10, 2023
Loading