Skip to content

Add metrics for swh-scrubber using the sql exporter

Nicolas Dandrimont requested to merge mr/swh-scrubber-metrics into production

Thanks to @vlorentz for providing the initial query.

Ref. swh/devel/swh-scrubber#4684 (closed)

octocatalog-diff (patched with https://github.com/github/octocatalog-diff/issues/261#issuecomment-1334980032) output:

belvedere and db1.staging
Found host belvedere.internal.softwareheritage.org
Found host db1.internal.staging.swh.network
Cloning into '/tmp/swh-ocd.zVnjkR15/swh-site'...
done.
Switched to a new branch 'mr/swh-scrubber-metrics'
branch 'mr/swh-scrubber-metrics' set up to track 'origin/mr/swh-scrubber-metrics'.
Switched to a new branch 'mr/puppet-7-compat'
branch 'mr/puppet-7-compat' set up to track 'origin/mr/puppet-7-compat'.
Switched to branch 'production'
Your branch is up to date with 'origin/production'.
[production 63ac75e4] WIP: Update Puppetfile for compatibility with Puppet 7
 Date: Mon Nov 28 14:17:01 2022 +0100
 1 file changed, 7 insertions(+), 1 deletion(-)
Switched to branch 'mr/swh-scrubber-metrics'
Your branch is up to date with 'origin/mr/swh-scrubber-metrics'.
[mr/swh-scrubber-metrics b0b5f9d8] WIP: Update Puppetfile for compatibility with Puppet 7
 Date: Mon Nov 28 14:17:01 2022 +0100
 1 file changed, 7 insertions(+), 1 deletion(-)
WARN	 -> Branch mr/puppet-7-compat filtered out by ignore_branch_prefixes ["mr/puppet-7-compat"]
WARN	 -> Environment "mr/swh-scrubber-metrics" contained non-word characters, correcting name to mr_swh_scrubber_metrics
WARN	 -> Branch mr/puppet-7-compat filtered out by ignore_branch_prefixes ["mr/puppet-7-compat"]
WARN	 -> Environment "mr/swh-scrubber-metrics" contained non-word characters, correcting name to mr_swh_scrubber_metrics
Cloning into '/tmp/swh-ocd.zVnjkR15/environments/production/data/private'...
done.
Cloning into '/tmp/swh-ocd.zVnjkR15/environments/mr_swh_scrubber_metrics/data/private'...
done.
*** Running octocatalog-diff on host belvedere.internal.softwareheritage.org
I, [2023-03-28T17:00:57.513543 #1506303]  INFO -- : Catalogs compiled for belvedere.internal.softwareheritage.org
I, [2023-03-28T17:00:57.753757 #1506303]  INFO -- : Diffs computed for belvedere.internal.softwareheritage.org
diff origin/production/belvedere.internal.softwareheritage.org current/belvedere.internal.softwareheritage.org
*******************************************
+ File[/etc/prometheus-sql-exporter/swh-scrubber.yml] =>
   parameters =>
      "ensure": "present"
      "group": "root"
      "mode": "0644"
      "notify": "Service[prometheus-sql-exporter]"
      "owner": "root"
      "content": >>>
- name: swh_scrubber_coverage
  scope: database
  database: ^(swh|softwareheritage)-scrubber$
  interval: '1h'
  help: "Software Heritage Scheduler scrubber coverage"
  query: |
    SELECT
        datastore.package AS datastore_package,
        datastore.class AS datastore_class,
        datastore.instance AS datastore_instance,
        checked_partition.object_type AS object_type,
        checked_partition.nb_partitions AS partitions_total,
        COUNT(*) AS partitions_checked_total,
        EXTRACT (EPOCH FROM MIN(last_date)) AS oldest_partition_check_timestamp_seconds,
        EXTRACT (EPOCH FROM MAX(last_date)) AS newest_partition_check_timestamp_seconds
    FROM checked_partition
    INNER JOIN datastore ON (datastore.id=checked_partition.datastore)
    GROUP BY 1,2,3,4,5;
  labels:
    - datastore_package
    - datastore_class
    - datastore_instance
    - object_type
  values:
    - partitions_total
    - partitions_checked_total
    - oldest_partition_check_timestamp_seconds
    - newest_partition_check_timestamp_seconds
<<<
*******************************************
  Profile::Prometheus::Export_scrape_config[sql] =>
   parameters =>
     metric_relabel_configs =>
      - [{"source_labels"=>["__name__", "col"], "regex"=>"sql_swh_scheduler_origins;(.*)", "action"=>"replace", "target_label"=>"__name__", "replacement"=>"swh_scheduler_origins_${1}"}]
      + [{"source_labels"=>["__name__", "col"], "regex"=>"sql_swh_scheduler_origins;(.*)", "action"=>"replace", "target_label"=>"__name__", "replacement"=>"swh_scheduler_origins_${1}"}, {"source_labels"=>["__name__", "col"], "regex"=>"sql_swh_scrubber_coverage;(.*)", "action"=>"replace", "target_label"=>"__name__", "replacement"=>"swh_scrubber_coverage_${1}"}]
*******************************************
*** End octocatalog-diff on belvedere.internal.softwareheritage.org
*** Running octocatalog-diff on host db1.internal.staging.swh.network
I, [2023-03-28T17:01:02.390662 #1506402]  INFO -- : Catalogs compiled for db1.internal.staging.swh.network
I, [2023-03-28T17:01:02.738085 #1506402]  INFO -- : Diffs computed for db1.internal.staging.swh.network
diff origin/production/db1.internal.staging.swh.network current/db1.internal.staging.swh.network
*******************************************
+ File[/etc/prometheus-sql-exporter/swh-scrubber.yml] =>
   parameters =>
      "ensure": "present"
      "group": "root"
      "mode": "0644"
      "notify": "Service[prometheus-sql-exporter]"
      "owner": "root"
      "content": >>>
- name: swh_scrubber_coverage
  scope: database
  database: ^(swh|softwareheritage)-scrubber$
  interval: '1h'
  help: "Software Heritage Scheduler scrubber coverage"
  query: |
    SELECT
        datastore.package AS datastore_package,
        datastore.class AS datastore_class,
        datastore.instance AS datastore_instance,
        checked_partition.object_type AS object_type,
        checked_partition.nb_partitions AS partitions_total,
        COUNT(*) AS partitions_checked_total,
        EXTRACT (EPOCH FROM MIN(last_date)) AS oldest_partition_check_timestamp_seconds,
        EXTRACT (EPOCH FROM MAX(last_date)) AS newest_partition_check_timestamp_seconds
    FROM checked_partition
    INNER JOIN datastore ON (datastore.id=checked_partition.datastore)
    GROUP BY 1,2,3,4,5;
  labels:
    - datastore_package
    - datastore_class
    - datastore_instance
    - object_type
  values:
    - partitions_total
    - partitions_checked_total
    - oldest_partition_check_timestamp_seconds
    - newest_partition_check_timestamp_seconds
<<<
*******************************************
  Profile::Prometheus::Export_scrape_config[sql] =>
   parameters =>
     metric_relabel_configs =>
      - [{"source_labels"=>["__name__", "col"], "regex"=>"sql_swh_scheduler_origins;(.*)", "action"=>"replace", "target_label"=>"__name__", "replacement"=>"swh_scheduler_origins_${1}"}]
      + [{"source_labels"=>["__name__", "col"], "regex"=>"sql_swh_scheduler_origins;(.*)", "action"=>"replace", "target_label"=>"__name__", "replacement"=>"swh_scheduler_origins_${1}"}, {"source_labels"=>["__name__", "col"], "regex"=>"sql_swh_scrubber_coverage;(.*)", "action"=>"replace", "target_label"=>"__name__", "replacement"=>"swh_scrubber_coverage_${1}"}]
*******************************************
*** End octocatalog-diff on db1.internal.staging.swh.network
Edited by Antoine R. Dumont

Merge request reports