Skip to content

elasticsearch/curator: Replace actions name filter with a timestamp filter

Guillaume Samson requested to merge curator_filters into production

The current curator actions use a name_based_ages filter with this timestring: %Y.%m.%d.
Some indices have name with this ending pattern: %Y-%m-%d. These indices were not closed and deleted.
These modifications will upgrade the curator actions to use a field_stats_based_ages.

octocatalog-diff 
diff origin/production/logstash0.internal.softwareheritage.org current/logstash0.internal.softwareheritage.org
*******************************************
  File[/etc/default/curator_action_close_indices] =>
   parameters =>
     content =>
      @@ -24,7 +24,8 @@
             value: '^(systemlogs|production-system-logs|production-swh-logs|swh_workers|apache_logs|staging-system-logs|staging-swh-logs).*$'
           - filtertype: age
      -      source: name
      +      source: field_stats
             direction: older
      -      timestring: '%Y.%m.%d'
             unit: days
             unit_count: 30
      +      field: '@timestamp'
      +      stats_result: min_value
*******************************************
  File[/etc/default/curator_action_delete_indices] =>
   parameters =>
     content =>
      @@ -24,9 +24,10 @@
             value: '^(systemlogs|production-system-logs|production-swh-logs|swh_workers|apache_logs).*$'
           - filtertype: age
      -      source: name
      +      source: field_stats
             direction: older
      -      timestring: '%Y.%m.%d'
             unit: days
             unit_count: 365
      +      field: '@timestamp'
      +      stats_result: min_value
         2:
           action: delete_indices
      @@ -48,7 +49,8 @@
             value: '^(staging-system-logs|staging-swh-logs).*$'
           - filtertype: age
      -      source: name
      +      source: field_stats
             direction: older
      -      timestring: '%Y.%m.%d'
             unit: days
             unit_count: 182
      +      field: '@timestamp'
      +      stats_result: min_value
*******************************************
*** End octocatalog-diff on logstash0.internal.softwareheritage.org
Delete test on logstash0 
root@logstash0:~# cat curator_action_delete_indices 
---
# File managed by puppet, changes will be lost.
actions:
  1:
    action: delete_indices
    description: >-

      Long Lifecycle Indices.

      Delete indices older than 365 days (based on index name), for
              - systemlogs
              - production-system-logs
              - production-swh-logs
              - swh_workers
              - apache_logs
            prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(systemlogs|production-system-logs|production-swh-logs|swh_workers|apache_logs).*$'
    - filtertype: age
      source: field_stats
      direction: older
      unit: days
      unit_count: 365
      field: '@timestamp'
      stats_result: min_value
  2:
    action: delete_indices
    description: >-

      Short Lifecycle Indices.

      Delete indices older than 182 days (based on index name), for
              - staging-system-logs
              - staging-swh-logs
            prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(staging-system-logs|staging-swh-logs).*$'
    - filtertype: age
      source: field_stats
      direction: older
      unit: days
      unit_count: 182
      field: '@timestamp'
      stats_result: min_value
root@logstash0:~# /opt/curatorVenv/bin/curator --config /etc/default/curator_config.yaml curator_action_delete_indices --dry-run
2024-12-03 13:23:29,878 INFO      Preparing Action ID: 1, "delete_indices"
2024-12-03 13:23:29,878 INFO      Creating client object and testing connection
2024-12-03 13:23:29,907 INFO      Trying Action ID: 1, "delete_indices": 
Long Lifecycle Indices.
Delete indices older than 365 days (based on index name), for
        - systemlogs
        - production-system-logs
        - production-swh-logs
        - swh_workers
        - apache_logs
      prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
2024-12-03 13:24:23,645 INFO      DRY-RUN MODE.  No changes will be made.
2024-12-03 13:24:23,646 INFO      (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2024-12-03 13:24:23,646 INFO      DRY-RUN: delete_indices: production-swh-logs with arguments: {}
2024-12-03 13:24:23,646 INFO      DRY-RUN: delete_indices: production-swh-logs- with arguments: {}
2024-12-03 13:24:23,646 INFO      DRY-RUN: delete_indices: production-system-logs- with arguments: {}
2024-12-03 13:24:23,646 INFO      Action ID: 1, "delete_indices" completed.
2024-12-03 13:24:23,647 INFO      Preparing Action ID: 2, "delete_indices"
2024-12-03 13:24:23,647 INFO      Creating client object and testing connection
2024-12-03 13:24:23,659 INFO      Trying Action ID: 2, "delete_indices": 
Short Lifecycle Indices.
Delete indices older than 182 days (based on index name), for
        - staging-system-logs
        - staging-swh-logs
      prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
2024-12-03 13:24:38,270 INFO      DRY-RUN MODE.  No changes will be made.
2024-12-03 13:24:38,270 INFO      (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2024-12-03 13:24:38,270 INFO      DRY-RUN: delete_indices: staging-swh-logs with arguments: {}
2024-12-03 13:24:38,270 INFO      DRY-RUN: delete_indices: staging-swh-logs- with arguments: {}
2024-12-03 13:24:38,270 INFO      DRY-RUN: delete_indices: staging-system-logs with arguments: {}
2024-12-03 13:24:38,270 INFO      DRY-RUN: delete_indices: staging-system-logs- with arguments: {}
2024-12-03 13:24:38,270 INFO      Action ID: 2, "delete_indices" completed.
2024-12-03 13:24:38,270 INFO      All actions completed.
Close test on logstash0 
root@logstash0:~# cat curator_action_close_indices 
---
# File managed by puppet, changes will be lost.
actions:
  1:
    action: close
    description: >-
      Close indices older than 30 days (based on index name), for
              - systemlogs
              - production-system-logs
              - production-swh-logs
              - swh_workers
              - apache_logs
              - staging-system-logs
              - staging-swh-logs
            prefixed indices.
    options:
      skip_flush: False
      delete_aliases: False
      disable_action: False
      ignore_empty_list: True
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(systemlogs|production-system-logs|production-swh-logs|swh_workers|apache_logs|staging-system-logs|staging-swh-logs).*$'
    - filtertype: age
      source: field_stats
      direction: older
      unit: days
      unit_count: 30
      field: '@timestamp'
      stats_result: min_value
root@logstash0:~# /opt/curatorVenv/bin/curator --config /etc/default/curator_config.yaml curator_action_close_indices --dry-run
2024-12-03 13:26:12,160 INFO      Preparing Action ID: 1, "close"
2024-12-03 13:26:12,160 INFO      Creating client object and testing connection
2024-12-03 13:26:12,176 INFO      Trying Action ID: 1, "close": Close indices older than 30 days (based on index name), for
        - systemlogs
        - production-system-logs
        - production-swh-logs
        - swh_workers
        - apache_logs
        - staging-system-logs
        - staging-swh-logs
      prefixed indices.
2024-12-03 13:27:19,430 INFO      DRY-RUN MODE.  No changes will be made.
2024-12-03 13:27:19,430 INFO      (CLOSED) indices may be shown that may not be acted on by action "close".
2024-12-03 13:27:19,430 INFO      DRY-RUN: close: production-swh-logs with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,430 INFO      DRY-RUN: close: production-swh-logs- with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,430 INFO      DRY-RUN: close: production-swh-logs-2024-10-26 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,430 INFO      DRY-RUN: close: production-swh-logs-2024-10-27 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,430 INFO      DRY-RUN: close: production-swh-logs-2024-10-28 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,430 INFO      DRY-RUN: close: production-swh-logs-2024-10-29 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,430 INFO      DRY-RUN: close: production-swh-logs-2024-10-30 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: production-swh-logs-2024-10-31 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: production-swh-logs-2024-11-01 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: production-swh-logs-2024-11-02 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: production-swh-logs-2024-11-03 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: production-system-logs- with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: production-system-logs-2024-10-26 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: production-system-logs-2024-10-27 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: production-system-logs-2024-10-28 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: production-system-logs-2024-10-29 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: production-system-logs-2024-10-30 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: production-system-logs-2024-10-31 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: production-system-logs-2024-11-01 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: production-system-logs-2024-11-02 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: production-system-logs-2024-11-03 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: staging-swh-logs with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: staging-swh-logs- with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: staging-swh-logs-2024-10-26 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: staging-swh-logs-2024-10-27 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: staging-swh-logs-2024-10-28 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: staging-swh-logs-2024-10-29 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: staging-swh-logs-2024-10-30 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: staging-swh-logs-2024-10-31 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,431 INFO      DRY-RUN: close: staging-swh-logs-2024-11-01 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,432 INFO      DRY-RUN: close: staging-swh-logs-2024-11-02 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,432 INFO      DRY-RUN: close: staging-swh-logs-2024-11-03 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,432 INFO      DRY-RUN: close: staging-system-logs with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,432 INFO      DRY-RUN: close: staging-system-logs- with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,432 INFO      DRY-RUN: close: staging-system-logs-2024-10-26 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,432 INFO      DRY-RUN: close: staging-system-logs-2024-10-27 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,432 INFO      DRY-RUN: close: staging-system-logs-2024-10-28 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,432 INFO      DRY-RUN: close: staging-system-logs-2024-10-29 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,432 INFO      DRY-RUN: close: staging-system-logs-2024-10-30 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,432 INFO      DRY-RUN: close: staging-system-logs-2024-10-31 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,432 INFO      DRY-RUN: close: staging-system-logs-2024-11-01 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,432 INFO      DRY-RUN: close: staging-system-logs-2024-11-02 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,432 INFO      DRY-RUN: close: staging-system-logs-2024-11-03 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,432 INFO      DRY-RUN: close: systemlogs-2024.11.03 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,432 INFO      DRY-RUN: close: systemlogs-7.17.23-2024.11.03 with arguments: {'delete_aliases': False}
2024-12-03 13:27:19,432 INFO      Action ID: 1, "close" completed.
2024-12-03 13:27:19,432 INFO      All actions completed.

Related to product-management/core-platform#106

Edited by Guillaume Samson

Merge request reports