Deploy elasticsearch indexes cleaner
These modifications will deploy curator on logstash0:
- Create a virtual environment and install
elasticsearch-curator; - Deploy curator configuration file;
- Deploy curator delete_indices script;
- Deploy a cron task to execute curator periodically (once a day);
- Ensure curator logs directory exists;
- Deploy a logrotate file (weekly rotation with 2 weeks of retention).
delete_indices script is split into 2 actions (long and short indices life cycles).
~~helm~~ octo-diff
diff origin/production/logstash0.internal.softwareheritage.org current/logstash0.internal.softwareheritage.org
*******************************************
+ Concat_fragment[profile::cron::elasticsearch-delete-indices] =>
parameters =>
"content": "# Cron snippet elasticsearch-delete-indices\n34 20 * * * root ch...
"order": "10",
"tag": "profile::cron::elasticsearch",
"target": "profile::cron::elasticsearch"
*******************************************
+ Exec[create curator venv] =>
parameters =>
"command": "/usr/bin/python3 -m venv /opt/curatorVenv",
"creates": "/opt/curatorVenv"
*******************************************
+ Exec[pip install elasticsearch-curator] =>
parameters =>
"command": "/opt/curatorVenv/bin/pip install --upgrade elasticsearch-curator...
"unless": "/bin/sh -c \"/opt/curatorVenv/bin/pip list | grep -qF elasticsear...
*******************************************
+ File[/etc/default/curator_action_delete_indices] =>
parameters =>
"content": "---\n# File managed by puppet, changes will be lost.\nactions:\n...
"ensure": "present",
"group": "root",
"mode": "0640",
"owner": "root"
*******************************************
+ File[/etc/default/curator_config.yaml] =>
parameters =>
"content": "elasticsearch:\n client:\n hosts:\n - http://esnode...
"ensure": "present",
"group": "root",
"mode": "0640",
"owner": "root"
*******************************************
+ File[/etc/logrotate.d/curator] =>
parameters =>
"content": "/var/log/curator/*.log {\n weekly\n missingok\n rotate ...
"ensure": "present",
"group": "root",
"mode": "0640",
"owner": "root"
*******************************************
+ File[/var/log/curator] =>
parameters =>
"ensure": "directory",
"group": "root",
"mode": "0755",
"owner": "root",
"purge": true,
"recurse": true
*******************************************
+ Package[python3-venv] =>
parameters =>
"ensure": "installed"
*******************************************
+ Profile::Cron::D[elasticsearch-delete-indices] =>
parameters =>
"command": "chronic /opt/curatorVenv/bin/curator --config /etc/default/curat...
"hour": "fqdn_rand",
"minute": "fqdn_rand",
"target": "elasticsearch",
"unique_tag": "elasticsearch-delete-indices",
"user": "root"
*******************************************
*** End octocatalog-diff on logstash0.internal.softwareheritage.orgcurator tests
Virtual env and files:
root@logstash0:~# tree -L 1 -hDp curator/
[drwxr-xr-x 4.0K Nov 20 14:43] curator/
|-- [drwxr-xr-x 4.0K Nov 19 14:42] Venv
|-- [-rw-r--r-- 752 Nov 19 15:20] action_close
|-- [-rw-r--r-- 1.7K Nov 20 14:07] action_delete_indices
|-- [-rw-r--r-- 873 Nov 19 15:21] action_open
|-- [-rw-r--r-- 12K Nov 20 14:08] curator-week-47.log
`-- [-rw-r--r-- 596 Nov 19 15:51] curator_config.yaml
2 directories, 5 filesDry-run execution:
(Venv) root@logstash0:~# curator --config curator/curator_config.yaml curator/action_delete_indices --dry-run
2024-11-20 14:44:17,921 INFO Preparing Action ID: 1, "delete_indices"
2024-11-20 14:44:17,921 INFO Creating client object and testing connection
2024-11-20 14:44:17,950 INFO Trying Action ID: 1, "delete_indices":
Long Lifecycle Indices.
Delete indices older than 365 days (based on index name), for
- systemlogs
- production-system-logs
- production-swh-logs
- swh_workers
- apache_logs
prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2024-11-20 14:44:53,323 INFO DRY-RUN MODE. No changes will be made.
2024-11-20 14:44:53,323 INFO (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2024-11-20 14:44:53,323 INFO DRY-RUN: delete_indices: apache_logs-2023.11.16 with arguments: {}
2024-11-20 14:44:53,323 INFO DRY-RUN: delete_indices: apache_logs-2023.11.17 with arguments: {}
2024-11-20 14:44:53,323 INFO DRY-RUN: delete_indices: apache_logs-2023.11.18 with arguments: {}
2024-11-20 14:44:53,323 INFO DRY-RUN: delete_indices: apache_logs-2023.11.19 with arguments: {}
2024-11-20 14:44:53,323 INFO DRY-RUN: delete_indices: apache_logs-2023.11.20 with arguments: {}
2024-11-20 14:44:53,323 INFO DRY-RUN: delete_indices: apache_logs-2023.11.21 with arguments: {}
2024-11-20 14:44:53,323 INFO DRY-RUN: delete_indices: swh_workers-7.15.2-2023.11.21 with arguments: {}
2024-11-20 14:44:53,323 INFO DRY-RUN: delete_indices: systemlogs-2023.11.16 with arguments: {}
2024-11-20 14:44:53,323 INFO DRY-RUN: delete_indices: systemlogs-2023.11.17 with arguments: {}
2024-11-20 14:44:53,323 INFO DRY-RUN: delete_indices: systemlogs-2023.11.18 with arguments: {}
2024-11-20 14:44:53,323 INFO DRY-RUN: delete_indices: systemlogs-2023.11.19 with arguments: {}
2024-11-20 14:44:53,324 INFO DRY-RUN: delete_indices: systemlogs-2023.11.20 with arguments: {}
2024-11-20 14:44:53,324 INFO DRY-RUN: delete_indices: systemlogs-2023.11.21 with arguments: {}
2024-11-20 14:44:53,324 INFO DRY-RUN: delete_indices: systemlogs-7.15.2-2023.11.16 with arguments: {}
2024-11-20 14:44:53,324 INFO DRY-RUN: delete_indices: systemlogs-7.15.2-2023.11.17 with arguments: {}
2024-11-20 14:44:53,324 INFO DRY-RUN: delete_indices: systemlogs-7.15.2-2023.11.18 with arguments: {}
2024-11-20 14:44:53,324 INFO DRY-RUN: delete_indices: systemlogs-7.15.2-2023.11.19 with arguments: {}
2024-11-20 14:44:53,324 INFO DRY-RUN: delete_indices: systemlogs-7.15.2-2023.11.20 with arguments: {}
2024-11-20 14:44:53,324 INFO DRY-RUN: delete_indices: systemlogs-7.15.2-2023.11.21 with arguments: {}
2024-11-20 14:44:53,324 INFO Action ID: 1, "delete_indices" completed.
2024-11-20 14:44:53,324 INFO Preparing Action ID: 2, "delete_indices"
2024-11-20 14:44:53,324 INFO Creating client object and testing connection
2024-11-20 14:44:53,336 INFO Trying Action ID: 2, "delete_indices":
Short Lifecycle Indices.
Delete indices older than 182 days (based on index name), for
- staging-system-logs
- staging-swh-logs
prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2024-11-20 14:45:07,230 INFO DRY-RUN MODE. No changes will be made.
2024-11-20 14:45:07,230 INFO (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2024-11-20 14:45:07,230 INFO Action ID: 2, "delete_indices" completed.
2024-11-20 14:45:07,230 INFO All actions completed.delete_indices action script:
(Venv) root@logstash0:~# cat curator/action_delete_indices
---
actions:
1:
action: delete_indices
description: >-
Long Lifecycle Indices.
Delete indices older than 365 days (based on index name), for
- systemlogs
- production-system-logs
- production-swh-logs
- swh_workers
- apache_logs
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
disable_action: False
filters:
- filtertype: pattern
kind: regex
value: '^(systemlogs|production-system-logs|production-swh-logs|swh_workers|apache_logs).*$'
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 365
2:
action: delete_indices
description: >-
Short Lifecycle Indices.
Delete indices older than 182 days (based on index name), for
- staging-system-logs
- staging-swh-logs
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
disable_action: False
filters:
- filtertype: pattern
kind: regex
value: '^(staging-system-logs|staging-swh-logs).*$'
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 182I didn't know where to put these indexes (in both lists they will be deleted):
old(?) indexes
(Venv) root@logstash0:~# curl -s http://esnode1.internal.softwareheritage.org:9200/_cat/indices | sort | \
> grep -v 'systemlogs\|staging-system-logs\|staging-swh-logs\|production-system-logs\|production-swh-logs\|swh_workers\|apache_logs\|\.[a-zA-Z]*'
green close reindexed-v7-swh-tasks-2016-09 t08BZfB3TzOD8b-U7WqcdQ 1 1
green close reindexed-v7-swh-tasks-2016-10 LIrVBSGoQyKSM8_9I5FPbA 1 1
green close reindexed-v7-swh-tasks-2016-11 K2_-r5EFTZCXurJSd8ViSw 1 1
green close reindexed-v7-swh-tasks-2016-12 _naPXx4qTOyYOIZV5-g2TQ 1 1
green close reindexed-v7-swh-tasks-2017-05 aox4Y0OcSsq4XeG3FV2cWw 1 1
green close reindexed-v7-swh-tasks-2017-06 x67dpZeHSdeqppzmaOB1bw 1 1
green close reindexed-v7-swh-tasks-2017-07 UGjsyCFaRQSPgB6zzjGAiA 1 1
green close reindexed-v7-swh-tasks-2017-09 Iiv7Z7rPRBSyqASMfBbD8A 1 1
green close reindexed-v7-swh-tasks-2017-10 vd-d9Tj-QE-edKZj68rdug 1 1
green close reindexed-v7-swh-tasks-2017-11 Tg_Lq5FySxaVBrk9wrxE3w 1 1
green close reindexed-v7-swh-tasks-2017-12 nt-hXOYKTHOCWi7asA7cjg 1 1
green close reindexed-v7-swh-tasks-2018-01 CaD4J9m5Skmdpmhu-aXj8g 1 1
green close reindexed-v7-swh-tasks-2018-02 2HN0o6-FTTaZKSWS1B91TA 1 1
green close reindexed-v7-swh-tasks-2018-03 slUPKPvfS1WtYiopzJh9GQ 1 1
green close reindexed-v7-swh-tasks-2018-04 dwK4yrUeS3iauzByNPM3CA 1 1
green close reindexed-v7-swh-tasks-2018-05 TuHsamp4QuC9z2TJx3muDQ 1 1
green close reindexed-v7-swh-tasks-2018-06 1dB5vCnbTVaoU0j2UmzrsA 1 1
green close reindexed-v7-swh-tasks-2018-07 zAPaZezYQxSGSFAv6K93HA 1 1
green close reindexed-v7-swh-tasks-2018-08 7lUYmukrSciERmrtwdkxBw 1 1
green close reindexed-v7-swh-tasks-2018-09 H1tKCN7ZS5az3tFYLjKwsQ 1 1
green close reindexed-v7-swh-tasks-2018-10 XBO-ixHQTcSB-u5ulgeMDA 1 1
green close reindexed-v7-swh-tasks-2018-11 eZUmhC2LS6-oJlI8gnjbnA 1 1
green close reindexed-v7-swh-tasks-2018-12 KV7OGsbvQGm3JB9k7q-a3Q 1 1
green close reindexed-v7-swh-tasks-2019-01 S283iNFITpiW12PIFlopeQ 1 1
green close reindexed-v7-swh-tasks-2019-04 zcpAdNQmTAqxVE-QNilpxQ 1 1
green close reindexed-v7-swh-tasks-2019-05 r6TU4TJuQ2q4XgxjbEYp-A 1 1
green close reindexed-v7-swh-tasks-2019-06 R0nF2rV_Q_ukGooD5zcD7g 1 1
green close reindexed-v7-swh-tasks-2019-09 wMGykjz5S5CwMpgcldFN8g 1 1
green close swh-tasks-2017-08 2_sD8jukRvOKgIIKJsavOA 1 1
green open reindexed-v7-swh-tasks-2016-01 QmcM4JVBRaqvxAmBYFy2aQ 1 1 0 0 498b 249b 249b
green open reindexed-v7-swh-tasks-2019-02 -lBUKZh_SpWEvf1X4X5LEA 1 1 6262076 0 1gb 539mb 539mb
green open swh-tasks-2020-01 bNCCThTjRt22-asMW_7hHw 1 1 0 0 500b 250b 250b
green open swh-tasks-2020-02 4M9xzjudQZix-A8WAz6BmA 1 1 0 0 498b 249b 249b
green open swh-tasks-2020-03 rSYrWqQCTnuI9A763cXd4Q 1 1 0 0 498b 249b 249b
green open swh-tasks-2020-04 nW9FxeCdT0iLdHCT03zoEA 1 1 0 0 498b 249b 249b
green open swh-tasks-2020-05 g4AmltOaQQurJ4b8sRvx8g 1 1 0 0 498b 249b 249b
green open swh-tasks-2020-06 lBeluAOnRXaMx_jeX2LeXw 1 1 0 0 498b 249b 249b
green open swh-tasks-2020-08 I5c2cBguSpaEixA6-XifQA 1 1 0 0 498b 249b 249b
green open swh-tasks-2020-09 8I3m8OASSy-CIbrLwXvFiA 1 1 0 0 498b 249b 249bEdited by Antoine R. Dumont