Disk space alerts on elk nodes
There are some alerts on the disk space of elk nodes:
root@esnode2:~# df -hT /srv/elasticsearch/
Filesystem Type Size Used Avail Use% Mounted on
elasticsearch-data zfs 6.9T 6.8T 125G 99% /srv/elasticsearchroot@esnode3:~# df -hT /srv/elasticsearch/
Filesystem Type Size Used Avail Use% Mounted on
elasticsearch-data zfs 6.9T 6.5T 422G 95% /srv/elasticsearchroot@esnode1:~# curl -s http://192.168.100.61:9200/_cat/allocation?v\&s=node
shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent host ip node node.role
670 0 0.0 6.2tb 6.2tb 6.2tb 631.6gb 6.8tb 91 192.168.100.61 192.168.100.61 esnode1 cdfhilmrstw
648 0 0.0 6.7tb 6.7tb 6.7tb 124.5gb 6.8tb 98 192.168.100.62 192.168.100.62 esnode2 cdfhilmrstw
660 0 0.0 6.4tb 6.4tb 6.4tb 421.9gb 6.8tb 94 192.168.100.63 192.168.100.63 esnode3 cdfhilmrstw
663 0 0.0 6.3tb 6.3tb 6.3tb 7.4tb 13.8tb 46 192.168.100.64 192.168.100.64 esnode7 cdfhilmrstw
656 0 0.0 6.5tb 6.5tb 6.5tb 4.9tb 11.4tb 57 192.168.100.65 192.168.100.65 esnode8 cdfhilmrstw
681 0 0.0 6tb 6tb 6tb 5.4tb 11.4tb 52 192.168.100.66 192.168.100.66 esnode9 cdfhilmrstw@teams/sysadmin We should think about data-tiers.
Activity
added activity::MRO label
Deleting some old closed indices:
root@esnode1:~# curl -s http://192.168.100.61:9200/_cat/indices?v | awk '$3~/reindexed-v7-swh-tasks/{print $3}' | xargs -i{} -t curl -s -XDELETE "http://192.168.100.61:9200/{}" curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2017-05 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2017-06 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2017-07 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2017-09 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2017-10 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2017-11 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2017-12 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2019-01 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2019-06 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2019-07 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2019-08 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2019-09 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2019-02 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2019-03 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2019-04 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2019-05 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2019-10 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2019-11 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2019-12 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2016-09 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2016-01 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2016-12 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2016-11 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2016-10 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2018-01 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2018-02 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2018-03 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2018-04 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2018-05 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2018-06 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2018-07 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2018-08 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2018-09 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2018-10 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2018-11 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/reindexed-v7-swh-tasks-2018-12do not free up much space:
root@esnode1:~# curl -s http://192.168.100.61:9200/_cat/allocation?v\&s=node shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent host ip node node.role 656 13 0.0 6.2tb 6.2tb 6.2tb 630.9gb 6.8tb 91 192.168.100.61 192.168.100.61 esnode1 cdfhilmrstw 635 10 0.0 6.7tb 6.7tb 6.7tb 128.4gb 6.8tb 98 192.168.100.62 192.168.100.62 esnode2 cdfhilmrstw 647 7 0.0 6.4tb 6.4tb 6.4tb 427.3gb 6.8tb 93 192.168.100.63 192.168.100.63 esnode3 cdfhilmrstw 655 9 0.0 6.3tb 6.3tb 6.3tb 7.4tb 13.8tb 46 192.168.100.64 192.168.100.64 esnode7 cdfhilmrstw 645 11 0.0 6.5tb 6.5tb 6.5tb 4.9tb 11.4tb 57 192.168.100.65 192.168.100.65 esnode8 cdfhilmrstw 668 7 0.0 6tb 6tb 6tb 5.4tb 11.4tb 52 192.168.100.66 192.168.100.66 esnode9 cdfhilmrstwDeleting old closed indices:
root@esnode1:~# curl -s http://192.168.100.61:9200/_cat/indices?v | awk '$3~/staging-swh-logs-2024-08/{print $3}' | xargs -i{} -t curl -s -XDELETE "http://192.168.100.61:9200/{}" curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-08-16 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-08-17 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-08-18 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-08-19 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-08-20 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-08-21 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-08-22 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-08-23 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-08-24 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-08-25 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-08-28 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-08-29 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-08-30 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-08-31root@esnode1:~# curl -s http://192.168.100.61:9200/_cat/indices?v | awk '$3~/staging-swh-logs-2024-09/{print $3}' | xargs -i{} -t curl -s -XDELETE "http://192.168.100.61:9200/{}" curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-01 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-02 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-03 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-04 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-05 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-06 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-07 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-08 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-09 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-10 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-11 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-12 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-13 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-14 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-15 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-16 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-17 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-18 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-19 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-20 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-21 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-22 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-23 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-24 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-25 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-26 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-27 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-28 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-29 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/staging-swh-logs-2024-09-30root@esnode1:~# curl -s http://192.168.100.61:9200/_cat/allocation?v\&s=node shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent host ip node node.role 638 22 0.0 6.2tb 6.2tb 6.2tb 636.7gb 6.8tb 90 192.168.100.61 192.168.100.61 esnode1 cdfhilmrstw 622 23 0.0 6.7tb 6.7tb 6.7tb 128gb 6.8tb 98 192.168.100.62 192.168.100.62 esnode2 cdfhilmrstw 632 18 0.0 6.4tb 6.4tb 6.4tb 441.3gb 6.8tb 93 192.168.100.63 192.168.100.63 esnode3 cdfhilmrstw 646 22 0.0 6.3tb 6.3tb 6.3tb 7.4tb 13.8tb 46 192.168.100.64 192.168.100.64 esnode7 cdfhilmrstw 627 21 0.0 6.5tb 6.5tb 6.5tb 4.9tb 11.4tb 57 192.168.100.65 192.168.100.65 esnode8 cdfhilmrstw 653 20 0.0 6tb 6tb 6tb 5.4tb 11.4tb 52 192.168.100.66 192.168.100.66 esnode9 cdfhilmrstwroot@esnode1:~# curl -s http://192.168.100.61:9200/_cat/indices?v | awk '$3~/production-system-logs-2024-0[23]/{print $3}' | xargs -i{} -t curl -s -XDELETE "http://192.168.100.61:9200/{}" curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-01 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-02 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-03 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-04 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-05 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-06 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-07 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-08 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-09 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-10 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-11 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-12 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-13 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-14 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-15 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-16 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-17 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-18 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-19 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-20 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-21 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-22 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-23 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-24 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-25 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-26 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-27 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-28 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-29 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-31 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-03-30 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-02-25 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-02-26 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-02-28 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-02-29 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-system-logs-2024-02-27root@esnode1:~# curl -s http://192.168.100.61:9200/_cat/allocation?v\&s=node shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent host ip node node.role 627 34 0.0 6.1tb 6.1tb 6.1tb 770.1gb 6.8tb 89 192.168.100.61 192.168.100.61 esnode1 cdfhilmrstw 611 41 0.0 6.6tb 6.6tb 6.6tb 245.6gb 6.8tb 96 192.168.100.62 192.168.100.62 esnode2 cdfhilmrstw 623 30 0.0 6.3tb 6.3tb 6.3tb 578.9gb 6.8tb 91 192.168.100.63 192.168.100.63 esnode3 cdfhilmrstw 630 33 0.0 6.1tb 6.1tb 6.2tb 7.6tb 13.8tb 44 192.168.100.64 192.168.100.64 esnode7 cdfhilmrstw 614 39 0.0 6.3tb 6.4tb 6.4tb 5tb 11.4tb 55 192.168.100.65 192.168.100.65 esnode8 cdfhilmrstw 641 37 0.0 5.8tb 5.8tb 5.8tb 5.6tb 11.4tb 50 192.168.100.66 192.168.100.66 esnode9 cdfhilmrstwroot@esnode1:~# curl -s http://192.168.100.61:9200/_cat/indices?v | awk '$3~/production-swh-logs-2024-0[23]/{print $3}' | xargs -i{} -t curl -s -XDELETE "http://192.168.100.61:9200/{}" curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-02-25 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-02-26 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-02-27 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-02-28 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-02-29 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-01 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-02 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-03 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-04 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-05 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-06 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-08 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-09 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-07 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-10 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-11 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-12 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-13 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-14 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-15 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-16 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-17 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-18 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-19 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-20 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-21 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-22 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-23 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-24 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-25 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-26 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-27 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-28 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-29 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-30 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/production-swh-logs-2024-03-31root@esnode1:~# curl -s http://192.168.100.61:9200/_cat/allocation?v\&s=node shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent host ip node node.role 614 43 0.0 6tb 6tb 6tb 816gb 6.8tb 88 192.168.100.61 192.168.100.61 esnode1 cdfhilmrstw 600 45 0.0 6.5tb 6.5tb 6.5tb 300gb 6.8tb 95 192.168.100.62 192.168.100.62 esnode2 cdfhilmrstw 613 39 0.0 6.2tb 6.2tb 6.2tb 620.8gb 6.8tb 91 192.168.100.63 192.168.100.63 esnode3 cdfhilmrstw 618 43 0.0 6.1tb 6.1tb 6.1tb 7.6tb 13.8tb 44 192.168.100.64 192.168.100.64 esnode7 cdfhilmrstw 600 42 0.0 6.3tb 6.3tb 6.3tb 5.1tb 11.4tb 55 192.168.100.65 192.168.100.65 esnode8 cdfhilmrstw 629 41 0.0 5.8tb 5.8tb 5.8tb 5.6tb 11.4tb 50 192.168.100.66 192.168.100.66 esnode9 cdfhilmrstwroot@esnode1:~# curl -s http://192.168.100.61:9200/_cat/indices?v | awk '$3~/systemlogs-2024.0[23]/{print $3}' | xargs -i{} -t curl -s -XDELETE "http://192.168.100.61:9200/{}" curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.02 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.03 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.04 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.05 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.06 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.07 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.08 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.09 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.01 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.10 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.11 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.12 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.13 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.14 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.15 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.16 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.17 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.18 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.19 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.20 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.21 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.22 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.23 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.24 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.25 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.26 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.27 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.28 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.29 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.30 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.03.31 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.02.25 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.02.26 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.02.27 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.02.28 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-2024.02.29root@esnode1:~# curl -s http://192.168.100.61:9200/_cat/allocation?v\&s=node shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent host ip node node.role 603 47 0.0 6tb 6tb 6tb 816.5gb 6.8tb 88 192.168.100.61 192.168.100.61 esnode1 cdfhilmrstw 589 51 0.0 6.5tb 6.5tb 6.5tb 298.4gb 6.8tb 95 192.168.100.62 192.168.100.62 esnode2 cdfhilmrstw 600 44 0.0 6.2tb 6.2tb 6.2tb 617.1gb 6.8tb 91 192.168.100.63 192.168.100.63 esnode3 cdfhilmrstw 601 47 0.0 6.1tb 6.1tb 6.1tb 7.6tb 13.8tb 44 192.168.100.64 192.168.100.64 esnode7 cdfhilmrstw 591 49 0.0 6.3tb 6.3tb 6.3tb 5.1tb 11.4tb 55 192.168.100.65 192.168.100.65 esnode8 cdfhilmrstw 618 48 0.0 5.8tb 5.8tb 5.8tb 5.6tb 11.4tb 50 192.168.100.66 192.168.100.66 esnode9 cdfhilmrstwroot@esnode1:~# curl -s http://192.168.100.61:9200/_cat/indices?v | awk '$3~/systemlogs-7.15.2-2024.0[23]/{print $3}' | xargs -i{} -t curl -s -XDELETE "http://192.168.100.61:9200/{}" curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.01 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.02 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.03 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.04 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.05 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.06 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.07 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.08 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.09 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.10 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.11 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.12 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.13 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.14 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.15 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.16 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.17 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.18 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.19 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.23 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.22 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.25 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.24 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.21 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.20 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.27 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.26 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.29 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.28 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.30 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.03.31 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.02.29 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.02.26 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.02.25 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.02.28 {"acknowledged":true}curl -s -XDELETE http://192.168.100.61:9200/systemlogs-7.15.2-2024.02.27root@esnode1:~# curl -s http://192.168.100.61:9200/_cat/allocation?v\&s=node shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent host ip node node.role 594 57 0.0 6tb 6tb 6tb 871.7gb 6.8tb 87 192.168.100.61 192.168.100.61 esnode1 cdfhilmrstw 573 58 0.0 6.4tb 6.4tb 6.4tb 390.9gb 6.8tb 94 192.168.100.62 192.168.100.62 esnode2 cdfhilmrstw 587 50 0.0 6.1tb 6.1tb 6.1tb 745.2gb 6.8tb 89 192.168.100.63 192.168.100.63 esnode3 cdfhilmrstw 588 53 0.0 6tb 6tb 6tb 7.7tb 13.8tb 43 192.168.100.64 192.168.100.64 esnode7 cdfhilmrstw 580 58 0.0 6.2tb 6.2tb 6.2tb 5.2tb 11.4tb 54 192.168.100.65 192.168.100.65 esnode8 cdfhilmrstw 608 57 0.0 5.7tb 5.7tb 5.7tb 5.7tb 11.4tb 49 192.168.100.66 192.168.100.66 esnode9 cdfhilmrstwDisk space freed up:
gsamson@esnode2 ~ % df -hT /srv/elasticsearch Filesystem Type Size Used Avail Use% Mounted on elasticsearch-data zfs 6.9T 6.5T 391G 95% /srv/elasticsearchgsamson@esnode3 ~ % df -hT /srv/elasticsearch Filesystem Type Size Used Avail Use% Mounted on elasticsearch-data zfs 6.9T 6.2T 749G 90% /srv/elasticsearchEdited by Guillaume Samsonassigned to @guillaume
The watermark thresholds were reduced when we we were limit in disk space with only esnode1,2,3 and 7.
Now we have 2 another ones, the limits can be increased to ensure there is more free disk space.
To stay arount the limit od 10% (recommended for zfs), I changed the limits to
curl -XPUT -H "Content-Type: application/json" http://$ES_NODE/_cluster/settings -d '{ "persistent": { "cluster.routing.allocation.disk.watermark.low": "90%", "cluster.routing.allocation.disk.watermark.low.max_headroom": "700GB", "cluster.routing.allocation.disk.watermark.high": "92%", "cluster.routing.allocation.disk.watermark.high.max_headroom": "600GB", "cluster.routing.allocation.disk.watermark.flood_stage": "98%", "cluster.routing.allocation.disk.watermark.flood_stage.max_headroom": "5GB", "cluster.routing.allocation.disk.watermark.flood_stage.frozen": "98%", "cluster.routing.allocation.disk.watermark.flood_stage.frozen.max_headroom": "5GB" } }'There is a couple of reallacations are in progress but some space will be eventually released on esnode3 node:
[2025-03-04T14:16:28,816][INFO ][o.e.c.r.a.DiskThresholdMonitor] [esnode3] low disk watermark [max_headroom=700gb] exceeded on [9NK1YMQBTgms_6ud8V4X5g][esnode1][/srv/elasticsearch] free: 666.8gb[9.4%], replicas will not be assigned to this node [2025-03-04T14:16:28,816][INFO ][o.e.c.r.a.DiskThresholdMonitor] [esnode3] low disk watermark [max_headroom=700gb] exceeded on [L_nwvjevQ6qJsXU1ZVBbaw][esnode2][/srv/elasticsearch] free: 661.5gb[9.3%], replicas will not be assigned to this node [2025-03-04T14:16:59,784][WARN ][o.e.c.r.a.DiskThresholdMonitor] [esnode3] high disk watermark [92%] exceeded on [Oex7saSnQyOjLKSFDBYmDA][esnode3][/srv/elasticsearch] free: 190.3gb[2.7%], shards will be relocated away from this node; currently relocating away shards totalling [139701449625] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are completeThe cluster is still green:
% curl -s http://192.168.100.61:9200/_cat/health\?v epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent 1741097898 14:18:18 swh-logging-prod green 6 6 3606 1803 19 0 0 0 - 100.0%mentioned in commit swh/infra/puppet/puppet-swh-site@8ed04ddf
mentioned in merge request swh/infra/puppet/puppet-swh-site!728 (merged)
All the shards are not yet completely at there final place due to some network instabilities of esnode2. It should be solved when we'll switch to the optical network.
shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent host ip node node.role 578 0 0.0 6.1tb 6.1tb 6.1tb 762.2gb 6.8tb 89 192.168.100.61 192.168.100.61 esnode1 cdfhilmrstw 590 70 0.0 5.4tb 5.3tb 5.3tb 1.5tb 6.8tb 78 192.168.100.62 192.168.100.62 esnode2 cdfhilmrstw 516 0 0.0 6.2tb 6.2tb 6.2tb 611.3gb 6.8tb 91 192.168.100.63 192.168.100.63 esnode3 cdfhilmrstw 647 59 0.0 6.7tb 6.7tb 6.7tb 7tb 13.8tb 49 192.168.100.64 192.168.100.64 esnode7 cdfhilmrstw 634 52 0.0 7tb 7.1tb 7.1tb 4.3tb 11.4tb 61 192.168.100.65 192.168.100.65 esnode8 cdfhilmrstw 673 79 0.0 7tb 7tb 7tb 4.4tb 11.4tb 61 192.168.100.66 192.168.100.66 esnode9 cdfhilmrstwThe configuration should now avoid a node to have a too high disk consumption with almost enough free space globally to allow the lost of a complete server.
It will probably be a little short if we lose one of the esnode7-8-9 nodes but we could temporary play with the limits if it happens.
Please register or sign in to reply