Upgrade logging elasticsearch cluster to 8.15

The logging elasticsearch cluster has been pinned to version 7.15.2 forever.

We should upgrade it to the latest elasticsearch version.

Before doing the upgrade to the 8.x branch, an upgrade to the (latest version of the) 7.17 branch is mandatory.

https://www.elastic.co/guide/en/elastic-stack/master/upgrading-elastic-stack.html#prepare-to-upgrade

The 8.x branch doesn't look like it has breaking changes that would affect us, it's not like the logging cluster is doing anything fancy.

https://www.elastic.co/guide/en/elasticsearch/reference/8.15/migrating-8.0.html

Upgrade steps should be as follows:

• Rolling upgrade of elasticsearch cluster to 7.17.24 (including upgrade to Debian 12)
  • esnode1
  • esnode2
  • esnode3
  • esnode7
• Upgrade kibana to 7.17.24 (including upgrade to Debian 12)
• Upgrade logstash to 7.17.24 (including upgrade to Debian 12)
• Check upgrade assistant in kibana
• Rolling upgrade of elasticsearch cluster to 8.15.1
  • esnode1
  • esnode2
  • esnode3
  • esnode7
• Upgrade kibana to 8.15.1
• (bonus) upgrade logstash to 8.15.1

marked this issue as related to #5415 (closed)

changed title from Upgrade logging elasticsearch cluster to 8.16 to Upgrade logging elasticsearch cluster to 8.15

mentioned in commit swh/infra/puppet/puppet-swh-site@6defbeb5

mentioned in commit swh/infra/puppet/puppet-swh-site@59b452b9

marked the checklist item esnode1 as completed

esnode1 upgraded, waiting for shards to reallocate and the cluster to go green.

marked the checklist item esnode2 as completed

esnode2 upgraded, waiting for shards to reallocate and the cluster to go green (again).

marked the checklist item esnode3 as completed

esnode3 upgraded and cluster back to green.

marked the checklist item esnode7 as completed

esnode7 upgraded, cluster recovery in progress. kibana is being upgraded.

mentioned in commit swh/infra/puppet/puppet-swh-site@4c33647f

Now that kibana is upgraded, the 8.0 upgrade assistant is available.

There are a few old indexes that had been created before 7.x and need reindexing. I've triggered kibana to do such a reindex:

curl -X POST "kibana0.internal.softwareheritage.org:5601/api/upgrade_assistant/reindex/batch" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d'
{
  "indexNames": [
...
  ]
}'

marked the checklist item Upgrade kibana to 7.17.24 (including upgrade to Debian 12) as completed

marked the checklist item Check upgrade assistant in kibana as completed

mentioned in commit swh/infra/puppet/puppet-swh-site@310c7dd0

All pre-7.x indexes have been reindexed, except for an old origin index which dates back to the initial swh.search implementation in 2020 (and hasn't been used since 2021). This index has been removed.

The remaining issues before the upgrade to 8.x are:

• a couple of deprecation warnings for elasticsearch configuration entries that don't exist anymore
• the deprecation of the "freeze" lifecycle step (we don't use it anymore, so it's just a matter of calling _unfreeze on these old indexes)
• a baffling kibana deprecation for a setting I can't find anywhere: "Configuring the "container ID" field has been deprecated and will be removed in 8.0.0. This plugin is designed to work with ECS, and expects this field to have a value of container.id. It has a different value configured in Source Configuration: Default"

Unfreeze indices:

curl -s http://192.168.100.61:9200/_cat/indices?v\&h=i,sth | awk ' ($2=="true") { print $1 }' | while read idx; do 
  echo $idx
  curl -XPOST http://192.168.100.61:9200/$idx/_unfreeze
  curl -XPOST http://192.168.100.61:9200/$idx/_close
  echo
done

I'll upgrade logstash0 to 7.17.24 next.

mentioned in commit swh/infra/puppet/puppet-swh-site@b74cec0e

marked the checklist item Upgrade logstash to 7.17.24 (including upgrade to Debian 12) as completed

logstash has been upgraded. I've taken the opportunity to fix the mapping issue with the journald.custom.syslog_timestamp field.

There were a couple of issues when upgrading esnode1 to 8.15.1:

• the contents of /srv/elasticsearch/nodes/0 are being migrated to /srv/elasticsearch, so the mountpoint /srv/elasticsearch/nodes needs to be moved one level up.
• One of the internal indexes had not been reindexed since 6.x, so the node start failed (after a rollback to 7.17.25 and a migration, the start succeeded).
• The elasticsearch systemd unit is missing a setting for proper notification of service start (fixed in puppet)

The cluster is now recovering, and I'll do the other nodes next

marked the checklist item esnode1 as completed

esnode2 upgrade done, recovery in progress

esnode3 upgrade done, recovery in progress

marked the checklist item esnode2 as completed

marked the checklist item esnode3 as completed

esnode7 upgrade done, recovery in progress

marked the checklist item esnode7 as completed

Cluster upgrade to Elasticsearch 8.15.1 is complete.

marked the checklist item Upgrade kibana to 8.15.1 as completed

Kibana has been upgraded to 8.15.1.

There was a weird error message for a config setting that wasn't in kibana.yml. The error was due to a leftover /etc/systemd/system/kibana.service from the old package, removing it solved the error.

This can be closed, the logstash upgrade will be moved to its own ticket.

closed