Puppetize elasticsearch nodes

This allows declaration of the elasticsearch which is configured manually so far. We reused the actual production configuration in /etc/elasticsearch/{elasticsearch.yml,jvm_options} as default.

The following diff configures both for production and staging node the following:

• /etc/elasticsearch/elasticsearch.yml (overriding the one from the debian package)
• /etc/elasticsearch/jvm.options.d/jvm.options (adding some Xms/Xmx override)

This also fixed a couple of current paper cuts:

• uid/gid creation

• fix the inter-dependency on package/service/apt-config order

• remove a deprecated xpack configuration (since 7.8.0 which is the prod version)

• unmanage the no longer required openjdk-8 dependency (es complained about it) [1]

• [1] We'll need to uninstall that jdk from the production esnodes

• [2] We'll need to apply the following configuration in production on node at a time.

Related to T2817

Test Plan

vagrant up staging-esnode0 ~> happily configures and starts elasticsearch accordingly

bin/octocatalog-diff on an esnode production node (there is some diff but the actual configuration in the end is the same as the current one):

bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging_add_elasticsearch_node esnode1
Found host esnode1.internal.softwareheritage.org
WARN     -> Environment "arcpatch-!250" contained non-word characters, correcting name to arcpatch_D4460
WARN     -> Environment "open-template1" contained non-word characters, correcting name to open_template1
WARN     -> Environment "update-writer-config" contained non-word characters, correcting name to update_writer_config
WARN     -> Environment "wip-pg-hba-rules-in-yaml" contained non-word characters, correcting name to wip_pg_hba_rules_in_yaml
Cloning into '/tmp/swh-ocd.idXBDTTy/environments/production/data/private'...
done.
Cloning into '/tmp/swh-ocd.idXBDTTy/environments/staging_add_elasticsearch_node/data/private'...
done.
*** Running octocatalog-diff on host esnode1.internal.softwareheritage.org
I, [2020-12-02T16:40:00.786190 #28052]  INFO -- : Catalogs compiled for esnode1.internal.softwareheritage.org
I, [2020-12-02T16:40:02.157247 #28052]  INFO -- : Diffs computed for esnode1.internal.softwareheritage.org
diff origin/production/esnode1.internal.softwareheritage.org current/esnode1.internal.softwareheritage.org
*******************************************
+ Concat::Fragment[0_es_jvm_option] =>
   parameters =>
      "content": "-Xms16g"
      "order": "00"
      "target": "es_jvm_options"
*******************************************
+ Concat::Fragment[1_es_jvm_option] =>
   parameters =>
      "content": "-Xmx16g"
      "order": "00"
      "target": "es_jvm_options"
*******************************************
+ Concat[es_jvm_options] =>
   parameters =>
      "backup": "puppet"
      "ensure": "present"
      "ensure_newline": true
      "force": false
      "format": "plain"
      "group": 119
      "mode": "0644"
      "notify": "Service[elasticsearch]"
      "order": "alpha"
      "owner": 114
      "path": "/etc/elasticsearch/jvm.options.d/jvm.options"
      "replace": true
      "show_diff": true
      "warn": false
*******************************************
+ Concat_file[es_jvm_options] =>
   parameters =>
      "backup": "puppet"
      "ensure_newline": true
      "force": false
      "format": "plain"
      "group": 119
      "mode": "0644"
      "order": "alpha"
      "owner": 114
      "path": "/etc/elasticsearch/jvm.options.d/jvm.options"
      "replace": true
      "show_diff": true
      "tag": "es_jvm_options"
*******************************************
+ Concat_fragment[0_es_jvm_option] =>
   parameters =>
      "content": "-Xms16g"
      "order": "00"
      "tag": "es_jvm_options"
      "target": "es_jvm_options"
*******************************************
+ Concat_fragment[1_es_jvm_option] =>
   parameters =>
      "content": "-Xmx16g"
      "order": "00"
      "tag": "es_jvm_options"
      "target": "es_jvm_options"
*******************************************
+ File[/etc/elasticsearch/elasticsearch.yml] =>
   parameters =>
      "ensure": "file"
      "group": 119
      "mode": "0644"
      "notify": "Service[elasticsearch]"
      "owner": 114
      "content": >>>
# File managed by puppet - modifications will be lost
cluster.name: swh-logging-prod
node.name: esnode1
network.host: 192.168.100.61
discovery.seed_hosts:
- esnode1.internal.softwareheritage.org
- esnode2.internal.softwareheritage.org
- esnode3.internal.softwareheritage.org
cluster.initial_master_nodes:
- esnode1
- esnode2
- esnode3
path.data: "/srv/elasticsearch"
path.logs: "/var/log/elasticsearch"
index.store.type: hybridfs
indices.memory.index_buffer_size: 50%
<<<
*******************************************
+ File[/srv/elasticsearch] =>
   parameters =>
      "ensure": "directory"
      "group": 119
      "mode": "2755"
      "owner": 114
*******************************************
- File_line[elasticsearch store type]
*******************************************
+ Group[elasticsearch] =>
   parameters =>
      "ensure": "present"
      "gid": 119
*******************************************
- Package[openjdk-8-jre-headless]
*******************************************
  Systemd::Dropin_file[elasticsearch.conf] =>
   parameters =>
     notify =>
      + Service[elasticsearch]
*******************************************
  User[elasticsearch] =>
   parameters =>
     gid =>
      - "119"
      + 119
     uid =>
      - "114"
      + 114
*******************************************
*** End octocatalog-diff on esnode1.internal.softwareheritage.org

---

Migrated from D4651 (view on Phabricator)