elk: upgrade journalbeat and filebeat

- Remove the journalbeat 5.5.0 service configuration
- remove the journalbeat user not needed anymore
- deploy the elasticsearch packages (journalbeat and filebeat)
- cleanup the temporary configuration
- cleanup the old cursor position, when the version will be updated
  the beginning of the journal will be reingested
- update the logstash configuration to support the messages
  from the old journalbeat version and the new version in parallel
- allow to override the beat version for node with an elasticsearch
  version specified (swh-search)
- upate the `check_journal` script to use the new registry file
  to check the cursor position
- Automatically declare the journalbeat index templates from the logstash host

The mappings of the new systemlogs-7.15.1-* and swh_worker-7.15.1-* index need to be manually declared on elasticsearch before.

Related to T3705 Depends on !440 (closed)

Test Plan

• logstash0:

diff origin/production/logstash0.internal.softwareheritage.org current/logstash0.internal.softwareheritage.org
*******************************************
+ Apt::Pin[journalbeat] =>
   parameters =>
     "codename": "",
     "component": "",
     "ensure": "present",
     "explanation": "Use the elk stack version",
     "label": "",
     "order": 50,
     "origin": "",
     "originator": "",
     "packages": [
       "journalbeat"
     ],
     "priority": 1001,
     "release": "",
     "release_version": "",
     "version": "7.15.1"
*******************************************
  Apt::Pin[swh-journalbeat] =>
   parameters =>
     ensure =>
      - present
      + absent
     explanation =>
      - Use journalbeat packages from Software Heritage
     originator =>
      - softwareheritage
      +_
     packages =>
      - ["journalbeat"]
      + *
     priority =>
      - 990
      + 0
*******************************************
+ Apt::Setting[pref-journalbeat] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\nExplanation: Use...
     "ensure": "present",
     "notify_update": false,
     "priority": 50
*******************************************
  Apt::Setting[pref-swh-journalbeat] =>
   parameters =>
     content =>
      @@ -1,5 +1,5 @@
       # This file is managed by Puppet. DO NOT EDIT.
      -Explanation: Use journalbeat packages from Software Heritage
      -Package: journalbeat
      -Pin: release o=softwareheritage
      -Pin-Priority: 990
      +Explanation: profile: swh-journalbeat
      +Package: *
      +Pin: release a=swh-journalbeat
      +Pin-Priority: 0
     ensure =>
      - present
      + absent
*******************************************
+ File[/etc/apt/preferences.d/journalbeat.pref] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\nExplanation: Use...
     "ensure": "present",
     "group": "root",
     "mode": "0644",
     "owner": "root"
*******************************************
  File[/etc/apt/preferences.d/swh-journalbeat.pref] =>
   parameters =>
     ensure =>
      - present
      + absent
*******************************************
  File[/etc/journalbeat/journalbeat.yml] =>
   parameters =>
     content =>
      @@ -2,4 +2,10 @@
      _
       journalbeat:
      +  inputs:
      +    # Paths that should be crawled and fetched. Possible values files and directories.
      +    # When setting a directory, all journals under it are merged.
      +    # When empty starts to read from local journal.
      +    - paths: []
      +
         # What position in journald to seek to at start up
         # options: cursor, tail, head (defaults to tail)
*******************************************
- File[/etc/journalbeat]
*******************************************
  File[/etc/logstash/conf.d/filter.conf] =>
   parameters =>
     content =>
      @@ -18,4 +18,5 @@
           }
         } else if "swh-worker@" in [systemd_unit] {
      +    # Temporary rule to delete after complete migration to 7.15.1
           mutate {
             add_field => {
      @@ -23,8 +24,23 @@
             }
           }
      -  } else {
      +  } else if "swh-worker@" in [systemd][unit] {
           mutate {
             add_field => {
      -        "[@metadata][target_index]" => "systemlogs-%{+YYYY.MM.dd}"
      +        "[@metadata][target_index]" => "swh_workers-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      +      }
      +    }
      +  } else {
      +    if [@metadata][version] {
      +      mutate {
      +        add_field => {
      +          "[@metadata][target_index]" => "systemlogs-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      +        }
      +      }
      +    } else {
      +      # Temporary rule to delete after complete migration to 7.15.1
      +      mutate {
      +        add_field => {
      +          "[@metadata][target_index]" => "systemlogs-%{+YYYY.MM.dd}"
      +        }
             }
           }
*******************************************
+ File[/etc/systemd/system/journalbeat.service.d/journalbeat.conf] =>
   parameters =>
     "content": "# Managed by puppet (class profile::systemd_journal::journalbeat...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Class[Systemd::Systemctl::Daemon_reload]"
     ],
     "owner": "root",
     "selinux_ignore_defaults": false,
     "show_diff": true
*******************************************
+ File[/etc/systemd/system/journalbeat.service.d] =>
   parameters =>
     "ensure": "directory",
     "group": "root",
     "owner": "root",
     "purge": true,
     "recurse": true,
     "selinux_ignore_defaults": false
*******************************************
  File[/etc/systemd/system/journalbeat.service] =>
   parameters =>
     ensure =>
      - file
      + absent
*******************************************
  Package[journalbeat] =>
   parameters =>
     ensure =>
      - present
      + 7.15.1
*******************************************
  Service[journalbeat] =>
   parameters =>
     subscribe =>
      + ["File[/etc/journalbeat/journalbeat.yml]", "Package[journalbeat]", "Systemd::Dropin_file[journalbeat.conf]"]
*******************************************
+ Systemd::Dropin_file[journalbeat.conf] =>
   parameters =>
     "content": "# Managed by puppet (class profile::systemd_journal::journalbeat...
     "daemon_reload": "lazy",
     "ensure": "present",
     "filename": "journalbeat.conf",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Service[journalbeat]"
     ],
     "owner": "root",
     "path": "/etc/systemd/system",
     "selinux_ignore_defaults": false,
     "show_diff": true,
     "unit": "journalbeat.service"
*******************************************
- Systemd::Unit_file[journalbeat.service]
*******************************************
*** End octocatalog-diff on logstash0.internal.softwareheritage.org

-swh-worker01

diff origin/production/worker01.softwareheritage.org current/worker01.softwareheritage.org
*******************************************
+ Anchor[apt_key 46095ACC8548582C1A2699A9D27D666CD88E42B4 present]
*******************************************
+ Apt::Key[Add key: 46095ACC8548582C1A2699A9D27D666CD88E42B4 from Apt::Source elasticsearch] =>
   parameters =>
     "content": "-----BEGIN PGP PUBLIC KEY BLOCK-----\nVersion: GnuPG v2.0.14 (GN...
     "ensure": "present",
     "id": "46095ACC8548582C1A2699A9D27D666CD88E42B4",
     "server": "keyserver.ubuntu.com",
     "weak_ssl": false
*******************************************
+ Apt::Pin[journalbeat] =>
   parameters =>
     "codename": "",
     "component": "",
     "ensure": "present",
     "explanation": "Use the elk stack version",
     "label": "",
     "order": 50,
     "origin": "",
     "originator": "",
     "packages": [
       "journalbeat"
     ],
     "priority": 1001,
     "release": "",
     "release_version": "",
     "version": "7.15.1"
*******************************************
  Apt::Pin[swh-journalbeat] =>
   parameters =>
     ensure =>
      - present
      + absent
     explanation =>
      - Use journalbeat packages from Software Heritage
     originator =>
      - softwareheritage
      +_
     packages =>
      - ["journalbeat"]
      + *
     priority =>
      - 990
      + 0
*******************************************
+ Apt::Setting[list-elastic-6.x] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\n# elastic-6.x\nd...
     "ensure": "absent",
     "notify_update": true,
     "priority": 50
*******************************************
+ Apt::Setting[list-elasticsearch] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\n# elasticsearch\...
     "ensure": "present",
     "notify_update": true,
     "priority": 50
*******************************************
+ Apt::Setting[pref-journalbeat] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\nExplanation: Use...
     "ensure": "present",
     "notify_update": false,
     "priority": 50
*******************************************
  Apt::Setting[pref-swh-journalbeat] =>
   parameters =>
     content =>
      @@ -1,5 +1,5 @@
       # This file is managed by Puppet. DO NOT EDIT.
      -Explanation: Use journalbeat packages from Software Heritage
      -Package: journalbeat
      -Pin: release o=softwareheritage
      -Pin-Priority: 990
      +Explanation: profile: swh-journalbeat
      +Package: *
      +Pin: release a=swh-journalbeat
      +Pin-Priority: 0
     ensure =>
      - present
      + absent
*******************************************
+ Apt::Source[elastic-6.x] =>
   parameters =>
     "allow_unsigned": false,
     "comment": "elastic-6.x",
     "ensure": "absent",
     "include": {
     },
     "notify_update": true,
     "repos": "main"
*******************************************
+ Apt::Source[elasticsearch] =>
   parameters =>
     "allow_unsigned": false,
     "comment": "elasticsearch",
     "ensure": "present",
     "include": {
     },
     "key": {
       "id": "46095ACC8548582C1A2699A9D27D666CD88E42B4",
       "content": "-----BEGIN PGP PUBLIC KEY BLOCK-----\nVersion: GnuPG v2.0.14 (...
     },
     "location": "https://artifacts.elastic.co/packages/7.x/apt",
     "notify_update": true,
     "release": "stable",
     "repos": "main"
*******************************************
+ Apt_key[Add key: 46095ACC8548582C1A2699A9D27D666CD88E42B4 from Apt::Source elasticsearch] =>
   parameters =>
     "content": "-----BEGIN PGP PUBLIC KEY BLOCK-----\nVersion: GnuPG v2.0.14 (GN...
     "ensure": "present",
     "id": "46095ACC8548582C1A2699A9D27D666CD88E42B4",
     "refresh": false,
     "server": "keyserver.ubuntu.com",
     "weak_ssl": false
*******************************************
+ File[/etc/apt/preferences.d/journalbeat.pref] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\nExplanation: Use...
     "ensure": "present",
     "group": "root",
     "mode": "0644",
     "owner": "root"
*******************************************
  File[/etc/apt/preferences.d/swh-journalbeat.pref] =>
   parameters =>
     ensure =>
      - present
      + absent
*******************************************
+ File[/etc/apt/sources.list.d/elastic-6.x.list] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\n# elastic-6.x\nd...
     "ensure": "absent",
     "group": "root",
     "mode": "0644",
     "notify": "Class[Apt::Update]",
     "owner": "root"
*******************************************
+ File[/etc/apt/sources.list.d/elasticsearch.list] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\n# elasticsearch\...
     "ensure": "present",
     "group": "root",
     "mode": "0644",
     "notify": "Class[Apt::Update]",
     "owner": "root"
*******************************************
  File[/etc/journalbeat/journalbeat.yml] =>
   parameters =>
     content =>
      @@ -2,4 +2,10 @@
      _
       journalbeat:
      +  inputs:
      +    # Paths that should be crawled and fetched. Possible values files and directories.
      +    # When setting a directory, all journals under it are merged.
      +    # When empty starts to read from local journal.
      +    - paths: []
      +
         # What position in journald to seek to at start up
         # options: cursor, tail, head (defaults to tail)
*******************************************
- File[/etc/journalbeat]
*******************************************
+ File[/etc/systemd/system/journalbeat.service.d/journalbeat.conf] =>
   parameters =>
     "content": "# Managed by puppet (class profile::systemd_journal::journalbeat...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Class[Systemd::Systemctl::Daemon_reload]"
     ],
     "owner": "root",
     "selinux_ignore_defaults": false,
     "show_diff": true
*******************************************
+ File[/etc/systemd/system/journalbeat.service.d] =>
   parameters =>
     "ensure": "directory",
     "group": "root",
     "owner": "root",
     "purge": true,
     "recurse": true,
     "selinux_ignore_defaults": false
*******************************************
  File[/etc/systemd/system/journalbeat.service] =>
   parameters =>
     ensure =>
      - file
      + absent
*******************************************
  Package[journalbeat] =>
   parameters =>
     ensure =>
      - present
      + 7.15.1
*******************************************
  Service[journalbeat] =>
   parameters =>
     subscribe =>
      + ["File[/etc/journalbeat/journalbeat.yml]", "Package[journalbeat]", "Systemd::Dropin_file[journalbeat.conf]"]
*******************************************
+ Systemd::Dropin_file[journalbeat.conf] =>
   parameters =>
     "content": "# Managed by puppet (class profile::systemd_journal::journalbeat...
     "daemon_reload": "lazy",
     "ensure": "present",
     "filename": "journalbeat.conf",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Service[journalbeat]"
     ],
     "owner": "root",
     "path": "/etc/systemd/system",
     "selinux_ignore_defaults": false,
     "show_diff": true,
     "unit": "journalbeat.service"
*******************************************
- Systemd::Unit_file[journalbeat.service]
*******************************************
*** End octocatalog-diff on worker01.softwareheritage.org

---

Migrated from D6635 (view on Phabricator)