elk: upgrade journalbeat and filebeat

keeping this diff as planned changes as it seems there is a conflict with the elasticsearch version for the staging and swh-search elastic nodes (redeclare the elastic::elk_version property for swh-search)

*** Running octocatalog-diff on host search-esnode4.internal.softwareheritage.org
I, [2021-11-10T12:54:23.710611 #16530]  INFO -- : Catalogs compiled for search-esnode4.internal.softwareheritage.org
I, [2021-11-10T12:54:23.998043 #16530]  INFO -- : Diffs computed for search-esnode4.internal.softwareheritage.org
diff origin/production/search-esnode4.internal.softwareheritage.org current/search-esnode4.internal.softwareheritage.org
*******************************************
+ Apt::Pin[journalbeat] =>
   parameters =>
     "codename": "",
     "component": "",
     "ensure": "present",
     "explanation": "Use the elk stack version",
     "label": "",
     "order": 50,
     "origin": "",
     "originator": "",
     "packages": [
       "journalbeat"
     ],
     "priority": 1001,
     "release": "",
     "release_version": "",
     "version": "7.9.3"    <--------------- Not good
*******************************************
  Apt::Pin[swh-journalbeat] =>
   parameters =>
     ensure =>
      - present
      + absent
     explanation =>
      - Use journalbeat packages from Software Heritage
     originator =>
      - softwareheritage
      +_
     packages =>
      - ["journalbeat"]
      + *
     priority =>
      - 990
      + 0
*******************************************
+ Apt::Setting[pref-journalbeat] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\nExplanation: Use...
     "ensure": "present",
     "notify_update": false,
     "priority": 50
*******************************************
  Apt::Setting[pref-swh-journalbeat] =>
   parameters =>
     content =>
      @@ -1,5 +1,5 @@
       # This file is managed by Puppet. DO NOT EDIT.
      -Explanation: Use journalbeat packages from Software Heritage
      -Package: journalbeat
      -Pin: release o=softwareheritage
      -Pin-Priority: 990
      +Explanation: profile: swh-journalbeat
      +Package: *
      +Pin: release a=swh-journalbeat
      +Pin-Priority: 0
     ensure =>
      - present
      + absent
*******************************************
+ File[/etc/apt/preferences.d/journalbeat.pref] =>
   parameters =>
     "content": "# This file is managed by Puppet. DO NOT EDIT.\nExplanation: Use...
     "ensure": "present",
     "group": "root",
     "mode": "0644",
     "owner": "root"
*******************************************
  File[/etc/apt/preferences.d/swh-journalbeat.pref] =>
   parameters =>
     ensure =>
      - present
      + absent
*******************************************
  File[/etc/journalbeat/journalbeat.yml] =>
   parameters =>
     content =>
      @@ -2,4 +2,10 @@
      _
       journalbeat:
      +  inputs:
      +    # Paths that should be crawled and fetched. Possible values files and directories.
      +    # When setting a directory, all journals under it are merged.
      +    # When empty starts to read from local journal.
      +    - paths: []
      +
         # What position in journald to seek to at start up
         # options: cursor, tail, head (defaults to tail)
*******************************************
- File[/etc/journalbeat]
*******************************************
+ File[/etc/systemd/system/journalbeat.service.d/journalbeat.conf] =>
   parameters =>
     "content": "# Managed by puppet (class profile::systemd_journal::journalbeat...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Class[Systemd::Systemctl::Daemon_reload]"
     ],
     "owner": "root",
     "selinux_ignore_defaults": false,
     "show_diff": true
*******************************************
+ File[/etc/systemd/system/journalbeat.service.d] =>
   parameters =>
     "ensure": "directory",
     "group": "root",
     "owner": "root",
     "purge": true,
     "recurse": true,
     "selinux_ignore_defaults": false
*******************************************
  File[/etc/systemd/system/journalbeat.service] =>
   parameters =>
     ensure =>
      - file
      + absent
*******************************************
  Package[journalbeat] =>
   parameters =>
     ensure =>
      - present
      + 7.9.3
*******************************************
  Service[journalbeat] =>
   parameters =>
     subscribe =>
      + ["File[/etc/journalbeat/journalbeat.yml]", "Package[journalbeat]", "Systemd::Dropin_file[journalbeat.conf]"]
*******************************************
+ Systemd::Dropin_file[journalbeat.conf] =>
   parameters =>
     "content": "# Managed by puppet (class profile::systemd_journal::journalbeat...
     "daemon_reload": "lazy",
     "ensure": "present",
     "filename": "journalbeat.conf",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Service[journalbeat]"
     ],
     "owner": "root",
     "path": "/etc/systemd/system",
     "selinux_ignore_defaults": false,
     "show_diff": true,
     "unit": "journalbeat.service"
*******************************************
- Systemd::Unit_file[journalbeat.service]
*******************************************
*** End octocatalog-diff on search-esnode4.internal.softwareheritage.org

Allow to override the beat version

Thanks for working on this!

I have a few questions:

Do we really care about running as a separate user? Maybe avoiding a gratuitous divergence from upstream would be worth it.
Could you fully update the config template to the new default config?
Does the icinga check for the journalbeat position still work? I assume so, as the config for it is still there.
Should we consider moving to cursor_seek_fallback: tail to work around the issues with the old indexes needing to be reopened, when rebooting a machine? This should not apply as the cursor state should be saved on service shutdown, but maybe it isn't being read back properly...

Do we really care about running as a separate user? Maybe avoiding a gratuitous divergence from upstream would be worth it. Nope, I keep it to remains as close as what we have today, I can let it per default and add the cleanup of the user previously created

Could you fully update the config template to the new default config? Yep, I will try to automize that

Does the icinga check for the journalbeat position still work? I assume so, as the config for it is still there. I think so too but I will double check to be sure

Should we consider moving to cursor_seek_fallback: tail to work around the issues with the old indexes needing to be reopened, when rebooting a machine? This should not apply as the cursor state should be saved on service shutdown, but maybe it isn't being read back properly... I will also try this because the upgrade changed nothing, the current behavior is reproduced locally with the new version when I restart the vms

Does the icinga check for the journalbeat position still work? I assume so, as the config for it is still there. I think so too but I will double check to be sure good point, it seems it's not working anymore, the new version of logstash is silently ignoring the property

The cursor state is now store in the /var/lib/journalbeat/registry file and the format has changed:

journal_entries:
- path: LOCAL_SYSTEM_JOURNAL
  cursor: s=fbf433e3151442d48a41f6bbe7178f76;i=7caea;b=a12f885575a64123a63b824ecbb3848c;m=1f56598ac;t=5d0e54cb8961a;x=a04cf1cd032d5b03
  realtime_timestamp: 1637058136806938
  monotonic_timestamp: 8412043436

compared to the previous /var/log/journalbeat/cursor-state:

vsellier@logstash0 ~ % sudo cat /var/lib/journalbeat/cursor-state
s=b3186d31ac0b4c0aa8a11239fb635349;i=27764513;b=7c201d02af8f41d3ae4f0462c271b721;m=8e01a7abbf9;t=5d0e565142685;x=9d6c9621135a86d%

The check_journal script need to be updated too.

Hopefully, the format of the cursor didn't changed:

root@logstash0:/var/lib/journalbeat# grep cursor registry | awk '{print $2}' > cursor-state
root@logstash0:/var/lib/journalbeat# /usr/lib/nagios/plugins/swh/check_journal
JOURNAL LAG OK - 0s behind, 0 entries behind | entries_lag=0 monotonic_lag=0.0s;1200;3600 realtime_lag=0.0s;1200;3600

Improve the check_journal script to check the new registry file and fallback to the old cursor_state file if not found

diff updates:

remove the journal beat user
cleanup the cursor_state file

diff update:

upgrade to 7.15.2
automatically manage the journalbeat index template from the logstash server

lgtm

couple of absolutely non-blocking remarks inline.

Merge request was accepted

approved this merge request

update the log formating in python script
fix a couple of typos

Some references in the commit message have been migrated:

T3705 is now swh/meta#3705 (closed)

rebase

Merge request was merged

closed

elk: upgrade journalbeat and filebeat

Test Plan

Closed by Vincent Sellier 3 years ago (Nov 17, 2021 4:11pm UTC) 3 years ago

Activity

elk: upgrade journalbeat and filebeat

Test Plan

Merge request reports

Closed by Vincent Sellier 3 years ago (Nov 17, 2021 4:11pm UTC) 3 years ago

Activity