Draft: logstash: Route new elastic infrastructure logs into dedicated index

Out of the current latest run on the loader git [1].

This seems like the simplest implementation, any thoughts?

Octodiff is happy [2].

[1]

loaders {"asctime": "2023-02-01 10:35:39,553", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/worker/consumer/connection.py", "l
ineno": 22, "funcName": "start", "task_name": null, "task_id": null, "name": "celery.worker.consumer.connection", "levelname": "INFO", "message": "Connected to amqp://swhcons
umer:**@scheduler0.internal.staging.swh.network:5672//"}
loaders {"asctime": "2023-02-01 10:35:39,616", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/worker/strategy.py", "lineno": 161
, "funcName": "task_message_handler", "task_name": null, "task_id": null, "name": "celery.worker.strategy", "levelname": "INFO", "message": "Task swh.loader.git.tasks.UpdateG
itRepository[6124105e-51c1-4c0e-a277-efa864b446d4] received", "data": {"id": "6124105e-51c1-4c0e-a277-efa864b446d4", "name": "swh.loader.git.tasks.UpdateGitRepository", "args
": "[]", "kwargs": "{'url': 'https://bitbucket.org/apolnikovs/skyline-33.git', 'lister_name': 'bitbucket', 'lister_instance_name': 'bitbucket'}"}}
loaders {"asctime": "2023-02-01 10:35:39,617", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/apps/worker.py", "lineno": 176, "f
uncName": "on_consumer_ready", "task_name": null, "task_id": null, "name": "celery.apps.worker", "levelname": "INFO", "message": "loader@loader-git-54fbcc775f-bgjkl ready."}
loaders {"asctime": "2023-02-01 10:35:39,724", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/worker/strategy.py", "lineno": 161
, "funcName": "task_message_handler", "task_name": null, "task_id": null, "name": "celery.worker.strategy", "levelname": "INFO", "message": "Task swh.loader.git.tasks.UpdateG
itRepository[2ee6504d-1841-4ba0-b397-774c4930c6ca] received", "data": {"id": "2ee6504d-1841-4ba0-b397-774c4930c6ca", "name": "swh.loader.git.tasks.UpdateGitRepository", "args
": "[]", "kwargs": "{'url': 'https://bitbucket.org/apolnikovs/skyline-34.git', 'lister_name': 'bitbucket', 'lister_instance_name': 'bitbucket'}"}}
loaders {"asctime": "2023-02-01 10:35:39,891", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/loader.py", "lineno": 369
, "funcName": "load", "task_name": null, "task_id": null, "name": "swh.loader.git.loader.GitLoader", "levelname": "INFO", "message": "Load origin 'https://bitbucket.org/apoln
ikovs/skyline-33.git' with type 'git'"}

Refs. swh/infra/sysadm-environment#4524 (closed)

[2]

octo-diff logstash0

$ $SWH_PUPPET_ENVIRONMENT_HOME/bin/octocatalog-diff --to update_logstash_route logstash0.internal.softwareheritage.org
Found host logstash0.internal.softwareheritage.org
...
*** Running octocatalog-diff on host logstash0.internal.softwareheritage.org
I, [2023-02-01T11:56:03.396308 #2968320]  INFO -- : Catalogs compiled for logstash0.internal.softwareheritage.org
I, [2023-02-01T11:56:03.670220 #2968320]  INFO -- : Diffs computed for logstash0.internal.softwareheritage.org
diff origin/production/logstash0.internal.softwareheritage.org current/logstash0.internal.softwareheritage.org

File[/etc/logstash/conf.d/filter.conf] =>
parameters =>
content =>
@@ -30,4 +30,8 @@
}
}
+  } else if [threadName] == "MainThread" {  # elastic worker
+      add_field => {
+        "[@metadata][target_index]" => "swh_workers-%{[@metadata][version]}-%{+YYYY.MM.dd}"
+      }
} else {
if [syslog][facility] =~ /[A-Za-z]+/ {

*** End octocatalog-diff on logstash0.internal.softwareheritage.org