Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Register
  • Sign in
  • P puppet-swh-site
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 4
    • Issues 4
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 2
    • Merge requests 2
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Platform
  • Infrastructure
  • Puppet
  • puppet-swh-site
  • Merge requests
  • !597

Draft: logstash: Route new elastic infrastructure logs into dedicated index

  • Review changes

  • Download
  • Patches
  • Plain diff
Closed Antoine R. Dumont requested to merge update_logstash_route into staging Feb 01, 2023
  • Overview 9
  • Commits 1
  • Pipelines 0
  • Changes 1

Out of the current latest run on the loader git [1].

This seems like the simplest implementation, any thoughts?

Octodiff is happy [2].

[1]

loaders {"asctime": "2023-02-01 10:35:39,553", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/worker/consumer/connection.py", "l
ineno": 22, "funcName": "start", "task_name": null, "task_id": null, "name": "celery.worker.consumer.connection", "levelname": "INFO", "message": "Connected to amqp://swhcons
umer:**@scheduler0.internal.staging.swh.network:5672//"}
loaders {"asctime": "2023-02-01 10:35:39,616", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/worker/strategy.py", "lineno": 161
, "funcName": "task_message_handler", "task_name": null, "task_id": null, "name": "celery.worker.strategy", "levelname": "INFO", "message": "Task swh.loader.git.tasks.UpdateG
itRepository[6124105e-51c1-4c0e-a277-efa864b446d4] received", "data": {"id": "6124105e-51c1-4c0e-a277-efa864b446d4", "name": "swh.loader.git.tasks.UpdateGitRepository", "args
": "[]", "kwargs": "{'url': 'https://bitbucket.org/apolnikovs/skyline-33.git', 'lister_name': 'bitbucket', 'lister_instance_name': 'bitbucket'}"}}
loaders {"asctime": "2023-02-01 10:35:39,617", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/apps/worker.py", "lineno": 176, "f
uncName": "on_consumer_ready", "task_name": null, "task_id": null, "name": "celery.apps.worker", "levelname": "INFO", "message": "loader@loader-git-54fbcc775f-bgjkl ready."}
loaders {"asctime": "2023-02-01 10:35:39,724", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/celery/worker/strategy.py", "lineno": 161
, "funcName": "task_message_handler", "task_name": null, "task_id": null, "name": "celery.worker.strategy", "levelname": "INFO", "message": "Task swh.loader.git.tasks.UpdateG
itRepository[2ee6504d-1841-4ba0-b397-774c4930c6ca] received", "data": {"id": "2ee6504d-1841-4ba0-b397-774c4930c6ca", "name": "swh.loader.git.tasks.UpdateGitRepository", "args
": "[]", "kwargs": "{'url': 'https://bitbucket.org/apolnikovs/skyline-34.git', 'lister_name': 'bitbucket', 'lister_instance_name': 'bitbucket'}"}}
loaders {"asctime": "2023-02-01 10:35:39,891", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/site-packages/swh/loader/core/loader.py", "lineno": 369
, "funcName": "load", "task_name": null, "task_id": null, "name": "swh.loader.git.loader.GitLoader", "levelname": "INFO", "message": "Load origin 'https://bitbucket.org/apoln
ikovs/skyline-33.git' with type 'git'"}

Refs. swh/infra/sysadm-environment#4524 (closed)

[2]

octo-diff logstash0
$ $SWH_PUPPET_ENVIRONMENT_HOME/bin/octocatalog-diff --to update_logstash_route logstash0.internal.softwareheritage.org
Found host logstash0.internal.softwareheritage.org
...

*** Running octocatalog-diff on host logstash0.internal.softwareheritage.org I, [2023-02-01T11:56:03.396308 #2968320] INFO -- : Catalogs compiled for logstash0.internal.softwareheritage.org I, [2023-02-01T11:56:03.670220 #2968320] INFO -- : Diffs computed for logstash0.internal.softwareheritage.org diff origin/production/logstash0.internal.softwareheritage.org current/logstash0.internal.softwareheritage.org


File[/etc/logstash/conf.d/filter.conf] => parameters => content => @@ -30,4 +30,8 @@ } } + } else if [threadName] == "MainThread" { # elastic worker + add_field => { + "[@metadata][target_index]" => "swh_workers-%{[@metadata][version]}-%{+YYYY.MM.dd}" + } } else { if [syslog][facility] =~ /[A-Za-z]+/ {


*** End octocatalog-diff on logstash0.internal.softwareheritage.org

Edited Feb 01, 2023 by Antoine R. Dumont
Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: update_logstash_route