Skip to content

rabbitmq: Enable http log

Guillaume Samson requested to merge rabbitmq_http_log into production

These modifications will activate http logging on rabbitmq servers. This will be useful for understanding scaledObject1 errors, as the probe on port 15672 on rabbitmq servers has never been triggered when we receive a large scaledObject errors packet.
We will need to add a new logrotate configuration file 2.

Testing in a local docker environment
ᐅ docker ps --format json | \
jq -r '"\(.Image) \(.Names) \(.State)"' | \
awk '{printf "%-25s %-16s %s\n",$1,$2,$3}'
rabbitmq:3.8.9-management rabbit-test-mgmt running
rabbitmq:3.8.9            rabbit-test      running

ᐅ curl -sI http://localhost:15672/ | awk 'NR==1'
HTTP/1.1 200 OK
ᐅ curl -sI http://localhost:15672/ | awk 'NR==1'
HTTP/1.1 200 OK

ᐅ docker exec -ti rabbit-test-mgmt cat /etc/rabbitmq/rabbitmq.config
% This file managed by Puppet
% Template Path: rabbitmq/templates/rabbitmq.config
[
  {rabbit, [
    {heartbeat, 0},
    {loopback_users, [<<"xxx">>]},
    {tcp_listen_options, [
         {backlog,       128},
         {nodelay,       true},
         {linger,        {true, 0}},
         {exit_on_close, false}
    ]},
    {tcp_listeners, [{"0.0.0.0", 5672}]},
    {vm_memory_high_watermark, 0.6},
    {default_user, <<"xxx">>},
    {default_pass, <<"xxx">>}
  ]},
  {kernel, [

  ]}
,
  {rabbitmq_management, [
    {http_log_dir, "/var/log/rabbitmq/http-access"},
    {listener, [
      {ip, "0.0.0.0"},
      {port, 15672}
    ]}
  ]}
].
% EOF

ᐅ docker exec -ti rabbit-test-mgmt ls /var/log/rabbitmq/http-access
access.log.2024_10_31_14
ᐅ docker exec -ti rabbit-test-mgmt cat /var/log/rabbitmq/http-access/access.log.2024_10_31_14
172.17.0.1 - - [31/Oct/2024:14:07:36 +0000] "HEAD / HTTP/1.1" 200 0 "" "curl/7.88.1"
172.17.0.1 - - [31/Oct/2024:14:07:42 +0000] "HEAD / HTTP/1.1" 200 0 "" "curl/7.88.1"
octocatalog-diff
diff origin/production/saatchi.internal.softwareheritage.org current/saatchi.internal.softwareheritage.org
*******************************************
  File[/etc/rabbitmq/rabbitmq.config] =>
   parameters =>
     content =>
      @@ -21,5 +21,6 @@
       ,
         {rabbitmq_management, [
      -    {listener, [
      +    {http_log_dir, /var/log/rabbitmq/http-access}
      +,    {listener, [
             {ip, "0.0.0.0"},
             {port, 15672}
*******************************************
*** End octocatalog-diff on saatchi.internal.softwareheritage.org
diff origin/production/scheduler0.internal.staging.swh.network current/scheduler0.internal.staging.swh.network
*******************************************
  File[/etc/rabbitmq/rabbitmq.config] =>
   parameters =>
     content =>
      @@ -21,5 +21,6 @@
       ,
         {rabbitmq_management, [
      -    {listener, [
      +    {http_log_dir, /var/log/rabbitmq/http-access}
      +,    {listener, [
             {ip, "0.0.0.0"},
             {port, 15672}
*******************************************
*** End octocatalog-diff on scheduler0.internal.staging.swh.network

  1. 16m Warning KEDAScalerFailed scaledobject/loader-svn-operators error inspecting rabbitMQ: Get "http://user:password@rabbitmq.internal.softwareheritage.org:15672/api/queues/%!F(MISSING)/swh.loader.svn.tasks.MountAndLoadSvnRepository": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

  2. root@scheduler0:~# head -1 /etc/logrotate.d/rabbitmq-server
    /var/log/rabbitmq/*.log {

Edited by Guillaume Samson

Merge request reports