Send data from RabbitMQ to Axiom

Monitor RabbitMQ logs and metrics using Elastic Beats to send data to Axiom.

RabbitMQ is a message broker that enables applications to communicate through asynchronous messaging. Monitoring RabbitMQ is essential for tracking message flow, identifying bottlenecks, and ensuring reliable message delivery.

This guide shows you how to send RabbitMQ logs and metrics to Axiom using Elastic Beats.

Prerequisites

• Create an Axiom account.
• Create a dataset in Axiom where you send your data.
• Create an API token in Axiom with permissions to ingest data to the dataset you have created.

• RabbitMQ installed and running
• Filebeat OSS version 8.12.1 or lower for collecting logs
• Metricbeat OSS version 8.12.1 or lower for collecting metrics

Why monitor RabbitMQ?

Monitoring RabbitMQ helps you:

• Track message rates: Monitor messages published, delivered, and acknowledged to identify throughput issues
• Identify bottlenecks: Detect queues with high message counts or slow consumer rates
• Monitor resource usage: Track memory, disk usage, and connection counts
• Ensure reliability: Alert on message delivery failures or queue buildup
• Optimize performance: Analyze consumer efficiency and message processing times

Setup

You can monitor RabbitMQ by collecting both logs and metrics. For complete observability, set up both Filebeat for logs and Metricbeat for metrics.

Collect RabbitMQ logs with Filebeat

Filebeat collects RabbitMQ logs and sends them to Axiom for analysis.

Configure RabbitMQ logging

First, ensure RabbitMQ is configured to write logs to a file. Add this to your RabbitMQ configuration file (rabbitmq.conf):

log.file.level = info
log.file = /var/log/rabbitmq/rabbit.log
log.console = false

Restart RabbitMQ after making configuration changes.

Configure Filebeat

Create or edit your filebeat.yml configuration file:

setup.ilm.enabled: false

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      # Adjust paths based on your RabbitMQ installation
      - /var/log/rabbitmq/*.log
    fields:
      service: rabbitmq
      log_type: application
    fields_under_root: true

output.elasticsearch:
  hosts: ['https://AXIOM_DOMAIN:443/v1/datasets/DATASET_NAME/elastic']
  api_key: 'axiom:API_TOKEN'
  allow_older_versions: true

Replace AXIOM_DOMAIN with api.axiom.co if your organization uses the US region. For more information, see Regions.

Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.

Replace DATASET_NAME with the name of the Axiom dataset where you send your data.

Start Filebeat

filebeat -e -c filebeat.yml

Collect RabbitMQ metrics with Metricbeat

Metricbeat collects detailed metrics from the RabbitMQ management API.

Enable RabbitMQ Management Plugin

The RabbitMQ management plugin exposes metrics through an HTTP API:

rabbitmq-plugins enable rabbitmq_management

The management interface is available at http://localhost:15672 by default.

Configure Metricbeat

Create or edit your metricbeat.yml configuration file:

setup.ilm.enabled: false

metricbeat.modules:
  - module: rabbitmq
    enabled: true
    period: 10s
    hosts: ['localhost:15672']
    username: guest
    password: guest
    metricsets:
      - node
      - queue
      - connection
      - exchange

output.elasticsearch:
  hosts: ['https://AXIOM_DOMAIN:443/v1/datasets/DATASET_NAME/elastic']
  api_key: 'axiom:API_TOKEN'
  allow_older_versions: true

Replace AXIOM_DOMAIN with api.axiom.co if your organization uses the US region. For more information, see Regions.

Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.

Replace DATASET_NAME with the name of the Axiom dataset where you send your data.

Replace username and password with your RabbitMQ management credentials.

Don’t use the default guest credentials in production. Create a dedicated monitoring user with read-only permissions.

Create a monitoring user (recommended)

For production environments, create a dedicated monitoring user:

# Create monitoring user
rabbitmqctl add_user monitoring secure_password

# Grant monitoring tag (read-only access to management API)
rabbitmqctl set_user_tags monitoring monitoring

# Grant permissions to monitor all vhosts
rabbitmqctl set_permissions -p / monitoring ".*" ".*" ".*"

Then update your metricbeat.yml with the monitoring credentials:

username: monitoring
password: secure_password

Start Metricbeat

metricbeat -e -c metricbeat.yml

Query examples

Once your RabbitMQ data is flowing into Axiom, use these example queries to analyze your message broker:

Monitor message rates

Track messages published and consumed over time:

['rabbitmq']
| where metricset_name == "queue"
| summarize 
    published = avg(rabbitmq_queue_messages_published_total),
    consumed = avg(rabbitmq_queue_messages_consumed_total)
  by bin(_time, 1m), rabbitmq_queue_name

Identify problematic queues

Find queues with high message counts that might indicate processing issues:

['rabbitmq']
| where metricset_name == "queue"
| where rabbitmq_queue_messages_ready > 1000
| summarize 
    max_messages = max(rabbitmq_queue_messages_ready),
    avg_consumers = avg(rabbitmq_queue_consumers)
  by rabbitmq_queue_name
| sort by max_messages desc

Monitor consumer efficiency

Analyze how quickly consumers are processing messages:

['rabbitmq']
| where metricset_name == "queue"
| extend consumer_utilization = todouble(rabbitmq_queue_consumer_utilisation)
| summarize 
    avg_utilization = avg(consumer_utilization),
    message_count = avg(rabbitmq_queue_messages)
  by bin(_time, 5m), rabbitmq_queue_name
| where avg_utilization < 0.8  // Alert on low utilization

Track connection and channel counts

Monitor RabbitMQ connections to detect connection leaks:

['rabbitmq']
| where metricset_name == "node"
| summarize 
    connections = avg(rabbitmq_node_fd_used),
    channels = avg(rabbitmq_node_channel_count)
  by bin(_time, 1m)

Analyze error logs

Search RabbitMQ logs for errors and warnings:

['rabbitmq']
| where log_type == "application"
| where message contains "error" or message contains "warning"
| project _time, message, severity
| sort by _time desc

Troubleshooting

Filebeat can’t access log files

If Filebeat can’t read RabbitMQ logs, ensure proper permissions:

# Grant read access to RabbitMQ log files
sudo chmod 644 /var/log/rabbitmq/*.log

# If running Filebeat as non-root, add the user to the rabbitmq group
sudo usermod -a -G rabbitmq filebeat-user

Metricbeat connection refused

If Metricbeat can’t connect to the management API:

1. Verify the management plugin is enabled:

   rabbitmq-plugins list | grep management
   

2. Check if the management port is accessible:

   curl http://localhost:15672/api/overview -u guest:guest
   

3. Verify RabbitMQ is listening on the correct interface in rabbitmq.conf:

   management.tcp.ip = 0.0.0.0
   management.tcp.port = 15672
   

High memory usage in RabbitMQ

If you notice high memory usage in your metrics:

1. Check queue depths and message sizes
2. Verify consumers are processing messages
3. Consider setting memory limits in rabbitmq.conf:
   vm_memory_high_watermark.relative = 0.6
   

Best practices

1. Monitor both logs and metrics: Logs provide detailed error information while metrics give you performance insights
2. Set up alerts: Create monitors in Axiom for critical metrics like queue depth, consumer count, and error rates
3. Secure credentials: Use dedicated monitoring users with read-only access instead of admin credentials
4. Adjust collection intervals: Balance between data granularity and resource usage by tuning Metricbeat’s period setting
5. Correlate data: Use the service: rabbitmq field to correlate logs and metrics in your queries

Related documentation

• Elastic Beats overview: Learn more about Filebeat and Metricbeat configuration options
• Monitors: Set up alerts for RabbitMQ metrics
• APL Query Language: Learn APL to create powerful RabbitMQ analytics queries