Log analysis and troubleshooting in RabbitMQ - Commands & Configuration

Introduction

RabbitMQ logs record what happens inside the message broker. Analyzing these logs helps find and fix problems quickly, like connection issues or message delivery failures.

When RabbitMQ is not starting properly and you need to find the cause.

When messages are not being delivered and you want to check for errors.

When connections from clients are dropping unexpectedly.

When you want to monitor RabbitMQ health and spot warnings early.

When troubleshooting performance issues related to queues or exchanges.

Commands

This command shows the last 50 lines of the main RabbitMQ log file to quickly see recent events and errors.

Terminal

sudo tail -n 50 /var/log/rabbitmq/rabbit@$(hostname).log

Expected Output

2024-06-01 10:15:23.456 [info] <0.123.0> Starting RabbitMQ 3.11.14 on Erlang 25.3

2024-06-01 10:15:23.789 [warning] <0.456.0> Connection closed unexpectedly from client 192.168.1.10

2024-06-01 10:15:24.012 [error] <0.789.0> Failed to declare queue 'my-queue': already exists with different parameters

→

-n 50 - Show only the last 50 lines for quick recent log review

This command searches the RabbitMQ log file for lines containing 'error' to find problems quickly.

Terminal

sudo grep 'error' /var/log/rabbitmq/rabbit@$(hostname).log

Expected Output

2024-06-01 10:15:24.012 [error] <0.789.0> Failed to declare queue 'my-queue': already exists with different parameters

2024-06-01 10:16:00.345 [error] <0.890.0> Channel error on connection <0.1234.0>: not_found - no queue 'unknown-queue'

This command shows the current status of RabbitMQ, including running nodes and memory usage, to help identify if the server is healthy.

Terminal

sudo rabbitmqctl status

Expected Output

Status of node rabbit@myhost ...

[{pid,12345},

 {running_applications,[{rabbit,"RabbitMQ","3.11.14"},{os_mon,"CPO  CXC 138 46","2.4"}]},

 {memory,[{total,12345678},{connection_readers,123456},{connection_writers,123456}]},

 {listeners,[{clustering,25672,"::"},{amqp,5672,"0.0.0.0"}]},

 {uptime,3600}]

This command restarts the RabbitMQ server to apply fixes or clear transient errors after troubleshooting.

Terminal

sudo systemctl restart rabbitmq-server

Expected Output

No output (command runs silently)

This command continuously shows new log entries in real time to monitor RabbitMQ behavior after changes or during testing.

Terminal

sudo tail -f /var/log/rabbitmq/rabbit@$(hostname).log

Expected Output

2024-06-01 10:20:00.123 [info] <0.234.0> Server started successfully

2024-06-01 10:20:05.456 [info] <0.345.0> New connection from client 192.168.1.10

→

-f - Follow the log file live to see new entries as they happen

Key Concept

If you remember nothing else from this pattern, remember: RabbitMQ logs are your first and best tool to find what went wrong and when.

Common Mistakes

Trying to read logs without sudo or proper permissions

RabbitMQ log files are usually owned by root or rabbitmq user, so you get permission denied errors.

Use sudo to read or follow the log files to get access.

Ignoring warnings and only searching for 'error' in logs

Warnings often indicate problems that can lead to errors if not fixed early.

Check both warnings and errors in the logs to catch issues early.

Restarting RabbitMQ without checking logs first

Restarting blindly can cause downtime without understanding the root cause.

Always check logs to diagnose the problem before restarting the service.

Summary

Use 'tail' to quickly view recent RabbitMQ log entries for troubleshooting.

Use 'grep' to filter logs for errors and warnings to find issues faster.

Check RabbitMQ status with 'rabbitmqctl status' to verify server health.

Restart RabbitMQ only after analyzing logs to avoid unnecessary downtime.

Use 'tail -f' to monitor logs live during testing or after fixes.

Practice

1. What is the primary purpose of RabbitMQ logs?

easy

A. To show what RabbitMQ is doing and help find problems

B. To store user data permanently

C. To speed up message delivery

D. To configure RabbitMQ settings automatically

Solution

1. Step 1: Understand the role of logs in RabbitMQ

   Logs record events and actions RabbitMQ performs, which helps in monitoring and troubleshooting.

2. Step 2: Identify the main use of logs

   Logs are not for storing user data or speeding delivery but for tracking system behavior and errors.

3. Final Answer:

   To show what RabbitMQ is doing and help find problems -> Option A

4. Quick Check:

   Logs = system activity and troubleshooting [OK]

Hint: Logs track activity and errors, not data or speed [OK]

Common Mistakes:

• Thinking logs store user messages
• Assuming logs configure RabbitMQ automatically
• Believing logs improve message speed

2. Which command correctly shows the last 50 lines of the RabbitMQ log file located at /var/log/rabbitmq/rabbit.log?

easy

A. head -n 50 /var/log/rabbitmq/rabbit.log

B. tail -n 50 /var/log/rabbitmq/rabbit.log

C. cat -n 50 /var/log/rabbitmq/rabbit.log

D. grep -n 50 /var/log/rabbitmq/rabbit.log

Solution

1. Step 1: Identify command to view last lines of a file

   The tail command shows the end of a file; -n 50 limits output to last 50 lines.

2. Step 2: Check other commands

   head shows the start, cat outputs whole file, grep searches text, so they don't fit.

3. Final Answer:

   tail -n 50 /var/log/rabbitmq/rabbit.log -> Option B

4. Quick Check:

   tail = last lines [OK]

Hint: Use tail -n for last lines, head for first lines [OK]

Common Mistakes:

• Using head instead of tail for last lines
• Using grep without search pattern
• Using cat with invalid options

3. Given the command rabbitmqctl status, what kind of information will it display?

medium

A. Current server health and status details

B. List of all messages in queues

C. Configuration file contents

D. User permissions and roles

Solution

1. Step 1: Understand what rabbitmqctl status shows

   This command provides a summary of the RabbitMQ server's health, including running nodes and memory usage.

2. Step 2: Differentiate from other data

   It does not list messages, config files, or user roles directly.

3. Final Answer:

   Current server health and status details -> Option A

4. Quick Check:

   rabbitmqctl status = server health [OK]

Hint: rabbitmqctl status shows server health info [OK]

Common Mistakes:

• Expecting message lists from status command
• Confusing config files with status output
• Thinking it shows user permissions

4. You run tail -f /var/log/rabbitmq/rabbit.log but see no new log entries when RabbitMQ processes messages. What is the most likely cause?

medium

A. tail command does not work with RabbitMQ logs

B. The log file path is incorrect

C. RabbitMQ server is stopped

D. RabbitMQ logging level is set too low to show message processing

Solution

1. Step 1: Check why logs don't update with tail -f

   If RabbitMQ is running but logs don't show activity, logging level may be too low to capture message events.

2. Step 2: Eliminate other causes

   If path was wrong, tail would error; if server stopped, no processing happens; tail works with any file.

3. Final Answer:

   RabbitMQ logging level is set too low to show message processing -> Option D

4. Quick Check:

   Low logging level hides events [OK]

Hint: Check logging level if logs stay empty during activity [OK]

Common Mistakes:

• Assuming tail command is faulty
• Not verifying RabbitMQ server status
• Ignoring logging level settings

5. You notice RabbitMQ logs show frequent "connection refused" errors. Which combined steps help you troubleshoot this issue effectively?

hard

A. Restart RabbitMQ service and clear all logs immediately

B. Increase log verbosity and delete all user accounts

C. Check RabbitMQ server status with rabbitmqctl status and verify firewall rules

D. Change RabbitMQ port without checking current configuration

Solution

1. Step 1: Verify RabbitMQ server is running

   Use rabbitmqctl status to confirm the server is up and accepting connections.

2. Step 2: Check network and firewall settings

   Connection refused often means network blocks or firewall rules prevent access; verify these settings.

3. Final Answer:

   Check RabbitMQ server status with rabbitmqctl status and verify firewall rules -> Option C

4. Quick Check:

   Status + firewall check = fix connection refused [OK]

Hint: Check server status and firewall for connection refused errors [OK]

Common Mistakes:

• Restarting service without checking status
• Deleting users unnecessarily
• Changing ports blindly