CloudWatch Logs in AWS - Commands & Configuration

Introduction

CloudWatch Logs helps you collect and store logs from your applications and servers. It solves the problem of finding and analyzing errors or events by keeping all logs in one place.

When you want to see error messages from your web server to fix bugs quickly.

When you need to monitor application behavior over time to improve performance.

When you want to keep track of security events like login attempts.

When you want to create alerts based on specific log messages.

When you want to store logs safely for future audits or troubleshooting.

Config File - cloudwatch-log-group.json

cloudwatch-log-group.json

{
  "Resources": {
    "MyLogGroup": {
      "Type": "AWS::Logs::LogGroup",
      "Properties": {
        "LogGroupName": "my-app-logs",
        "RetentionInDays": 14
      }
    }
  }
}

This JSON file defines a CloudWatch Log Group named my-app-logs. The RetentionInDays key sets logs to be kept for 14 days before automatic deletion. This helps manage storage and costs.

Commands

This command creates a new CloudWatch Log Group named 'my-app-logs' to store your logs.

Terminal

aws logs create-log-group --log-group-name my-app-logs

Expected Output

No output (command runs silently)

→

--log-group-name - Specifies the name of the log group to create

This command sets the log retention period to 14 days, so logs older than that are deleted automatically.

Terminal

aws logs put-retention-policy --log-group-name my-app-logs --retention-in-days 14

Expected Output

No output (command runs silently)

→

--retention-in-days - Sets how many days to keep the logs

This command lists log groups starting with 'my-app-logs' to verify the log group was created.

Terminal

aws logs describe-log-groups --log-group-name-prefix my-app-logs

Expected Output

{"logGroups":[{"logGroupName":"my-app-logs","creationTime":1686000000000,"retentionInDays":14}]}

→

--log-group-name-prefix - Filters log groups by name prefix

This command sends a log event with a timestamp and message to the log stream 'my-stream' inside the 'my-app-logs' group.

Terminal

aws logs put-log-events --log-group-name my-app-logs --log-stream-name my-stream --log-events timestamp=1686000000000,message="Application started"

Expected Output

{"nextSequenceToken":"4953670123456789012345678901234567890123456789012345678901234567890"}

→

--log-stream-name - Specifies the log stream to send logs to

→

--log-events - Contains the log messages with timestamps

This command retrieves the log events from the 'my-stream' log stream to check the logs you sent.

Terminal

aws logs get-log-events --log-group-name my-app-logs --log-stream-name my-stream

Expected Output

{"events":[{"timestamp":1686000000000,"message":"Application started","ingestionTime":1686000001000}]}

→

--log-stream-name - Specifies which log stream to read from

Key Concept

If you remember nothing else from this pattern, remember: CloudWatch Logs groups and streams organize your logs so you can find and analyze them easily.

Common Mistakes

Trying to send logs to a log stream that does not exist.

CloudWatch Logs requires the log stream to exist before sending logs, otherwise the command fails.

Create the log stream first using 'aws logs create-log-stream' before sending logs.

Not setting a retention policy, causing logs to accumulate indefinitely.

Without retention, logs keep growing and can increase costs and storage usage.

Always set a retention period with 'put-retention-policy' to manage log storage.

Using incorrect timestamp format when sending log events.

Timestamps must be in milliseconds since epoch; wrong format causes errors or wrong log order.

Use Unix time in milliseconds for the timestamp value.

Summary

Create a CloudWatch Log Group to organize your logs.

Set a retention policy to automatically delete old logs and save costs.

Send log events to a log stream inside the log group.

Retrieve log events to verify and analyze your logs.

Practice

1. What is the primary purpose of a CloudWatch Logs log group?

easy

A. To act as a container for multiple log streams

B. To store individual log events directly

C. To trigger alarms based on log data

D. To archive logs permanently

Solution

1. Step 1: Understand log group role

   A log group is a container that holds multiple log streams, organizing logs by application or resource.

2. Step 2: Differentiate log streams and groups

   Log streams are individual sequences of log events within a log group, not the group itself.

3. Final Answer:

   To act as a container for multiple log streams -> Option A

4. Quick Check:

   Log group = container for streams [OK]

Hint: Log groups hold streams; streams hold logs [OK]

Common Mistakes:

• Confusing log groups with log streams
• Thinking logs are stored directly in groups
• Assuming groups trigger alarms directly

2. Which of the following is the correct AWS CLI command to create a new CloudWatch Logs log group named MyAppLogs?

easy

A. aws cloudwatch create-log-group --name MyAppLogs

B. aws logs create-log-group --log-group-name MyAppLogs

C. aws logs new-log-group --group-name MyAppLogs

D. aws logs add-log-group --name MyAppLogs

Solution

1. Step 1: Identify correct AWS CLI service and command

   The correct service is 'logs' and the command to create a log group is 'create-log-group'.

2. Step 2: Check command syntax

   The parameter for the log group name is '--log-group-name', matching aws logs create-log-group --log-group-name MyAppLogs exactly.

3. Final Answer:

   aws logs create-log-group --log-group-name MyAppLogs -> Option B

4. Quick Check:

   Correct CLI syntax = aws logs create-log-group --log-group-name MyAppLogs [OK]

Hint: Use 'aws logs create-log-group' with --log-group-name [OK]

Common Mistakes:

• Using wrong service name like 'cloudwatch' instead of 'logs'
• Incorrect command verbs like 'new-log-group'
• Wrong parameter names like '--name'

3. Given the following AWS CLI command, what will be the retention period set for the log group AppLogs?

aws logs put-retention-policy --log-group-name AppLogs --retention-in-days 14

medium

A. Logs will be kept for 7 days

B. Logs will be kept indefinitely

C. Logs will be kept for 14 days

D. Logs will be deleted immediately

Solution

1. Step 1: Understand the put-retention-policy command

   This command sets how many days logs are retained before deletion.

2. Step 2: Interpret the retention-in-days value

   The parameter '--retention-in-days 14' means logs are kept for 14 days.

3. Final Answer:

   Logs will be kept for 14 days -> Option C

4. Quick Check:

   Retention days = 14 [OK]

Hint: Retention days = value in --retention-in-days [OK]

Common Mistakes:

• Assuming default retention instead of set value
• Confusing retention period with log group creation
• Thinking logs are kept indefinitely by default

4. You run this command but get an error:

aws logs put-retention-policy --log-group-name NonExistentGroup --retention-in-days 30

What is the most likely cause?

medium

A. The log group 'NonExistentGroup' does not exist

B. Retention period 30 days is not allowed

C. You must specify a log stream name instead

D. The AWS CLI command is misspelled

Solution

1. Step 1: Check existence of log group

   The command requires the log group to exist before setting retention.

2. Step 2: Validate retention period and command syntax

   Retention of 30 days is valid and the command syntax is correct.

3. Final Answer:

   The log group 'NonExistentGroup' does not exist -> Option A

4. Quick Check:

   Retention needs existing log group [OK]

Hint: Retention policy requires existing log group [OK]

Common Mistakes:

• Assuming retention period is invalid
• Confusing log stream with log group
• Thinking command syntax is wrong

5. You want to organize logs from multiple microservices in CloudWatch Logs. Which approach best follows AWS best practices?

hard

A. Store all logs in a single log stream for simplicity

B. Create one log stream for all microservices in a single log group

C. Create one log group per instance and one log stream per microservice

D. Create one log group per microservice and multiple log streams per instance

Solution

1. Step 1: Understand log group and stream organization

   Best practice is to use log groups to separate different applications or microservices.

2. Step 2: Assign log streams per instance or source

   Within each log group, use multiple log streams to separate logs by instance or container.

3. Step 3: Evaluate options

   Create one log group per microservice and multiple log streams per instance matches best practice: one log group per microservice, multiple streams per instance.

4. Final Answer:

   Create one log group per microservice and multiple log streams per instance -> Option D

5. Quick Check:

   Group per microservice, streams per instance [OK]

Hint: Group by microservice, stream by instance [OK]

Common Mistakes:

• Mixing microservices in one log group
• Using one stream for all logs causing confusion
• Creating groups per instance instead of per service