Log groups and log streams in AWS - Step-by-Step Execution

Process Flow - Log groups and log streams

Create Log Group

↓

Create Log Stream inside Log Group

↓

Send Logs to Log Stream

↓

Logs Stored in Log Stream

↓

View Logs via Console or API

Logs are organized first by log groups, then by log streams inside those groups. Logs are sent to streams and stored there for viewing.

Execution Sample

AWS

aws logs create-log-group --log-group-name MyApp
aws logs create-log-stream --log-group-name MyApp --log-stream-name Stream1
aws logs put-log-events --log-group-name MyApp --log-stream-name Stream1 --log-events '[{"timestamp":123456789,"message":"Start"}]'

This sequence creates a log group, a log stream inside it, and sends a log message to that stream.

Process Table

Step	Action	Resource Created/Used	Result
1	Create Log Group 'MyApp'	Log Group: MyApp	Log group 'MyApp' created
2	Create Log Stream 'Stream1' in 'MyApp'	Log Stream: Stream1 in MyApp	Log stream 'Stream1' created inside 'MyApp'
3	Send log event to 'Stream1'	Log Event: timestamp=123456789, message='Start'	Log event stored in 'Stream1'
4	View logs	Log Stream: Stream1	Logs retrieved: message='Start'
5	Attempt to send log to non-existent stream	Log Stream: StreamX	Error: The specified log stream does not exist

💡 Execution stops after sending logs and viewing them; error occurs if sending logs to a non-existent stream.

Status Tracker

Resource	Initial	After Step 1	After Step 2	After Step 3	After Step 4
Log Groups	None	MyApp	MyApp	MyApp	MyApp
Log Streams in MyApp	None	None	Stream1	Stream1	Stream1
Log Events in Stream1	None	None	None	[{timestamp:123456789, message:'Start'}]	[{timestamp:123456789, message:'Start'}]

Key Moments - 2 Insights

Why do we need to create a log stream inside a log group before sending logs?

Because logs are stored inside streams, which are inside groups. Step 2 shows creating the stream; without it, step 3 fails (see step 5 error).

What happens if you try to send logs to a stream that does not exist?

An error occurs and logs are not stored, as shown in step 5 of the execution table.

Visual Quiz - 3 Questions

Look at the execution table, what resource is created at step 2?

A Log Group 'MyApp'

B Log Stream 'Stream1' inside 'MyApp'

C Log Event with message 'Start'

D Log Stream 'StreamX'

💡 Hint

Check the 'Resource Created/Used' column at step 2 in the execution table.

At which step does the log event get stored in the log stream?

A Step 1

B Step 2

C Step 3

D Step 5

💡 Hint

Look at the 'Result' column for when the log event is stored.

If you skip creating the log stream, what will happen when sending logs?

A An error will occur because the stream does not exist

B Logs will be stored in the log group directly

C Logs will be stored in a default stream

D Logs will be discarded silently

💡 Hint

Refer to step 5 in the execution table showing the error when sending logs to a non-existent stream.

Concept Snapshot

Log groups organize logs by application or service.
Log streams are inside log groups and hold sequences of log events.
You must create a log group, then a log stream before sending logs.
Logs are sent to a specific stream and stored there.
Trying to send logs to a missing stream causes an error.

Full Transcript

This visual execution shows how AWS CloudWatch Logs organizes logs. First, a log group is created to group related logs. Inside this group, log streams are created to hold sequences of log events. Logs are sent to a specific log stream. The execution table traces creating a log group named 'MyApp', then a log stream 'Stream1' inside it, then sending a log event with a timestamp and message. The variable tracker shows how resources appear step by step. Key moments clarify why streams must exist before sending logs and what happens if they don't. The quiz tests understanding of resource creation and error handling. The snapshot summarizes the core concepts simply.

Practice

1. What is the main purpose of a log group in AWS CloudWatch Logs?

easy

A. To organize multiple log streams into a single container

B. To store individual log events directly

C. To control user access to AWS services

D. To monitor AWS billing and cost usage

Solution

1. Step 1: Understand log group role

   A log group acts as a container for multiple log streams, grouping logs by application or resource.

2. Step 2: Differentiate from other options

   Log events are stored in log streams, not directly in log groups. Access control and billing are unrelated.

3. Final Answer:

   To organize multiple log streams into a single container -> Option A

4. Quick Check:

   Log group = container for log streams [OK]

Hint: Log groups hold streams, streams hold logs [OK]

Common Mistakes:

• Confusing log groups with log streams
• Thinking logs are stored directly in log groups
• Mixing log groups with access control or billing

2. Which of the following is the correct way to create a log stream named appStream inside a log group appGroup using AWS CLI?

easy

A. aws logs create-log-group --log-stream-name appStream --log-group-name appGroup

B. aws logs create-log-stream --name appStream --group appGroup

C. aws logs create-log-stream appGroup appStream

D. aws logs create-log-stream --log-group-name appGroup --log-stream-name appStream

Solution

1. Step 1: Identify correct AWS CLI command syntax

   The command to create a log stream is aws logs create-log-stream with parameters --log-group-name and --log-stream-name.

2. Step 2: Verify aws logs create-log-stream --log-group-name appGroup --log-stream-name appStream matches syntax

   aws logs create-log-stream --log-group-name appGroup --log-stream-name appStream correctly uses the command and parameters in the right order and names.

3. Final Answer:

   aws logs create-log-stream --log-group-name appGroup --log-stream-name appStream -> Option D

4. Quick Check:

   Correct CLI syntax = aws logs create-log-stream --log-group-name appGroup --log-stream-name appStream [OK]

Hint: Remember: create-log-stream needs both group and stream names [OK]

Common Mistakes:

• Using create-log-group instead of create-log-stream
• Swapping parameter names
• Omitting required flags

3. Given the following AWS CLI commands executed in order:

aws logs create-log-group --log-group-name myAppLogs
aws logs create-log-stream --log-group-name myAppLogs --log-stream-name stream1
aws logs put-log-events --log-group-name myAppLogs --log-stream-name stream1 --log-events '[{"timestamp":1609459200000,"message":"Start"}]'
aws logs describe-log-streams --log-group-name myAppLogs

What will the describe-log-streams command output include?

medium

A. A list containing one log stream named 'stream1' with stored events

B. An empty list because no log streams exist yet

C. An error stating the log group does not exist

D. A list of all log groups in the account

Solution

1. Step 1: Trace commands creating log group and stream

   The first two commands create a log group 'myAppLogs' and a log stream 'stream1' inside it.

2. Step 2: Put log event and describe streams

   The third command adds a log event to 'stream1'. The last command lists streams in 'myAppLogs', so it will show 'stream1' with events.

3. Final Answer:

   A list containing one log stream named 'stream1' with stored events -> Option A

4. Quick Check:

   Created stream appears in describe output [OK]

Hint: Create group and stream before putting logs [OK]

Common Mistakes:

• Expecting empty list before adding streams
• Confusing log groups with streams in output
• Assuming describe lists groups, not streams

4. You run this AWS CLI command:

aws logs put-log-events --log-group-name myGroup --log-stream-name myStream --log-events '[{"timestamp":1609459200000,"message":"Hello"}]'

But you get an error saying the log stream does not exist. What is the most likely fix?

medium

A. Remove the timestamp from the log event

B. Change the log group name to an existing group

C. Create the log stream 'myStream' inside 'myGroup' before putting events

D. Use the create-log-group command instead

Solution

1. Step 1: Understand error cause

   The error indicates the log stream 'myStream' does not exist in 'myGroup'. You must create the stream before sending logs.

2. Step 2: Identify correct fix

   Creating the log stream with aws logs create-log-stream resolves the error. Changing group or removing timestamp won't help.

3. Final Answer:

   Create the log stream 'myStream' inside 'myGroup' before putting events -> Option C

4. Quick Check:

   Log stream must exist before putting logs [OK]

Hint: Create log stream first to avoid put-log-events error [OK]

Common Mistakes:

• Trying to put logs before stream creation
• Ignoring error details about missing stream
• Confusing log group creation with stream creation

5. You want to organize logs from multiple microservices in AWS CloudWatch. Each microservice should have its own log stream, but all streams must be grouped for easy management and retention. Which setup is best?

hard

A. Store logs outside CloudWatch to avoid managing groups and streams

B. Create one log group for all microservices, and create separate log streams for each microservice inside it

C. Create one log group per microservice and put all logs in a single stream inside each group

D. Create one log stream per microservice without using log groups

Solution

1. Step 1: Understand grouping and streaming roles

   Log groups organize logs for management and retention. Log streams separate logs by source within groups.

2. Step 2: Choose best structure for microservices

   Using one log group for all microservices and separate streams per microservice allows easy management and searching.

3. Step 3: Evaluate other options

   Create one log stream per microservice without using log groups lacks grouping, C creates many groups complicating retention, A ignores CloudWatch benefits.

4. Final Answer:

   Create one log group for all microservices, and create separate log streams for each microservice inside it -> Option B

5. Quick Check:

   Group = container; streams = sources [OK]

Hint: Group logs by app, stream by source inside group [OK]

Common Mistakes:

• Skipping log groups and using only streams
• Creating too many log groups unnecessarily
• Ignoring CloudWatch log management features