Log groups and log streams in AWS - Deep Dive

Overview - Log groups and log streams

What is it?

Log groups and log streams are ways to organize and store logs in AWS CloudWatch. A log group is like a folder that holds many log streams. Each log stream is a sequence of log events from a single source, like an application or server. Together, they help you collect, view, and manage logs efficiently.

Why it matters

Without log groups and streams, logs would be scattered and hard to find, making it difficult to troubleshoot problems or monitor systems. They solve the problem of organizing large amounts of log data so you can quickly find what you need. This helps keep systems reliable and saves time when fixing issues.

Where it fits

Before learning about log groups and streams, you should understand basic cloud concepts and what logs are. After this, you can learn about log retention, metric filters, and how to analyze logs with AWS tools like CloudWatch Insights.

Mental Model

Core Idea

Log groups are folders that hold related log streams, which are ordered sequences of log events from a single source.

Think of it like...

Imagine a library where each bookshelf is a log group, and each book on the shelf is a log stream containing pages of events. The bookshelf groups similar books, and each book tells a story from one source.

┌─────────────┐
│ Log Group A │
│ ┌─────────┐ │
│ │Stream 1 │ │
│ │(App 1)  │ │
│ └─────────┘ │
│ ┌─────────┐ │
│ │Stream 2 │ │
│ │(App 2)  │ │
│ └─────────┘ │
└─────────────┘

Each Stream contains ordered log events from its source.

Build-Up - 7 Steps

1

Foundation

Understanding Logs and Their Purpose

Concept: Logs are records of events or messages generated by software or systems.

Logs capture what happens inside applications or servers, like errors, warnings, or normal operations. They help people understand system behavior and find problems.

Result

You know what logs are and why they are important for monitoring and troubleshooting.

Understanding logs as stories told by systems helps you see why organizing them matters.

2

Foundation

Introducing Log Groups as Containers

Concept: Log groups are containers that hold related log streams to organize logs by source or purpose.

In AWS CloudWatch, a log group is like a folder that groups logs from similar sources, such as all logs from one application or environment. This keeps logs tidy and easier to manage.

Result

You can imagine logs grouped logically, making it easier to find and control them.

Knowing that log groups act as folders helps you plan how to organize logs effectively.

3

Intermediate

Exploring Log Streams as Event Sequences

🤔Before reading on: do you think a log stream can contain logs from multiple sources or just one? Commit to your answer.

Concept: Log streams are sequences of log events from a single source within a log group.

Each log stream holds ordered log events from one source, like one server or one application instance. This order helps track what happened over time.

Result

You understand that log streams keep logs from one source in order, making it easier to follow events.

Recognizing that streams are single-source sequences clarifies how logs stay organized and traceable.

4

Intermediate

How Log Groups and Streams Work Together

🤔Before reading on: do you think log groups can contain log streams from different applications or just one? Commit to your answer.

Concept: Log groups hold multiple log streams, each from different sources but related by purpose or system.

A log group might contain streams from several servers running the same app or different parts of a system. This grouping helps manage permissions and retention policies at once.

Result

You see how grouping streams under one log group simplifies management and access control.

Understanding this relationship helps you design scalable and secure logging setups.

5

Intermediate

Setting Retention and Access on Log Groups

Concept: Retention and access policies apply at the log group level to control how long logs are kept and who can see them.

You can set how many days logs stay before deletion and who can read or write logs in a group. This protects data and controls costs.

Result

You know how to manage log lifecycle and security efficiently.

Knowing that policies apply to groups, not streams, helps avoid mistakes in log management.

6

Advanced

Handling Log Event Ordering and Timestamps

🤔Before reading on: do you think log events in a stream are always stored in the order they arrive or can they be out of order? Commit to your answer.

Concept: Log events in streams are ordered by timestamp but can arrive out of order and be reordered by CloudWatch.

Sometimes logs arrive late or out of order. CloudWatch uses timestamps to sort them correctly in streams, ensuring accurate event sequences.

Result

You understand how CloudWatch maintains event order despite network delays or retries.

Knowing this prevents confusion when logs seem unordered and helps design reliable logging.

7

Expert

Optimizing Log Group and Stream Design for Scale

🤔Before reading on: do you think having too many log streams in one group affects performance? Commit to your answer.

Concept: Designing log groups and streams carefully affects performance, cost, and usability at scale.

Too many streams in one group can slow queries and increase costs. Splitting logs by environment, app version, or region helps. Also, naming conventions and tagging improve management.

Result

You can create logging setups that stay efficient and manageable as systems grow.

Understanding scaling tradeoffs helps avoid costly mistakes in large production environments.

Under the Hood

AWS CloudWatch stores logs in a hierarchical structure: log groups contain log streams, which hold ordered log events. When an event is sent, it includes a timestamp and is appended to a stream. CloudWatch indexes these events for fast retrieval and applies retention and access policies at the group level. Internally, streams are sharded and distributed for scalability.

Why designed this way?

This design balances organization, scalability, and security. Grouping logs allows shared policies and easier management. Streams keep event order from individual sources, which is critical for troubleshooting. Alternatives like flat log storage would be chaotic and inefficient at scale.

┌───────────────┐
│  Log Group    │
│ ┌───────────┐ │
│ │ Log Stream│ │
│ │  (Source) │ │
│ │ ┌───────┐ │ │
│ │ │Event 1│ │ │
│ │ │Event 2│ │ │
│ │ └───────┘ │ │
│ └───────────┘ │
└───────────────┘

Policies apply at Log Group level; events ordered in streams.

Myth Busters - 4 Common Misconceptions

Quick: Do you think a log stream can contain logs from multiple servers? Commit to yes or no.

Common Belief: A log stream can hold logs from many different servers or applications.

Reality: Each log stream contains logs from only one source, like a single server or app instance.

Why it matters: Mixing sources in one stream would make it hard to trace issues and understand event order.

Quick: Do you think retention policies apply to individual log streams? Commit to yes or no.

Common Belief: Retention settings can be applied separately to each log stream.

Reality: Retention policies apply only at the log group level, affecting all streams inside.

Why it matters: Trying to set retention per stream leads to confusion and mismanagement of log data.

Quick: Do you think log events always arrive in perfect time order? Commit to yes or no.

Common Belief: Log events are always stored in the exact order they are generated.

Reality: Events can arrive out of order due to network delays, but CloudWatch sorts them by timestamp.

Why it matters: Assuming perfect order can cause misinterpretation of logs and troubleshooting errors.

Quick: Do you think having many log streams in one group has no impact on performance? Commit to yes or no.

Common Belief: You can have unlimited log streams in one group without affecting performance.

Reality: Too many streams in one group can degrade query speed and increase costs.

Why it matters: Ignoring this leads to slow log searches and higher AWS bills.

Expert Zone

1

Log stream names often include timestamps or instance IDs to help identify sources quickly, a practice many overlook.

2

CloudWatch limits the rate of log event ingestion per stream; exceeding this causes throttling, which can silently drop logs if not monitored.

3

Retention policies can be automated with lifecycle rules, but forgetting to set them leads to unexpected storage costs.

When NOT to use

If you need real-time log analytics or complex querying beyond CloudWatch's capabilities, consider using specialized log management services like Elasticsearch or third-party tools. Also, for very high-volume logs, direct streaming to data lakes or analytics platforms may be better.

Production Patterns

In production, teams create separate log groups per environment (dev, test, prod) and per application component. They use consistent naming conventions and tags for automation. Retention policies are set to balance compliance and cost. Metric filters on log groups trigger alarms for critical events.

Connections

File System Hierarchy

Log groups and streams mimic folders and files in a file system.

Understanding file systems helps grasp how logs are organized and accessed efficiently.

Event Sourcing in Software Design

Log streams are like event logs that record state changes over time.

Knowing event sourcing clarifies why ordered logs per source are crucial for reconstructing system behavior.

Library Cataloging Systems

Grouping and indexing logs is similar to how libraries organize books for easy retrieval.

This connection shows the importance of metadata and grouping for managing large collections.

Common Pitfalls

#1 Mixing logs from multiple sources in one log stream.

Wrong approach:

Sending logs from two different servers to the same log stream named 'app-logs'.

Correct approach:

Create separate log streams for each server, e.g., 'app-logs-server1' and 'app-logs-server2'.

Root cause:

Misunderstanding that log streams must represent a single source to maintain event order and traceability.

#2 Setting retention policies on individual streams instead of groups.

Wrong approach:

Trying to configure retention for a single log stream in CloudWatch console.

Correct approach:

Set retention policy on the entire log group that contains the streams.

Root cause:

Confusing the scope of retention settings, which apply only at the group level.

#3 Ignoring log event timestamps causing misordered logs.

Wrong approach:

Assuming logs appear in the order sent and not checking timestamps.

Correct approach:

Use timestamps in log events and rely on CloudWatch to order them correctly.

Root cause:

Not accounting for network delays or retries that cause out-of-order arrival.

Key Takeaways

Log groups are containers that organize related log streams, making log management easier and more secure.

Log streams hold ordered log events from a single source, preserving the sequence of events for accurate troubleshooting.

Retention and access policies apply at the log group level, not per stream, simplifying lifecycle and security management.

CloudWatch sorts log events by timestamp to handle out-of-order arrivals, ensuring correct event sequences.

Designing log groups and streams thoughtfully is essential for performance, cost control, and effective monitoring at scale.

Practice

1. What is the main purpose of a log group in AWS CloudWatch Logs?

easy

A. To organize multiple log streams into a single container

B. To store individual log events directly

C. To control user access to AWS services

D. To monitor AWS billing and cost usage

Solution

1. Step 1: Understand log group role

   A log group acts as a container for multiple log streams, grouping logs by application or resource.

2. Step 2: Differentiate from other options

   Log events are stored in log streams, not directly in log groups. Access control and billing are unrelated.

3. Final Answer:

   To organize multiple log streams into a single container -> Option A

4. Quick Check:

   Log group = container for log streams [OK]

Hint: Log groups hold streams, streams hold logs [OK]

Common Mistakes:

• Confusing log groups with log streams
• Thinking logs are stored directly in log groups
• Mixing log groups with access control or billing

2. Which of the following is the correct way to create a log stream named appStream inside a log group appGroup using AWS CLI?

easy

A. aws logs create-log-group --log-stream-name appStream --log-group-name appGroup

B. aws logs create-log-stream --name appStream --group appGroup

C. aws logs create-log-stream appGroup appStream

D. aws logs create-log-stream --log-group-name appGroup --log-stream-name appStream

Solution

1. Step 1: Identify correct AWS CLI command syntax

   The command to create a log stream is aws logs create-log-stream with parameters --log-group-name and --log-stream-name.

2. Step 2: Verify aws logs create-log-stream --log-group-name appGroup --log-stream-name appStream matches syntax

   aws logs create-log-stream --log-group-name appGroup --log-stream-name appStream correctly uses the command and parameters in the right order and names.

3. Final Answer:

   aws logs create-log-stream --log-group-name appGroup --log-stream-name appStream -> Option D

4. Quick Check:

   Correct CLI syntax = aws logs create-log-stream --log-group-name appGroup --log-stream-name appStream [OK]

Hint: Remember: create-log-stream needs both group and stream names [OK]

Common Mistakes:

• Using create-log-group instead of create-log-stream
• Swapping parameter names
• Omitting required flags

3. Given the following AWS CLI commands executed in order:

aws logs create-log-group --log-group-name myAppLogs
aws logs create-log-stream --log-group-name myAppLogs --log-stream-name stream1
aws logs put-log-events --log-group-name myAppLogs --log-stream-name stream1 --log-events '[{"timestamp":1609459200000,"message":"Start"}]'
aws logs describe-log-streams --log-group-name myAppLogs

What will the describe-log-streams command output include?

medium

A. A list containing one log stream named 'stream1' with stored events

B. An empty list because no log streams exist yet

C. An error stating the log group does not exist

D. A list of all log groups in the account

Solution

1. Step 1: Trace commands creating log group and stream

   The first two commands create a log group 'myAppLogs' and a log stream 'stream1' inside it.

2. Step 2: Put log event and describe streams

   The third command adds a log event to 'stream1'. The last command lists streams in 'myAppLogs', so it will show 'stream1' with events.

3. Final Answer:

   A list containing one log stream named 'stream1' with stored events -> Option A

4. Quick Check:

   Created stream appears in describe output [OK]

Hint: Create group and stream before putting logs [OK]

Common Mistakes:

• Expecting empty list before adding streams
• Confusing log groups with streams in output
• Assuming describe lists groups, not streams

4. You run this AWS CLI command:

aws logs put-log-events --log-group-name myGroup --log-stream-name myStream --log-events '[{"timestamp":1609459200000,"message":"Hello"}]'

But you get an error saying the log stream does not exist. What is the most likely fix?

medium

A. Remove the timestamp from the log event

B. Change the log group name to an existing group

C. Create the log stream 'myStream' inside 'myGroup' before putting events

D. Use the create-log-group command instead

Solution

1. Step 1: Understand error cause

   The error indicates the log stream 'myStream' does not exist in 'myGroup'. You must create the stream before sending logs.

2. Step 2: Identify correct fix

   Creating the log stream with aws logs create-log-stream resolves the error. Changing group or removing timestamp won't help.

3. Final Answer:

   Create the log stream 'myStream' inside 'myGroup' before putting events -> Option C

4. Quick Check:

   Log stream must exist before putting logs [OK]

Hint: Create log stream first to avoid put-log-events error [OK]

Common Mistakes:

• Trying to put logs before stream creation
• Ignoring error details about missing stream
• Confusing log group creation with stream creation

5. You want to organize logs from multiple microservices in AWS CloudWatch. Each microservice should have its own log stream, but all streams must be grouped for easy management and retention. Which setup is best?

hard

A. Store logs outside CloudWatch to avoid managing groups and streams

B. Create one log group for all microservices, and create separate log streams for each microservice inside it

C. Create one log group per microservice and put all logs in a single stream inside each group

D. Create one log stream per microservice without using log groups

Solution

1. Step 1: Understand grouping and streaming roles

   Log groups organize logs for management and retention. Log streams separate logs by source within groups.

2. Step 2: Choose best structure for microservices

   Using one log group for all microservices and separate streams per microservice allows easy management and searching.

3. Step 3: Evaluate other options

   Create one log stream per microservice without using log groups lacks grouping, C creates many groups complicating retention, A ignores CloudWatch benefits.

4. Final Answer:

   Create one log group for all microservices, and create separate log streams for each microservice inside it -> Option B

5. Quick Check:

   Group = container; streams = sources [OK]

Hint: Group logs by app, stream by source inside group [OK]

Common Mistakes:

• Skipping log groups and using only streams
• Creating too many log groups unnecessarily
• Ignoring CloudWatch log management features