CloudWatch Logs in AWS - Time & Space Complexity
When working with CloudWatch Logs, it's important to understand how the time to process logs grows as you add more log events.
We want to know how the number of API calls changes as the amount of log data increases.
Analyze the time complexity of the following operation sequence.
// Put multiple log events into a log stream
aws logs put-log-events \
--log-group-name MyLogGroup \
--log-stream-name MyLogStream \
--log-events '[{"timestamp": 1609459200000, "message": "Log message 1"}, {"timestamp": 1609459201000, "message": "Log message 2"}]'
// Retrieve log events from the log stream
aws logs get-log-events \
--log-group-name MyLogGroup \
--log-stream-name MyLogStream
This sequence sends batches of log messages to CloudWatch Logs and then retrieves them from the log stream.
Identify the API calls, resource provisioning, data transfers that repeat.
- Primary operation: The
put-log-eventsAPI call to send log batches. - How many times: Once per batch of log events sent.
- Secondary operation: The
get-log-eventsAPI call to read logs. - How many times: Once per retrieval request, which may paginate if logs are large.
As you increase the number of log events, the number of put-log-events calls grows roughly in proportion to how many batches you need.
| Input Size (n log events) | Approx. API Calls |
|---|---|
| 10 | 1 (one batch) |
| 1000 | 1 (one batch) |
| 100000 | About 100 batches |
Pattern observation: The number of API calls grows linearly with the number of log events divided by batch size.
Time Complexity: O(n)
This means the time to send or retrieve logs grows directly in proportion to the number of log events.
[X] Wrong: "Sending more logs will always take the same time because it's just one API call."
[OK] Correct: CloudWatch limits batch sizes, so large numbers of logs require multiple API calls, increasing total time.
Understanding how log ingestion scales helps you design systems that handle large volumes efficiently and predict costs and delays.
"What if we changed the batch size limit to a smaller number? How would the time complexity change?"
Practice
Solution
Step 1: Understand log group role
A log group is a container that holds multiple log streams, organizing logs by application or resource.Step 2: Differentiate log streams and groups
Log streams are individual sequences of log events within a log group, not the group itself.Final Answer:
To act as a container for multiple log streams -> Option AQuick Check:
Log group = container for streams [OK]
- Confusing log groups with log streams
- Thinking logs are stored directly in groups
- Assuming groups trigger alarms directly
MyAppLogs?Solution
Step 1: Identify correct AWS CLI service and command
The correct service is 'logs' and the command to create a log group is 'create-log-group'.Step 2: Check command syntax
The parameter for the log group name is '--log-group-name', matching aws logs create-log-group --log-group-name MyAppLogs exactly.Final Answer:
aws logs create-log-group --log-group-name MyAppLogs -> Option BQuick Check:
Correct CLI syntax = aws logs create-log-group --log-group-name MyAppLogs [OK]
- Using wrong service name like 'cloudwatch' instead of 'logs'
- Incorrect command verbs like 'new-log-group'
- Wrong parameter names like '--name'
AppLogs?
aws logs put-retention-policy --log-group-name AppLogs --retention-in-days 14
Solution
Step 1: Understand the put-retention-policy command
This command sets how many days logs are retained before deletion.Step 2: Interpret the retention-in-days value
The parameter '--retention-in-days 14' means logs are kept for 14 days.Final Answer:
Logs will be kept for 14 days -> Option CQuick Check:
Retention days = 14 [OK]
- Assuming default retention instead of set value
- Confusing retention period with log group creation
- Thinking logs are kept indefinitely by default
aws logs put-retention-policy --log-group-name NonExistentGroup --retention-in-days 30What is the most likely cause?
Solution
Step 1: Check existence of log group
The command requires the log group to exist before setting retention.Step 2: Validate retention period and command syntax
Retention of 30 days is valid and the command syntax is correct.Final Answer:
The log group 'NonExistentGroup' does not exist -> Option AQuick Check:
Retention needs existing log group [OK]
- Assuming retention period is invalid
- Confusing log stream with log group
- Thinking command syntax is wrong
Solution
Step 1: Understand log group and stream organization
Best practice is to use log groups to separate different applications or microservices.Step 2: Assign log streams per instance or source
Within each log group, use multiple log streams to separate logs by instance or container.Step 3: Evaluate options
Create one log group per microservice and multiple log streams per instance matches best practice: one log group per microservice, multiple streams per instance.Final Answer:
Create one log group per microservice and multiple log streams per instance -> Option DQuick Check:
Group per microservice, streams per instance [OK]
- Mixing microservices in one log group
- Using one stream for all logs causing confusion
- Creating groups per instance instead of per service
Ready to prove it?
Pass the test and this pattern moves unknown
