Why stream processing transforms data in Kafka - Why It Works

Introduction

Data often arrives continuously and needs to be changed or enriched immediately to be useful. Stream processing transforms this flowing data in real time so systems can react quickly and keep information fresh.

When you want to filter out unwanted data from a live feed before saving it

When you need to enrich incoming data by adding extra details on the fly

When you want to aggregate or summarize data continuously as it arrives

When you must detect patterns or anomalies in data streams instantly

When you want to route data to different systems based on its content

Config File - stream-processing.properties

stream-processing.properties

bootstrap.servers=localhost:9092
application.id=stream-transform-app
processing.guarantee=exactly_once
cache.max.bytes.buffering=10485760
commit.interval.ms=1000

bootstrap.servers: Kafka server address to connect to.

application.id: Unique ID for this stream processing app.

processing.guarantee: Ensures data is processed exactly once to avoid duplicates.

cache.max.bytes.buffering: Controls memory used for buffering data before processing.

commit.interval.ms: How often to save progress to Kafka.

Commands

Create a Kafka topic named 'raw-data' where unprocessed data will be sent.

Terminal

kafka-topics --create --topic raw-data --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Expected Output

Created topic raw-data.

→

--partitions - Number of partitions for parallel processing

→

--replication-factor - Number of copies for fault tolerance

Create a Kafka topic named 'transformed-data' to hold the processed and transformed data.

Terminal

kafka-topics --create --topic transformed-data --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Expected Output

Created topic transformed-data.

→

--partitions - Number of partitions for parallel processing

→

--replication-factor - Number of copies for fault tolerance

Send sample raw data messages to the 'raw-data' topic to simulate incoming data stream.

Terminal

kafka-console-producer --topic raw-data --bootstrap-server localhost:9092

Expected Output

No output (command runs silently)

→

--topic - Specify the topic to send data to

Start the stream processing application that reads from 'raw-data', transforms the data, and writes to 'transformed-data'.

Terminal

kafka-streams-application-start --config stream-processing.properties

Expected Output

Stream processing application started with application.id=stream-transform-app

→

--config - Specify the configuration file for the stream app

Read and display the transformed data from the 'transformed-data' topic to verify the transformation.

Terminal

kafka-console-consumer --topic transformed-data --bootstrap-server localhost:9092 --from-beginning

Expected Output

transformed message 1

transformed message 2

transformed message 3

→

--from-beginning - Read all messages from the start of the topic

Key Concept

Stream processing transforms data immediately as it flows in, enabling real-time insights and actions.

Common Mistakes

Not creating the output topic before starting the stream processing app

The app will fail to write transformed data if the output topic does not exist.

Always create both input and output topics before running the stream processing application.

Sending data to the wrong topic

If raw data is sent to the transformed-data topic, the app will not process it correctly.

Send raw data only to the input topic configured for the stream processor.

Not setting processing.guarantee to exactly_once

Without exactly-once processing, data might be duplicated or lost during transformation.

Set processing.guarantee=exactly_once in the config to ensure reliable processing.

Summary

Create Kafka topics for raw input data and transformed output data.

Send raw data to the input topic to simulate a live data stream.

Run a stream processing app that reads raw data, transforms it, and writes to the output topic.

Use a consumer to verify the transformed data is correctly produced.

Practice

1. Why do we use stream processing to transform data in Kafka Streams?

easy

A. To make data more useful and meaningful in real time

B. To store data permanently without changes

C. To slow down data flow for analysis later

D. To delete unnecessary data immediately

Solution

1. Step 1: Understand the purpose of stream processing

   Stream processing changes data as it flows to add value or meaning instantly.

2. Step 2: Identify the correct purpose from options

   Only To make data more useful and meaningful in real time says it makes data more useful in real time, which matches the purpose.

3. Final Answer:

   To make data more useful and meaningful in real time -> Option A

4. Quick Check:

   Stream processing transforms data = Make data useful [OK]

Hint: Stream processing changes data to add value instantly [OK]

Common Mistakes:

• Thinking stream processing only stores data
• Believing it slows down data flow
• Confusing transformation with deletion

2. Which of the following is the correct way to apply a map transformation in Kafka Streams using Java?

easy

A. stream.reduce((v1, v2) -> v1 + v2)

B. stream.filter(value -> value.length())

C. stream.mapValues(value -> value.toUpperCase())

D. stream.collect(Collectors.toList())

Solution

1. Step 1: Identify the map transformation syntax

   In Kafka Streams, mapValues applies a function to each value, e.g., converting to uppercase.

2. Step 2: Match the correct syntax

   stream.mapValues(value -> value.toUpperCase()) uses mapValues with a lambda toUpperCase(), which is correct.

3. Final Answer:

   stream.mapValues(value -> value.toUpperCase()) -> Option C

4. Quick Check:

   map transformation syntax = mapValues with lambda [OK]

Hint: Map uses mapValues with a function on values [OK]

Common Mistakes:

• Using filter instead of map
• Using reduce for mapping
• Using collect which is not a stream operation here

3. Given the following Kafka Streams code snippet, what will be the output if the input stream has values ["apple", "banana", "cherry"]?

stream.filter((k, v) -> v.startsWith("b")).toList()

medium

A. ["banana"]

B. ["apple", "banana", "cherry"]

C. ["cherry"]

D. []

Solution

1. Step 1: Understand the filter condition

   The filter keeps only values starting with 'b'. From the list, only "banana" starts with 'b'.

2. Step 2: Determine the output list

   After filtering, the list contains only ["banana"].

3. Final Answer:

   ["banana"] -> Option A

4. Quick Check:

   Filter by startsWith('b') = ["banana"] [OK]

Hint: Filter keeps only matching values [OK]

Common Mistakes:

• Including all values without filtering
• Choosing values starting with wrong letters
• Confusing filter with map

4. Identify the error in this Kafka Streams code snippet that tries to filter out empty strings:

stream.filter((k, v) -> v != "")

medium

A. filter() cannot be used on streams

B. Missing semicolon at the end

C. Lambda syntax is incorrect

D. Using '!=' to compare strings instead of .equals()

Solution

1. Step 1: Check string comparison in Java

   In Java, '!=' compares object references, not string content. Use .equals() to compare strings.

2. Step 2: Identify the correct fix

   Replace 'value != ""' with '!value.equals("")' to properly filter out empty strings.

3. Final Answer:

   Using '!=' to compare strings instead of .equals() -> Option D

4. Quick Check:

   String comparison needs .equals() [OK]

Hint: Use .equals() to compare strings in Java [OK]

Common Mistakes:

• Using '==' or '!=' for string content comparison
• Assuming filter() is invalid on streams
• Ignoring lambda syntax correctness

5. You want to enrich a Kafka stream of user IDs by adding their names from a lookup table. Which approach best uses stream processing transformations?

hard

A. Reduce the stream to count user IDs

B. Use a join operation between the user ID stream and the user info table

C. Map the stream values to uppercase user IDs

D. Filter the stream to only user IDs present in the table

Solution

1. Step 1: Understand enrichment in stream processing

   Enrichment means adding extra data, here user names, to existing stream data.

2. Step 2: Identify the correct transformation

   Joining the user ID stream with the user info table adds the names, which is enrichment.

3. Final Answer:

   Use a join operation between the user ID stream and the user info table -> Option B

4. Quick Check:

   Enrich data = join streams/tables [OK]

Hint: Enrich data by joining streams with lookup tables [OK]

Common Mistakes:

• Using filter which only removes data
• Mapping to uppercase does not add info
• Reducing counts data, not enriches