Transform and converter chains in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of a Kafka Connect transform chain

Consider a Kafka Connect source connector with the following transform chain applied to each record:

{"transforms": "dropField,replaceValue", "transforms.dropField.type": "org.apache.kafka.connect.transforms.ReplaceField$Value", "transforms.dropField.blacklist": "password", "transforms.replaceValue.type": "org.apache.kafka.connect.transforms.ReplaceField$Value", "transforms.replaceValue.whitelist": "username"}

Given a record with value:

{"username": "alice", "password": "secret", "email": "[e-mail]"}

What will be the value of the record after the transform chain?

A {"password": "secret"}

B {"username": "alice", "email": "[e-mail]"}

C {"email": "[e-mail]"}

D {"username": "alice"}

💡 Hint

Think about the order of transforms and what each one keeps or removes.

The first transform drops the 'password' field, so the record becomes {"username": "alice", "email": "[e-mail]"}. The second transform keeps only the 'username' field, removing 'email'. So the final value is {"username": "alice"}.

❓ Predict Output

intermediate

Result of converter chain with JSON and Avro

A Kafka Connect sink connector uses a converter chain where the key converter is JSON and the value converter is Avro. The source sends a record with key:

{"id": 123}

and value:

{"name": "Bob", "age": 30}

What is the expected format of the key and value when the sink receives the record?

A Key as Avro bytes, value as JSON bytes

B Both key and value as Avro bytes

C Key as JSON bytes, value as Avro bytes

D Both key and value as JSON bytes

💡 Hint

Remember the converter chain applies separately to key and value.

The key converter is JSON, so the key is serialized as JSON bytes. The value converter is Avro, so the value is serialized as Avro bytes.

🔧 Debug

advanced

Identify the error in transform chain configuration

A user configures a Kafka Connect transform chain as follows:

{"transforms": "maskField", "transforms.maskField.type": "org.apache.kafka.connect.transforms.MaskField$Value", "transforms.maskField.fields": "ssn"}

But the connector fails to start with an error about transform class not found. What is the most likely cause?

A The transform chain name 'maskField' is reserved and cannot be used

B The transform class name is incorrect; it should be MaskField$Key or MaskField$Value with correct casing

C The 'fields' property is missing a comma-separated list of fields

D The connector requires a converter to be set before transforms

💡 Hint

Check the exact class name spelling and casing for the transform.

The class name is case-sensitive and must match exactly. 'MaskField$Value' is correct, but if the casing or spelling is wrong, the connector cannot find the class.

🧠 Conceptual

advanced

Understanding transform chain order impact

In Kafka Connect, you have two transforms in a chain:
1. ReplaceField to whitelist only 'user' and 'email'
2. MaskField to mask the 'email' field

If you reverse the order of these transforms, what is the impact on the final record?

A Reversing the order causes the MaskField to fail because 'email' is removed before masking

B Reversing the order has no impact; the final record is the same

C Reversing the order causes the 'user' field to be masked instead of 'email'

D Reversing the order causes the connector to crash due to invalid transform chain

💡 Hint

Think about which fields exist when each transform runs.

If ReplaceField runs first and removes all fields except 'user' and 'email', then MaskField can mask 'email'. If MaskField runs first, it tries to mask 'email' before ReplaceField removes other fields. But if ReplaceField runs second and removes 'email', the mask has no effect. Reversing the order can cause MaskField to fail if the field is missing.

❓ Predict Output

expert

Output of complex converter and transform chain

A Kafka Connect source connector uses the following configuration:

{
  "key.converter": "org.apache.kafka.connect.json.JsonConverter",
  "key.converter.schemas.enable": false,
  "value.converter": "io.confluent.connect.avro.AvroConverter",
  "value.converter.schema.registry.url": "http://localhost:8081",
  "transforms": "unwrap,dropFields",
  "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
  "transforms.dropFields.type": "org.apache.kafka.connect.transforms.ReplaceField$Value",
  "transforms.dropFields.blacklist": "metadata,internal"
}

Given a source record value:

{"before": null, "after": {"id": 1, "name": "John", "metadata": "info", "internal": "secret"}}

What will be the value passed to the sink after the transform chain?

A {"id": 1, "name": "John"}

B {"before": null, "after": {"id": 1, "name": "John"}}

C {"id": 1, "name": "John", "metadata": "info", "internal": "secret"}

D {"metadata": "info", "internal": "secret"}

💡 Hint

The unwrap transform extracts the 'after' field, then dropFields removes blacklisted fields.

The unwrap transform extracts the 'after' object, so the record becomes {"id": 1, "name": "John", "metadata": "info", "internal": "secret"}. Then the dropFields transform removes 'metadata' and 'internal', leaving {"id": 1, "name": "John"}.

Practice

1. What is the main purpose of using transform and converter chains in Kafka Connect?

easy

A. To monitor Kafka cluster health

B. To store data permanently in Kafka topics

C. To create new Kafka topics automatically

D. To change data step-by-step during processing

Solution

1. Step 1: Understand transform and converter chains

   Transform and converter chains let you modify data in stages as it moves through Kafka Connect.

2. Step 2: Identify the main purpose

   The main goal is to change or clean data step-by-step automatically before it reaches its destination.

3. Final Answer:

   To change data step-by-step during processing -> Option D

4. Quick Check:

   Transform chains = step-by-step data change [OK]

Hint: Transforms modify data stepwise in Kafka Connect [OK]

Common Mistakes:

• Confusing transform chains with topic creation
• Thinking transforms monitor cluster health
• Assuming transforms store data permanently

2. Which of the following is the correct way to list multiple transforms in Kafka Connect configuration?

easy

A. transforms=InsertField,ReplaceField

B. transforms=[InsertField, ReplaceField]

C. transforms: InsertField; ReplaceField

D. transforms = InsertField ReplaceField

Solution

1. Step 1: Recall Kafka Connect transform list syntax

   Transforms are listed as a comma-separated string without brackets or semicolons.

2. Step 2: Match the correct syntax

   transforms=InsertField,ReplaceField uses commas correctly: transforms=InsertField,ReplaceField.

3. Final Answer:

   transforms=InsertField,ReplaceField -> Option A

4. Quick Check:

   Comma-separated list = correct syntax [OK]

Hint: Use commas to separate transforms in config [OK]

Common Mistakes:

• Using brackets or semicolons instead of commas
• Adding spaces around commas incorrectly
• Using colons or spaces without commas

3. Given this Kafka Connect transform chain configuration:
transforms=InsertField,ReplaceField
transforms.InsertField.type=org.apache.kafka.connect.transforms.InsertField$Value
transforms.InsertField.static.field=source
transforms.InsertField.static.value=app1
transforms.ReplaceField.type=org.apache.kafka.connect.transforms.ReplaceField$Value
transforms.ReplaceField.renames=oldName:newName
What will happen to a record with value {"oldName":"data"}?

medium

A. The record value becomes {"newName":"data"} only

B. The record value becomes {"oldName":"data", "source":"app1"}

C. The record value becomes {"source":"app1", "newName":"data"}

D. The record value remains unchanged

Solution

1. Step 1: Apply InsertField transform

   This adds a field "source" with value "app1" to the record, so value becomes {"oldName":"data", "source":"app1"}.

2. Step 2: Apply ReplaceField transform

   This renames "oldName" to "newName", so final value is {"source":"app1", "newName":"data"}.

3. Final Answer:

   {"source":"app1", "newName":"data"} -> Option C

4. Quick Check:

   InsertField adds, ReplaceField renames = final value [OK]

Hint: Apply transforms in order listed [OK]

Common Mistakes:

• Applying transforms in wrong order
• Forgetting InsertField adds a new field
• Assuming ReplaceField deletes old field without rename

4. You configured a transform chain but get an error: "Unknown transform type". Which of these fixes the problem?

medium

A. Remove all transforms from the config

B. Correct the transform type class name in the config

C. Restart Kafka Connect without changing config

D. Increase Kafka Connect heap size

Solution

1. Step 1: Understand the error cause

   "Unknown transform type" means Kafka Connect cannot find the class name specified for the transform.

2. Step 2: Fix the transform type

   Correcting the transform type class name in the configuration ensures Kafka Connect can load it properly.

3. Final Answer:

   Correct the transform type class name in the config -> Option B

4. Quick Check:

   Fix class name to resolve unknown transform error [OK]

Hint: Check transform class names carefully [OK]

Common Mistakes:

• Ignoring typo in transform type
• Restarting without fixing config
• Assuming memory issues cause this error

5. You want to create a transform chain that first removes sensitive fields, then adds a timestamp, and finally converts the record to JSON string. Which order of transforms is correct?

hard

A. RemoveFields, InsertField (timestamp), StringConverter

B. StringConverter, RemoveFields, InsertField (timestamp)

C. InsertField (timestamp), RemoveFields, StringConverter

D. RemoveFields, StringConverter, InsertField (timestamp)

Solution

1. Step 1: Remove sensitive fields first

   Removing fields must happen before adding new data or converting formats to avoid leaking sensitive info.

2. Step 2: Add timestamp after cleaning

   InsertField adds the timestamp field next, so the record has the new info before conversion.

3. Step 3: Convert record to JSON string last

   StringConverter should be last to convert the final cleaned and enriched record into a JSON string.

4. Final Answer:

   RemoveFields, InsertField (timestamp), StringConverter -> Option A

5. Quick Check:

   Clean, enrich, then convert = correct order [OK]

Hint: Clean data first, then add fields, convert last [OK]

Common Mistakes:

• Converting to string before removing fields
• Adding fields after conversion
• Wrong transform order causing data errors