Kafka - Connect
To archive Kafka topic data daily to S3 and simultaneously enable search via Elasticsearch, which connector architecture is most appropriate?
To archive Kafka topic data daily to S3 and simultaneously enable search via Elasticsearch, which connector architecture is most appropriate?
hard🚀 Application Q9 of Q15
Step-by-Step Solution
Solution:
Step 1: Understand requirements
Data must be archived to S3 and indexed in Elasticsearch.Step 2: Evaluate connector capabilities
S3 sink connector handles storage; Elasticsearch sink connector handles indexing.Step 3: Analyze options
Use separate Kafka S3 sink and Elasticsearch sink connectors consuming the same topic uses two specialized connectors on the same topic, which is best practice. Use a single Kafka JDBC sink connector configured for both S3 and Elasticsearch is invalid as JDBC does not support S3 or Elasticsearch. Use only the Kafka Elasticsearch sink connector with S3 as a backup ignores S3 archiving. Use Kafka Connect REST API to manually push data to S3 and Elasticsearch is manual and inefficient.Final Answer:
Use separate Kafka S3 sink and Elasticsearch sink connectors consuming the same topic -> Option AQuick Check:
Separate connectors for storage and search [OK]
Quick Trick: Use dedicated connectors for S3 and Elasticsearch [OK]
Common Mistakes:
MISTAKES- Trying to combine S3 and Elasticsearch in one connector
- Ignoring the need for separate sinks
Master "Connect" in Kafka
Want More Practice?
15+ quiz questions · All difficulty levels · Free
Free Signup - Practice All QuestionsMore Kafka Quizzes