Flume for log collection in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the primary role of a Flume agent in log collection?

In Apache Flume, what does an agent mainly do when collecting logs?

A It converts logs into SQL queries for database insertion.

B It collects data from sources, processes it, and sends it to sinks.

C It only monitors the health of the Hadoop cluster.

D It stores logs permanently in HDFS without processing.

💡 Hint

Think about the flow of data from where logs are generated to where they are stored.

A Flume agent acts as a container for sources, channels, and sinks. It collects data from sources, optionally processes or buffers it in channels, and sends it to sinks like HDFS or HBase.

❓ query_result

intermediate

What output does this Flume configuration produce?

Given this Flume configuration snippet, what is the expected output destination for the logs?

Hadoop

agent.sources = source1
agent.channels = channel1
agent.sinks = sink1

agent.sources.source1.type = exec
agent.sources.source1.command = tail -F /var/log/syslog

agent.channels.channel1.type = memory
agent.channels.channel1.capacity = 1000

agent.sinks.sink1.type = hdfs
agent.sinks.sink1.hdfs.path = hdfs://namenode/flume/logs/
agent.sinks.sink1.hdfs.filePrefix = syslog-

agent.sources.source1.channels = channel1
agent.sinks.sink1.channel = channel1

A Logs are sent to a Kafka topic named 'syslog'.

B Logs are collected from /var/log/syslog and stored locally on the agent machine.

C Logs are continuously collected from /var/log/syslog and stored in HDFS under /flume/logs/ with prefix 'syslog-'.

D Logs are discarded after being read from /var/log/syslog.

💡 Hint

Look at the sink type and path settings.

The sink is configured as HDFS with a specified path and file prefix, so logs collected from the source command are stored in HDFS with the given prefix.

📝 Syntax

advanced

Identify the syntax error in this Flume configuration snippet

Which option contains a syntax error that will prevent the Flume agent from starting?

Hadoop

agent.sources = source1
agent.channels = channel1
agent.sinks = sink1

agent.sources.source1.type = exec
agent.sources.source1.command = tail -F /var/log/syslog

agent.channels.channel1.type = memory
agent.channels.channel1.capacity = 1000

agent.sinks.sink1.type = hdfs
agent.sinks.sink1.hdfs.path = hdfs://namenode/flume/logs/
agent.sinks.sink1.hdfs.filePrefix = syslog-

agent.sources.source1.channels = channel1
agent.sinks.sink1.channel = channel1

A agent.sinks.sink1.channel = channel1

B agent.sources.source1.channels = channel1

C agent.channels.channel1.capacity = 1000

D agent.sinks.sink1.channels = channel1

💡 Hint

Check the property names for sinks connecting to channels.

The correct property to link a sink to a channel is 'agent.sinks.sink1.channel' (singular). Using 'channels' (plural) is invalid and causes a syntax error.

❓ optimization

advanced

How to optimize Flume for high log throughput?

You want to improve Flume's performance to handle a large volume of logs with minimal delay. Which option is the best optimization?

A Use a memory channel instead of a file channel to reduce disk I/O latency.

B Disable batch processing to send each event immediately.

C Set the source type to 'avro' to compress logs before sending.

D Increase the number of sinks but keep a single channel to avoid complexity.

💡 Hint

Think about buffering and speed trade-offs.

Memory channels provide faster buffering than file channels because they avoid disk I/O, improving throughput. However, they risk data loss on failure. Increasing sinks or disabling batching can hurt performance. 'Avro' source type is unrelated to compression here.

🔧 Debug

expert

Why does this Flume agent fail to deliver logs to HDFS?

Given this Flume agent configuration, logs are not appearing in HDFS. What is the most likely cause?

Hadoop

agent.sources = source1
agent.channels = channel1
agent.sinks = sink1

agent.sources.source1.type = exec
agent.sources.source1.command = tail -F /var/log/syslog

agent.channels.channel1.type = file
agent.channels.channel1.checkpointDir = /tmp/flume/checkpoint
agent.channels.channel1.dataDirs = /tmp/flume/data

agent.sinks.sink1.type = hdfs
agent.sinks.sink1.hdfs.path = hdfs://namenode/flume/logs/
agent.sinks.sink1.hdfs.filePrefix = syslog-

agent.sources.source1.channels = channel1
agent.sinks.sink1.channel = channel1

A The file channel directories (/tmp/flume/checkpoint and /tmp/flume/data) do not have proper write permissions.

B The source command 'tail -F /var/log/syslog' is incorrect and does not produce output.

C The sink type 'hdfs' is deprecated and should be replaced with 'hdfs_sink'.

D The channel type 'file' is incompatible with the exec source.

💡 Hint

Check file system permissions for channel directories.

File channels require writable directories for checkpoint and data storage. If permissions are missing, the channel cannot buffer events, causing delivery failure. The source command is valid, 'hdfs' sink type is correct, and file channel works with exec source.