Flume for log collection in Hadoop - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to specify the Flume source type for collecting logs.

Hadoop

agent.sources.src.type = [1]

Drag options to blanks, or click blank then click option'

Aavro

Bexec

Cfile

Dhttp

Attempts:

3 left

💡 Hint

Common Mistakes

Choosing 'file' which is not a valid Flume source type.

Choosing 'avro' which is for Avro RPC source, not file logs.

The exec source type is used in Flume to collect logs by running an external command such as tail -F /path/to/logfile.

2fill in blank

medium

Complete the code to set the Flume sink type to write logs into HDFS.

Hadoop

agent.sinks.sink.type = [1]

Drag options to blanks, or click blank then click option'

Alogger

Bavro

Chdfs

Dfile_roll

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'logger' which only logs to console.

Choosing 'file_roll' which writes to local files, not HDFS.

The hdfs sink writes the collected data into HDFS storage.

3fill in blank

hard

Fix the error in the Flume configuration to correctly bind the source to the channel.

Hadoop

agent.sources.src.channels = [1]

Drag options to blanks, or click blank then click option'

Ach1

BsourceChannel

Cchannel-1

Dchannel1

Attempts:

3 left

💡 Hint

Common Mistakes

Using a channel name that does not match the defined channel.

Including special characters in the channel name.

The channel name must match the channel defined elsewhere. 'channel1' is the correct channel name.

4fill in blank

hard

Fill both blanks to configure the Flume agent's memory channel with capacity and transaction capacity.

Hadoop

agent.channels.memChannel.type = [1]
agent.channels.memChannel.capacity = [2]

Drag options to blanks, or click blank then click option'

Amemory

Bfile

C10000

D5000

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'file' as channel type instead of 'memory'.

Setting capacity to a non-numeric value.

The memory channel type is 'memory'. Capacity is set to 10000 events.

5fill in blank

hard

Fill all three blanks to configure the Flume HDFS sink with path, file prefix, and roll interval.

Hadoop

agent.sinks.hdfsSink.hdfs.path = [1]
agent.sinks.hdfsSink.hdfs.filePrefix = [2]
agent.sinks.hdfsSink.hdfs.rollInterval = [3]

Drag options to blanks, or click blank then click option'

A/logs/flume

Blogfile

C60

D/data/input

Attempts:

3 left

💡 Hint

Common Mistakes

Using a local path instead of HDFS path.

Setting roll interval to zero or negative number.

The HDFS path is '/logs/flume', file prefix is 'logfile', and roll interval is 60 seconds.

Practice

1. What is the main purpose of Apache Flume in Hadoop ecosystems?

easy

A. To perform complex SQL queries on Hadoop data

B. To manage user permissions in Hadoop clusters

C. To collect and transport large amounts of log data from various sources to storage systems

D. To visualize data stored in HDFS

Solution

1. Step 1: Understand Flume's role

   Flume is designed to collect and move log data efficiently from many sources.

2. Step 2: Identify Flume's main function

   It transports this data reliably to storage like HDFS for further processing.

3. Final Answer:

   To collect and transport large amounts of log data from various sources to storage systems -> Option C

4. Quick Check:

   Flume = log data collection and transport [OK]

Hint: Flume moves logs from sources to storage reliably [OK]

Common Mistakes:

• Confusing Flume with query engines
• Thinking Flume manages permissions
• Assuming Flume visualizes data

2. Which of the following is the correct way to define a source in a Flume configuration file?

easy

A. agent.channels = source1

B. agent.sinks = source1

C. agent.sources.source1.type = sink

D. agent.sources = source1

Solution

1. Step 1: Recall Flume config syntax for sources

   Sources are defined with 'agent.sources' followed by source names.

2. Step 2: Check each option

   agent.sources = source1 correctly assigns 'source1' as a source. Options A and B assign 'source1' incorrectly as channel or sink. agent.sources.source1.type = sink wrongly sets source type to sink.

3. Final Answer:

   agent.sources = source1 -> Option D

4. Quick Check:

   Sources start with 'agent.sources' [OK]

Hint: Sources start with 'agent.sources' in config [OK]

Common Mistakes:

• Mixing sources with sinks or channels
• Wrongly assigning types in config
• Using incorrect property names

3. Given this Flume configuration snippet:

agent.sources = src1
agent.channels = ch1
agent.sinks = sink1

agent.sources.src1.type = exec
agent.sources.src1.command = tail -F /var/log/syslog
agent.channels.ch1.type = memory
agent.sinks.sink1.type = hdfs

agent.sources.src1.channels = ch1
agent.sinks.sink1.channel = ch1

What will happen when Flume runs?

medium

A. Flume will collect syslog data and write it to HDFS using a memory channel

B. Flume will fail because the channel type is invalid

C. Flume will collect data but not send it anywhere

D. Flume will write logs directly to local disk without using HDFS

Solution

1. Step 1: Analyze source and command

   The source 'src1' uses 'exec' type to run 'tail -F /var/log/syslog', collecting live syslog data.

2. Step 2: Check channel and sink setup

   The channel 'ch1' is memory type, and sink 'sink1' writes to HDFS. Source and sink both use channel 'ch1'.

3. Final Answer:

   Flume will collect syslog data and write it to HDFS using a memory channel -> Option A

4. Quick Check:

   Exec source + memory channel + HDFS sink = data flows correctly [OK]

Hint: Exec source tails logs; memory channel buffers; sink writes to HDFS [OK]

Common Mistakes:

• Thinking memory channel is invalid
• Assuming data is not sent to sink
• Confusing sink type with local disk

4. You have this Flume config snippet:

agent.sources = src1
agent.channels = ch1
agent.sinks = sink1

agent.sources.src1.type = exec
agent.sources.src1.command = tail -F /var/log/syslog
agent.channels.ch1.type = memory
agent.sinks.sink1.type = hdfs

agent.sources.src1.channels = ch1
agent.sinks.sink1.channel = ch2

What is the problem in this configuration?

medium

A. The sink is connected to a channel 'ch2' which is not defined

B. The source command is invalid

C. The channel type 'memory' is not supported

D. The source is not assigned to any channel

Solution

1. Step 1: Check channel definitions

   Only channel 'ch1' is defined; 'ch2' is missing from config.

2. Step 2: Check sink channel assignment

   Sink 'sink1' is assigned to channel 'ch2', which does not exist, causing an error.

3. Final Answer:

   The sink is connected to a channel 'ch2' which is not defined -> Option A

4. Quick Check:

   Sink channel must be defined and match existing channel [OK]

Hint: Sink channel must exist in config [OK]

Common Mistakes:

• Ignoring missing channel definitions
• Assuming source command is wrong
• Thinking memory channel is unsupported

5. You want to configure Flume to collect logs from two different servers and write them into separate directories in HDFS. Which configuration approach is best?

hard

A. Use one agent with one source and one sink writing to a single HDFS directory

B. Create two agents, each with its own source, channel, and sink writing to different HDFS paths

C. Use one agent with multiple sinks but share the same channel for both sources

D. Configure two sources in one agent but only one sink writing to HDFS

Solution

1. Step 1: Understand requirement for separate directories

   Each server's logs must go to a different HDFS directory, so sinks must be separate.

2. Step 2: Design configuration for isolation

   Two agents with separate sources, channels, and sinks allow independent paths and better fault isolation.

3. Step 3: Evaluate other options

   Sharing channels or sinks mixes data and cannot guarantee separate directories.

4. Final Answer:

   Create two agents, each with its own source, channel, and sink writing to different HDFS paths -> Option B

5. Quick Check:

   Separate agents isolate data paths for different servers [OK]

Hint: Use separate agents for separate log paths [OK]

Common Mistakes:

• Sharing channels causing mixed data
• Using one sink for multiple sources
• Not isolating sinks for different directories