NiFi for data flow automation in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding NiFi FlowFiles

In Apache NiFi, what is a FlowFile?

A A container that holds data and attributes moving through NiFi

B A data record stored in HDFS for batch processing

C A script that automates data ingestion in NiFi

D A configuration file for NiFi processors

💡 Hint

Think about what moves inside NiFi between processors.

A FlowFile is the basic data packet in NiFi. It contains the actual data content and metadata attributes, allowing NiFi to track and route data through its flow.

❓ Predict Output

intermediate

NiFi Expression Language Output

What is the output of this NiFi Expression Language snippet if the attribute 'filename' is 'data_2024.csv'?

${filename:substring(5,9)}

A 024.

B 2024

C 202

D data

💡 Hint

Substring(start, end) extracts characters from start index up to but not including end index.

The substring from index 5 to 9 in 'data_2024.csv' is '2024'. Indexing starts at 0.

❓ data_output

advanced

NiFi Data Provenance Query Result

You run a NiFi Data Provenance query filtering events by a specific processor and time range. The query returns 3 events. What does this output represent?

A Three processors active during the time range

B Three configuration changes made to NiFi

C Three errors logged by NiFi in that time range

D Three FlowFiles processed by that processor in the time range

💡 Hint

Data Provenance tracks FlowFile events like creation, modification, and transfer.

The query output shows FlowFile events processed by the selected processor during the specified time.

🔧 Debug

advanced

Troubleshooting NiFi Processor Failure

Given this NiFi processor configuration snippet, which option explains why the processor fails to start?

Properties:
Input Directory: /data/input
File Filter: *.csv
Scheduling Strategy: Timer-driven
Run Schedule: 0 sec

A Input Directory path must be absolute, '/data/input' is relative

B File Filter '*.csv' is invalid syntax and causes failure

C Run Schedule set to 0 seconds causes invalid scheduling interval error

D Scheduling Strategy 'Timer-driven' is deprecated and causes failure

💡 Hint

Check if the run schedule value is valid for timer-driven scheduling.

Run Schedule cannot be zero seconds; it must be a positive number to schedule processor runs.

🚀 Application

expert

Designing a NiFi Flow for Real-Time Data Filtering

You need to design a NiFi flow that reads streaming data from Kafka, filters records where 'status' equals 'error', and writes them to HDFS. Which sequence of processors correctly implements this?

A ConsumeKafka -> QueryRecord (filter status=='error') -> PutHDFS

B GetFile -> UpdateAttribute (status=='error') -> PutHDFS

C ConsumeKafka -> RouteOnAttribute (status=='error') -> PutHDFS

D ListenHTTP -> RouteOnContent (status=='error') -> PutFile

💡 Hint

Consider processors that can consume Kafka and filter records by content.

ConsumeKafka reads streaming data, QueryRecord filters records by field values, and PutHDFS writes filtered data to HDFS.

Practice

1. What is the main purpose of Apache NiFi in data flow automation?

easy

A. To create visual dashboards for data analysis

B. To store large amounts of data permanently

C. To write complex SQL queries for databases

D. To automate moving and transforming data between systems

Solution

1. Step 1: Understand NiFi's role

   NiFi is designed to automate data movement and transformation between different systems.

2. Step 2: Compare options

   Options B, C, and D describe other tools or functions not related to NiFi's core purpose.

3. Final Answer:

   To automate moving and transforming data between systems -> Option D

4. Quick Check:

   NiFi = Data flow automation [OK]

Hint: NiFi moves and changes data automatically [OK]

Common Mistakes:

• Confusing NiFi with data storage tools
• Thinking NiFi is for database querying
• Assuming NiFi creates dashboards

2. Which of the following is the correct way to connect processors in NiFi?

easy

A. By writing code to link processors

B. By configuring database connections manually

C. By dragging arrows between processors in the UI

D. By exporting data to CSV files

Solution

1. Step 1: Recall NiFi interface

   NiFi uses a visual drag-and-drop interface where processors are connected by arrows.

2. Step 2: Eliminate incorrect options

   Options A, C, and D do not describe NiFi's method of connecting processors.

3. Final Answer:

   By dragging arrows between processors in the UI -> Option C

4. Quick Check:

   NiFi UI connections = drag arrows [OK]

Hint: NiFi connections are visual, not coded [OK]

Common Mistakes:

• Thinking connections require coding
• Confusing with database setup
• Assuming data export links processors

3. Consider this NiFi flow snippet:

Processor A -> Processor B -> Processor C

If Processor B fails, what happens to the data flow?

medium

A. Data flows normally through all processors

B. Data stops at Processor B and does not reach Processor C

C. Data skips Processor B and goes directly to Processor C

D. Processor A stops working immediately

Solution

1. Step 1: Understand processor dependencies

   Data flows sequentially; if Processor B fails, it blocks data from moving forward.

2. Step 2: Analyze options

   Data cannot skip processors, so Data skips Processor B and goes directly to Processor C is wrong. Processor A continues working, so Processor A stops working immediately is wrong.

3. Final Answer:

   Data stops at Processor B and does not reach Processor C -> Option B

4. Quick Check:

   Processor failure blocks downstream flow [OK]

Hint: Failed processor blocks next steps [OK]

Common Mistakes:

• Assuming data skips failed processors
• Thinking upstream processors stop immediately
• Believing flow continues despite failure

4. You created a NiFi flow but data is not moving between processors. Which is the most likely cause?

medium

A. Processors are not connected by arrows

B. NiFi server is running

C. Data format is correct

D. Processors are scheduled to run

Solution

1. Step 1: Check processor connections

   Data moves only if processors are connected by arrows in NiFi.

2. Step 2: Evaluate other options

   Options B, C, and D describe normal conditions that allow data flow, so they are not causes of no movement.

3. Final Answer:

   Processors are not connected by arrows -> Option A

4. Quick Check:

   Missing connections stop data flow [OK]

Hint: Check arrows connect processors [OK]

Common Mistakes:

• Ignoring missing connections
• Assuming server status causes no flow
• Confusing data format with flow control

5. You want to automate a data flow that reads files from a folder, filters only CSV files, and sends them to HDFS. Which NiFi processors should you use in order?

hard

A. ListFile -> RouteOnAttribute -> PutHDFS

B. PutHDFS -> ListFile -> RouteOnAttribute

C. RouteOnAttribute -> PutHDFS -> ListFile

D. ListFile -> PutHDFS -> RouteOnAttribute

Solution

1. Step 1: Identify processor roles

   ListFile reads files from a folder, RouteOnAttribute filters files by attributes (like file type), PutHDFS writes files to HDFS.

2. Step 2: Determine correct order

   First list files, then filter CSV files, then send filtered files to HDFS.

3. Final Answer:

   ListFile -> RouteOnAttribute -> PutHDFS -> Option A

4. Quick Check:

   Read -> Filter -> Write = D [OK]

Hint: Read files, filter, then write [OK]

Common Mistakes:

• Putting PutHDFS before filtering
• Filtering before listing files
• Wrong processor order