NiFi for data flow automation in Hadoop - Mini Project: Build & Apply

NiFi for Data Flow Automation

📖 Scenario: You work in a company that collects sales data from multiple stores every day. The data arrives as CSV files in a folder. You want to automate the process of reading these files, filtering out sales below a certain amount, and saving the filtered data to another folder for analysis.

🎯 Goal: Build a simple NiFi data flow that reads CSV files from an input folder, filters sales records where the amount is greater than a threshold, and writes the filtered records to an output folder.

📋 What You'll Learn

Create a NiFi flow to read CSV files from a directory

Add a configuration variable for the sales amount threshold

Use a processor to filter records based on the threshold

Write the filtered records to an output directory

💡 Why This Matters

🌍 Real World

Automating data ingestion and filtering is common in retail and many industries to prepare data for analysis without manual work.

💼 Career

NiFi skills are valuable for data engineers and analysts who build reliable data pipelines and automate data workflows.

1

Set up the input data folder and sample CSV file

Create a directory called /data/input_sales and place a CSV file named sales.csv inside it with these exact contents:
store_id,product,amount 1,apple,50 2,banana,30 3,orange,70

Hadoop

# Create the directory /data/input_sales and add sales.csv with the exact content
# Your code here

Hint

Use your terminal or file explorer to create the folder and file with the exact CSV content.

2

Configure the sales amount threshold variable

In NiFi, create a variable named sales_threshold and set its value to 40 to filter sales amounts greater than 40.

Hadoop

# Directory and file setup commands
# Create a NiFi variable called sales_threshold with value 40
# Your code here

Hint

Use NiFi's Variables tab in the Process Group to add sales_threshold with value 40.

3

Create the NiFi flow to filter sales records

Use the ListFile processor to read files from /data/input_sales, then connect it to a QueryRecord processor that filters records where amount > ${sales_threshold}. Use the correct Record Reader and Writer controllers for CSV format.

Hadoop

# Directory and file setup commands
# NiFi variable setup
# Add ListFile processor for /data/input_sales
# Add QueryRecord processor with filter amount > ${sales_threshold}
# Configure CSV Record Reader and Writer
# Your code here

Hint

Use NiFi UI to add and configure processors and controller services as described.

4

Output the filtered sales records

Add a PutFile processor connected to QueryRecord to write the filtered CSV files to /data/output_sales. Then, print the contents of the output CSV file to verify only sales with amount greater than 40 are saved.

Hadoop

# Directory and file setup commands
# NiFi variable setup
# NiFi flow setup
# Add PutFile processor to write to /data/output_sales
# Print the contents of /data/output_sales/sales.csv
# Your code here

Hint

Use NiFi to add PutFile processor and configure output folder. Then check the output file content.

Practice

1. What is the main purpose of Apache NiFi in data flow automation?

easy

A. To create visual dashboards for data analysis

B. To store large amounts of data permanently

C. To write complex SQL queries for databases

D. To automate moving and transforming data between systems

Solution

1. Step 1: Understand NiFi's role

   NiFi is designed to automate data movement and transformation between different systems.

2. Step 2: Compare options

   Options B, C, and D describe other tools or functions not related to NiFi's core purpose.

3. Final Answer:

   To automate moving and transforming data between systems -> Option D

4. Quick Check:

   NiFi = Data flow automation [OK]

Hint: NiFi moves and changes data automatically [OK]

Common Mistakes:

• Confusing NiFi with data storage tools
• Thinking NiFi is for database querying
• Assuming NiFi creates dashboards

2. Which of the following is the correct way to connect processors in NiFi?

easy

A. By writing code to link processors

B. By configuring database connections manually

C. By dragging arrows between processors in the UI

D. By exporting data to CSV files

Solution

1. Step 1: Recall NiFi interface

   NiFi uses a visual drag-and-drop interface where processors are connected by arrows.

2. Step 2: Eliminate incorrect options

   Options A, C, and D do not describe NiFi's method of connecting processors.

3. Final Answer:

   By dragging arrows between processors in the UI -> Option C

4. Quick Check:

   NiFi UI connections = drag arrows [OK]

Hint: NiFi connections are visual, not coded [OK]

Common Mistakes:

• Thinking connections require coding
• Confusing with database setup
• Assuming data export links processors

3. Consider this NiFi flow snippet:

Processor A -> Processor B -> Processor C

If Processor B fails, what happens to the data flow?

medium

A. Data flows normally through all processors

B. Data stops at Processor B and does not reach Processor C

C. Data skips Processor B and goes directly to Processor C

D. Processor A stops working immediately

Solution

1. Step 1: Understand processor dependencies

   Data flows sequentially; if Processor B fails, it blocks data from moving forward.

2. Step 2: Analyze options

   Data cannot skip processors, so Data skips Processor B and goes directly to Processor C is wrong. Processor A continues working, so Processor A stops working immediately is wrong.

3. Final Answer:

   Data stops at Processor B and does not reach Processor C -> Option B

4. Quick Check:

   Processor failure blocks downstream flow [OK]

Hint: Failed processor blocks next steps [OK]

Common Mistakes:

• Assuming data skips failed processors
• Thinking upstream processors stop immediately
• Believing flow continues despite failure

4. You created a NiFi flow but data is not moving between processors. Which is the most likely cause?

medium

A. Processors are not connected by arrows

B. NiFi server is running

C. Data format is correct

D. Processors are scheduled to run

Solution

1. Step 1: Check processor connections

   Data moves only if processors are connected by arrows in NiFi.

2. Step 2: Evaluate other options

   Options B, C, and D describe normal conditions that allow data flow, so they are not causes of no movement.

3. Final Answer:

   Processors are not connected by arrows -> Option A

4. Quick Check:

   Missing connections stop data flow [OK]

Hint: Check arrows connect processors [OK]

Common Mistakes:

• Ignoring missing connections
• Assuming server status causes no flow
• Confusing data format with flow control

5. You want to automate a data flow that reads files from a folder, filters only CSV files, and sends them to HDFS. Which NiFi processors should you use in order?

hard

A. ListFile -> RouteOnAttribute -> PutHDFS

B. PutHDFS -> ListFile -> RouteOnAttribute

C. RouteOnAttribute -> PutHDFS -> ListFile

D. ListFile -> PutHDFS -> RouteOnAttribute

Solution

1. Step 1: Identify processor roles

   ListFile reads files from a folder, RouteOnAttribute filters files by attributes (like file type), PutHDFS writes files to HDFS.

2. Step 2: Determine correct order

   First list files, then filter CSV files, then send filtered files to HDFS.

3. Final Answer:

   ListFile -> RouteOnAttribute -> PutHDFS -> Option A

4. Quick Check:

   Read -> Filter -> Write = D [OK]

Hint: Read files, filter, then write [OK]

Common Mistakes:

• Putting PutHDFS before filtering
• Filtering before listing files
• Wrong processor order