NiFi for data flow automation in Hadoop - Step-by-Step Execution

Concept Flow - NiFi for data flow automation

Start: Data Source

↓

NiFi Ingest Processor

↓

Data Flow Controller

↓

Routing & Transformation Processors

↓

Data Destination

↓

Monitoring & Logging

Data flows from source through NiFi processors that ingest, route, transform, and deliver data automatically.

Execution Sample

Hadoop

GenerateFlowFile -> LogAttribute -> UpdateAttribute -> PutFile

This NiFi flow creates a file, logs its attributes, updates attributes, then saves the file to disk.

Execution Table

Step	Processor	Action	Input	Output	Notes
1	GenerateFlowFile	Create flow file	None	FlowFile with content	Starts data flow with a file
2	LogAttribute	Log attributes	FlowFile	FlowFile unchanged	Attributes logged for monitoring
3	UpdateAttribute	Modify attributes	FlowFile	FlowFile with updated attributes	Adds or changes metadata
4	PutFile	Write file to disk	FlowFile	File saved	Data delivered to destination
5	End	Flow complete	None	None	No more processors, flow ends

💡 Flow ends after PutFile processor writes data to disk

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4
FlowFile Content	None	Created	Unchanged	Unchanged	Saved to disk
FlowFile Attributes	{}	Default set	Logged	{"updated": true}	{"updated": true}

Key Moments - 3 Insights

Why does the FlowFile content stay unchanged after LogAttribute?

LogAttribute only reads and logs metadata; it does not modify the FlowFile content as shown in step 2 of the execution_table.

How does UpdateAttribute affect the FlowFile?

UpdateAttribute changes the FlowFile's metadata attributes but does not alter the actual content, as seen in step 3 of the execution_table.

What causes the data flow to end?

The flow ends after PutFile writes the FlowFile content to disk, completing the data delivery, as noted in step 4 and the exit_note.

Visual Quiz - 3 Questions

Look at the execution_table, what is the state of FlowFile content after step 2?

A Deleted

B Created and unchanged

C Modified by LogAttribute

D Empty

💡 Hint

Refer to the 'Output' column in step 2 of execution_table showing 'FlowFile unchanged'

At which step does the FlowFile attributes get updated?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Check the 'Action' and 'Notes' columns in step 3 of execution_table

If PutFile fails, what happens to the data flow according to the table?

A Flow stops without saving data

B Flow continues to next processor

C Flow ends successfully

D Flow restarts from GenerateFlowFile

💡 Hint

Exit note says flow ends after PutFile writes data; failure means no save and no next step

Concept Snapshot

NiFi automates data flow by connecting processors.
Each processor performs a task: ingest, log, update, or deliver data.
FlowFiles carry data and metadata through processors.
Processors run in sequence until data reaches destination.
Failures stop flow or trigger retries.
Visualize flow as steps from source to sink.

Full Transcript

NiFi automates data flow by moving data through connected processors. It starts with a data source, then processors like GenerateFlowFile create data. LogAttribute reads metadata without changing content. UpdateAttribute modifies metadata. PutFile saves data to disk. The flow ends after data is saved. Variables like FlowFile content and attributes change step-by-step. Beginners often wonder why some processors don't change content; this is because some only log or update metadata. The flow stops after the last processor completes its task. Understanding each step helps visualize how NiFi manages data automatically.

Practice

1. What is the main purpose of Apache NiFi in data flow automation?

easy

A. To create visual dashboards for data analysis

B. To store large amounts of data permanently

C. To write complex SQL queries for databases

D. To automate moving and transforming data between systems

Solution

1. Step 1: Understand NiFi's role

   NiFi is designed to automate data movement and transformation between different systems.

2. Step 2: Compare options

   Options B, C, and D describe other tools or functions not related to NiFi's core purpose.

3. Final Answer:

   To automate moving and transforming data between systems -> Option D

4. Quick Check:

   NiFi = Data flow automation [OK]

Hint: NiFi moves and changes data automatically [OK]

Common Mistakes:

• Confusing NiFi with data storage tools
• Thinking NiFi is for database querying
• Assuming NiFi creates dashboards

2. Which of the following is the correct way to connect processors in NiFi?

easy

A. By writing code to link processors

B. By configuring database connections manually

C. By dragging arrows between processors in the UI

D. By exporting data to CSV files

Solution

1. Step 1: Recall NiFi interface

   NiFi uses a visual drag-and-drop interface where processors are connected by arrows.

2. Step 2: Eliminate incorrect options

   Options A, C, and D do not describe NiFi's method of connecting processors.

3. Final Answer:

   By dragging arrows between processors in the UI -> Option C

4. Quick Check:

   NiFi UI connections = drag arrows [OK]

Hint: NiFi connections are visual, not coded [OK]

Common Mistakes:

• Thinking connections require coding
• Confusing with database setup
• Assuming data export links processors

3. Consider this NiFi flow snippet:

Processor A -> Processor B -> Processor C

If Processor B fails, what happens to the data flow?

medium

A. Data flows normally through all processors

B. Data stops at Processor B and does not reach Processor C

C. Data skips Processor B and goes directly to Processor C

D. Processor A stops working immediately

Solution

1. Step 1: Understand processor dependencies

   Data flows sequentially; if Processor B fails, it blocks data from moving forward.

2. Step 2: Analyze options

   Data cannot skip processors, so Data skips Processor B and goes directly to Processor C is wrong. Processor A continues working, so Processor A stops working immediately is wrong.

3. Final Answer:

   Data stops at Processor B and does not reach Processor C -> Option B

4. Quick Check:

   Processor failure blocks downstream flow [OK]

Hint: Failed processor blocks next steps [OK]

Common Mistakes:

• Assuming data skips failed processors
• Thinking upstream processors stop immediately
• Believing flow continues despite failure

4. You created a NiFi flow but data is not moving between processors. Which is the most likely cause?

medium

A. Processors are not connected by arrows

B. NiFi server is running

C. Data format is correct

D. Processors are scheduled to run

Solution

1. Step 1: Check processor connections

   Data moves only if processors are connected by arrows in NiFi.

2. Step 2: Evaluate other options

   Options B, C, and D describe normal conditions that allow data flow, so they are not causes of no movement.

3. Final Answer:

   Processors are not connected by arrows -> Option A

4. Quick Check:

   Missing connections stop data flow [OK]

Hint: Check arrows connect processors [OK]

Common Mistakes:

• Ignoring missing connections
• Assuming server status causes no flow
• Confusing data format with flow control

5. You want to automate a data flow that reads files from a folder, filters only CSV files, and sends them to HDFS. Which NiFi processors should you use in order?

hard

A. ListFile -> RouteOnAttribute -> PutHDFS

B. PutHDFS -> ListFile -> RouteOnAttribute

C. RouteOnAttribute -> PutHDFS -> ListFile

D. ListFile -> PutHDFS -> RouteOnAttribute

Solution

1. Step 1: Identify processor roles

   ListFile reads files from a folder, RouteOnAttribute filters files by attributes (like file type), PutHDFS writes files to HDFS.

2. Step 2: Determine correct order

   First list files, then filter CSV files, then send filtered files to HDFS.

3. Final Answer:

   ListFile -> RouteOnAttribute -> PutHDFS -> Option A

4. Quick Check:

   Read -> Filter -> Write = D [OK]

Hint: Read files, filter, then write [OK]

Common Mistakes:

• Putting PutHDFS before filtering
• Filtering before listing files
• Wrong processor order