NiFi for data flow automation in Hadoop - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create a NiFi processor that reads data from a file.

Hadoop

processor = ProcessSessionFactory().createSession().get[1]('GetFile')

Drag options to blanks, or click blank then click option'

AprocessorConfig

BprocessorNode

CprocessorFactory

Dprocessor

Attempts:

3 left

💡 Hint

Common Mistakes

Using incorrect attribute names like processorNode or processorConfig.

Confusing factory methods with processor instances.

The processor object is created by calling getProcessor or similar method. Here, processor is the correct attribute to get the processor instance.

2fill in blank

medium

Complete the code to set the input directory for the GetFile processor.

Hadoop

processor.setProperty('Input Directory', [1])

Drag options to blanks, or click blank then click option'

A'/data/input'

B'input_dir'

C'/var/logs'

D'output_dir'

Attempts:

3 left

💡 Hint

Common Mistakes

Using output directory as input directory.

Using relative paths instead of absolute paths.

The input directory should be a valid path where files are read from. '/data/input' is a typical example.

3fill in blank

hard

Fix the error in the code to start the NiFi data flow.

Hadoop

flowController.[1]()

Drag options to blanks, or click blank then click option'

Ainitialize

Bstop

Cstart

Dshutdown

Attempts:

3 left

💡 Hint

Common Mistakes

Calling stop or shutdown instead of start.

Using initialize which does not start the flow.

To start the NiFi data flow, the start() method must be called on the flow controller.

4fill in blank

hard

Fill both blanks to create a processor that routes data based on success or failure.

Hadoop

processor = session.createProcessor('RouteOnAttribute')
processor.setProperty('Routing Strategy', [1])
processor.setProperty('Success Relationship', [2])

Drag options to blanks, or click blank then click option'

A'Route to Property name'

B'success'

C'failure'

D'Route to Expression Language'

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'Route to Expression Language' instead of 'Route to Property name'.

Confusing success and failure relationship names.

The routing strategy should be set to 'Route to Property name' to route based on attributes. The success relationship is named 'success'.

5fill in blank

hard

Fill all three blanks to create a data flow that reads, transforms, and writes data.

Hadoop

getFile = session.createProcessor('GetFile')
getFile.setProperty('Input Directory', [1])
transform = session.createProcessor([2])
putFile = session.createProcessor('PutFile')
putFile.setProperty('Output Directory', [3])

Drag options to blanks, or click blank then click option'

A'/data/input'

B'UpdateAttribute'

C'/data/output'

D'ReplaceText'

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'UpdateAttribute' instead of 'ReplaceText' for transformation.

Mixing up input and output directories.

The input directory is '/data/input'. The transform processor is 'ReplaceText' to modify data. The output directory is '/data/output'.

Practice

1. What is the main purpose of Apache NiFi in data flow automation?

easy

A. To create visual dashboards for data analysis

B. To store large amounts of data permanently

C. To write complex SQL queries for databases

D. To automate moving and transforming data between systems

Solution

1. Step 1: Understand NiFi's role

   NiFi is designed to automate data movement and transformation between different systems.

2. Step 2: Compare options

   Options B, C, and D describe other tools or functions not related to NiFi's core purpose.

3. Final Answer:

   To automate moving and transforming data between systems -> Option D

4. Quick Check:

   NiFi = Data flow automation [OK]

Hint: NiFi moves and changes data automatically [OK]

Common Mistakes:

• Confusing NiFi with data storage tools
• Thinking NiFi is for database querying
• Assuming NiFi creates dashboards

2. Which of the following is the correct way to connect processors in NiFi?

easy

A. By writing code to link processors

B. By configuring database connections manually

C. By dragging arrows between processors in the UI

D. By exporting data to CSV files

Solution

1. Step 1: Recall NiFi interface

   NiFi uses a visual drag-and-drop interface where processors are connected by arrows.

2. Step 2: Eliminate incorrect options

   Options A, C, and D do not describe NiFi's method of connecting processors.

3. Final Answer:

   By dragging arrows between processors in the UI -> Option C

4. Quick Check:

   NiFi UI connections = drag arrows [OK]

Hint: NiFi connections are visual, not coded [OK]

Common Mistakes:

• Thinking connections require coding
• Confusing with database setup
• Assuming data export links processors

3. Consider this NiFi flow snippet:

Processor A -> Processor B -> Processor C

If Processor B fails, what happens to the data flow?

medium

A. Data flows normally through all processors

B. Data stops at Processor B and does not reach Processor C

C. Data skips Processor B and goes directly to Processor C

D. Processor A stops working immediately

Solution

1. Step 1: Understand processor dependencies

   Data flows sequentially; if Processor B fails, it blocks data from moving forward.

2. Step 2: Analyze options

   Data cannot skip processors, so Data skips Processor B and goes directly to Processor C is wrong. Processor A continues working, so Processor A stops working immediately is wrong.

3. Final Answer:

   Data stops at Processor B and does not reach Processor C -> Option B

4. Quick Check:

   Processor failure blocks downstream flow [OK]

Hint: Failed processor blocks next steps [OK]

Common Mistakes:

• Assuming data skips failed processors
• Thinking upstream processors stop immediately
• Believing flow continues despite failure

4. You created a NiFi flow but data is not moving between processors. Which is the most likely cause?

medium

A. Processors are not connected by arrows

B. NiFi server is running

C. Data format is correct

D. Processors are scheduled to run

Solution

1. Step 1: Check processor connections

   Data moves only if processors are connected by arrows in NiFi.

2. Step 2: Evaluate other options

   Options B, C, and D describe normal conditions that allow data flow, so they are not causes of no movement.

3. Final Answer:

   Processors are not connected by arrows -> Option A

4. Quick Check:

   Missing connections stop data flow [OK]

Hint: Check arrows connect processors [OK]

Common Mistakes:

• Ignoring missing connections
• Assuming server status causes no flow
• Confusing data format with flow control

5. You want to automate a data flow that reads files from a folder, filters only CSV files, and sends them to HDFS. Which NiFi processors should you use in order?

hard

A. ListFile -> RouteOnAttribute -> PutHDFS

B. PutHDFS -> ListFile -> RouteOnAttribute

C. RouteOnAttribute -> PutHDFS -> ListFile

D. ListFile -> PutHDFS -> RouteOnAttribute

Solution

1. Step 1: Identify processor roles

   ListFile reads files from a folder, RouteOnAttribute filters files by attributes (like file type), PutHDFS writes files to HDFS.

2. Step 2: Determine correct order

   First list files, then filter CSV files, then send filtered files to HDFS.

3. Final Answer:

   ListFile -> RouteOnAttribute -> PutHDFS -> Option A

4. Quick Check:

   Read -> Filter -> Write = D [OK]

Hint: Read files, filter, then write [OK]

Common Mistakes:

• Putting PutHDFS before filtering
• Filtering before listing files
• Wrong processor order