Hive architecture in Hadoop - Step-by-Step Execution

Concept Flow - Hive architecture

User submits Hive query

↓

Hive Driver parses query

↓

Compiler converts query to execution plan

↓

Metastore provides metadata

↓

Execution Engine runs plan

↓

Hadoop executes MapReduce or Tez jobs

↓

Results returned to User

This flow shows how a Hive query moves from user input through parsing, planning, execution, and finally returning results using Hadoop components.

Execution Sample

Hadoop

User submits: SELECT * FROM sales WHERE year=2023;
Driver parses query
Compiler creates execution plan
Metastore provides table info
Execution Engine runs plan
Hadoop runs MapReduce job
Results returned

This example traces a simple Hive query execution from submission to result.

Execution Table

Step	Action	Component	Details	Output
1	Submit query	User	SELECT * FROM sales WHERE year=2023;	Query string
2	Parse query	Driver	Check syntax and semantics	Parsed query tree
3	Compile query	Compiler	Create execution plan (logical and physical)	Execution plan
4	Fetch metadata	Metastore	Get table schema and partitions	Metadata info
5	Execute plan	Execution Engine	Run MapReduce or Tez jobs on Hadoop	Job execution
6	Process data	Hadoop	Map and Reduce tasks process data	Intermediate results
7	Return results	Execution Engine	Collect and send results to user	Query results
8	End	System	Query completed	Results delivered

💡 Query execution ends after results are returned to the user.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	After Step 7	Final
query	SELECT * FROM sales WHERE year=2023;	Parsed query tree	Execution plan	Metadata info	Job execution started	Results collected	Results delivered

Key Moments - 3 Insights

Why does Hive need the Metastore during query execution?

The Metastore provides the schema and partition info needed to plan and run the query correctly, as shown in step 4 of the execution_table.

What happens if the query syntax is wrong?

The Driver fails at step 2 during parsing, so the query does not proceed to compilation or execution.

How does Hive execute the query on data?

Hive converts the query into MapReduce or Tez jobs run by Hadoop, as shown in steps 5 and 6.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does Hive get the table schema?

A Step 4

B Step 2

C Step 5

D Step 7

💡 Hint

Check the 'Fetch metadata' action in the execution_table at step 4.

According to variable_tracker, what is the state of 'query' after step 3?

A Raw query string

B Execution plan

C Parsed query tree

D Results delivered

💡 Hint

Look at the 'After Step 3' column for 'query' in variable_tracker.

If the query syntax is incorrect, which step will fail according to execution_table?

A Step 1

B Step 5

C Step 2

D Step 7

💡 Hint

Parsing happens at step 2; syntax errors stop execution there.

Concept Snapshot

Hive architecture flow:
User submits query -> Driver parses -> Compiler plans -> Metastore provides metadata -> Execution Engine runs jobs on Hadoop -> Results returned.
Metastore is key for schema info.
Execution uses MapReduce or Tez.
Driver stops on syntax errors.

Full Transcript

Hive architecture starts when a user submits a query. The Driver parses the query to check syntax and semantics. Then the Compiler creates an execution plan. The Metastore provides metadata like table schema and partitions. The Execution Engine runs the plan by launching MapReduce or Tez jobs on Hadoop. These jobs process data and produce results. Finally, results are returned to the user. If the query has syntax errors, parsing fails and execution stops early. The Metastore is essential for providing table information needed to run queries correctly.