Hive architecture in Hadoop - Step-by-Step Execution

Concept Flow - Hive architecture

User submits Hive query

↓

Hive Driver parses query

↓

Compiler converts query to execution plan

↓

Metastore provides metadata

↓

Execution Engine runs plan

↓

Hadoop executes MapReduce or Tez jobs

↓

Results returned to User

This flow shows how a Hive query moves from user input through parsing, planning, execution, and finally returning results using Hadoop components.

Execution Sample

Hadoop

User submits: SELECT * FROM sales WHERE year=2023;
Driver parses query
Compiler creates execution plan
Metastore provides table info
Execution Engine runs plan
Hadoop runs MapReduce job
Results returned

This example traces a simple Hive query execution from submission to result.

Execution Table

Step	Action	Component	Details	Output
1	Submit query	User	SELECT * FROM sales WHERE year=2023;	Query string
2	Parse query	Driver	Check syntax and semantics	Parsed query tree
3	Compile query	Compiler	Create execution plan (logical and physical)	Execution plan
4	Fetch metadata	Metastore	Get table schema and partitions	Metadata info
5	Execute plan	Execution Engine	Run MapReduce or Tez jobs on Hadoop	Job execution
6	Process data	Hadoop	Map and Reduce tasks process data	Intermediate results
7	Return results	Execution Engine	Collect and send results to user	Query results
8	End	System	Query completed	Results delivered

💡 Query execution ends after results are returned to the user.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	After Step 7	Final
query	SELECT * FROM sales WHERE year=2023;	Parsed query tree	Execution plan	Metadata info	Job execution started	Results collected	Results delivered

Key Moments - 3 Insights

Why does Hive need the Metastore during query execution?

The Metastore provides the schema and partition info needed to plan and run the query correctly, as shown in step 4 of the execution_table.

What happens if the query syntax is wrong?

The Driver fails at step 2 during parsing, so the query does not proceed to compilation or execution.

How does Hive execute the query on data?

Hive converts the query into MapReduce or Tez jobs run by Hadoop, as shown in steps 5 and 6.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does Hive get the table schema?

A Step 4

B Step 2

C Step 5

D Step 7

💡 Hint

Check the 'Fetch metadata' action in the execution_table at step 4.

According to variable_tracker, what is the state of 'query' after step 3?

A Raw query string

B Execution plan

C Parsed query tree

D Results delivered

💡 Hint

Look at the 'After Step 3' column for 'query' in variable_tracker.

If the query syntax is incorrect, which step will fail according to execution_table?

A Step 1

B Step 5

C Step 2

D Step 7

💡 Hint

Parsing happens at step 2; syntax errors stop execution there.

Concept Snapshot

Hive architecture flow:
User submits query -> Driver parses -> Compiler plans -> Metastore provides metadata -> Execution Engine runs jobs on Hadoop -> Results returned.
Metastore is key for schema info.
Execution uses MapReduce or Tez.
Driver stops on syntax errors.

Full Transcript

Hive architecture starts when a user submits a query. The Driver parses the query to check syntax and semantics. Then the Compiler creates an execution plan. The Metastore provides metadata like table schema and partitions. The Execution Engine runs the plan by launching MapReduce or Tez jobs on Hadoop. These jobs process data and produce results. Finally, results are returned to the user. If the query has syntax errors, parsing fails and execution stops early. The Metastore is essential for providing table information needed to run queries correctly.

Practice

1. What is the main role of the Hive Metastore in Hive architecture?

easy

A. It stores the actual data files in HDFS

B. It executes the SQL queries on Hadoop data

C. It manages user authentication and security

D. It stores metadata about tables and partitions

Solution

1. Step 1: Understand the components of Hive architecture

   Hive architecture includes Metastore, Driver, Compiler, Execution Engine, and HDFS storage.

2. Step 2: Identify the role of Metastore

   The Metastore stores metadata like table definitions, schema, and partition info, not the actual data.

3. Final Answer:

   It stores metadata about tables and partitions -> Option D

4. Quick Check:

   Metastore = Metadata storage [OK]

Hint: Metastore always holds metadata, not data itself [OK]

Common Mistakes:

• Confusing Metastore with data storage
• Thinking Metastore runs queries
• Mixing Metastore with security management

2. Which of the following is the correct sequence of components involved when a Hive query is executed?

easy

A. Driver -> Compiler -> Metastore -> Execution Engine

B. Compiler -> Driver -> Execution Engine -> Metastore

C. Driver -> Compiler -> Execution Engine -> Metastore

D. Metastore -> Driver -> Compiler -> Execution Engine

Solution

1. Step 1: Recall the Hive query execution flow

   The Driver receives the query, then the Compiler parses and plans it.

2. Step 2: Understand component order

   The Compiler consults the Metastore for metadata, then the Execution Engine runs the tasks.

3. Final Answer:

   Driver -> Compiler -> Metastore -> Execution Engine -> Option A

4. Quick Check:

   Execution flow = Driver, Compiler, Metastore, Execution Engine [OK]

Hint: Driver starts, Compiler plans, Metastore provides metadata, then Execution Engine runs [OK]

Common Mistakes:

• Mixing order of Compiler and Driver
• Placing Metastore after Execution Engine
• Ignoring Metastore's role in query planning

3. Given the following Hive query execution steps, which component is responsible for converting the query into a directed acyclic graph (DAG) of tasks?

SELECT * FROM sales WHERE year = 2023;

medium

A. Driver

B. Metastore

C. Compiler

D. Execution Engine

Solution

1. Step 1: Identify the role of Compiler in Hive

   The Compiler parses the query and creates a logical plan, then converts it into a DAG of tasks.

2. Step 2: Differentiate from other components

   The Driver manages the session, Metastore stores metadata, and Execution Engine runs the DAG tasks.

3. Final Answer:

   Compiler -> Option C

4. Quick Check:

   Compiler creates DAG = C [OK]

Hint: Compiler turns query into task graph (DAG) [OK]

Common Mistakes:

• Thinking Driver creates the DAG
• Confusing Metastore with query planning
• Assuming Execution Engine creates the DAG

4. A Hive query fails with an error saying metadata not found for a table. Which component is most likely causing this error?

medium

A. Metastore

B. Compiler

C. Driver

D. Execution Engine

Solution

1. Step 1: Understand the error context

   Metadata not found means Hive cannot find table schema or info.

2. Step 2: Identify responsible component

   The Metastore stores all metadata; if missing, it causes this error.

3. Final Answer:

   Metastore -> Option A

4. Quick Check:

   Metadata error = Metastore issue [OK]

Hint: Metadata errors point to Metastore problems [OK]

Common Mistakes:

• Blaming Execution Engine for metadata errors
• Confusing Driver with metadata storage
• Assuming Compiler handles metadata retrieval

5. You want to optimize Hive query execution by reducing the number of MapReduce jobs. Which Hive architecture component should you focus on tuning or configuring?

hard

A. Metastore

B. Execution Engine

C. Driver

D. Compiler

Solution

1. Step 1: Identify which component runs jobs

   The Execution Engine runs the MapReduce or Tez jobs generated by Hive.

2. Step 2: Understand optimization target

   Reducing jobs means tuning Execution Engine settings or choosing efficient engines like Tez or Spark.

3. Final Answer:

   Execution Engine -> Option B

4. Quick Check:

   Job optimization = Execution Engine tuning [OK]

Hint: Tune Execution Engine to reduce jobs and improve speed [OK]

Common Mistakes:

• Trying to optimize Metastore for job count
• Confusing Driver with job execution
• Ignoring Execution Engine's role in running jobs