External vs managed tables in Hadoop - Visual Side-by-Side Comparison

Concept Flow - External vs managed tables

Create Table Command

↓

Specify Table Type?

↓

Managed

↓

Data stored

↓

in warehouse

↓

Drop table

↓

Data deleted

Shows the decision flow when creating a table: managed tables store data inside warehouse and delete data on drop; external tables point to external data and keep data after drop.

Execution Sample

Hadoop

CREATE TABLE sales_managed (id INT, amount FLOAT);
CREATE EXTERNAL TABLE sales_external (id INT, amount FLOAT)
LOCATION '/data/sales';
DROP TABLE sales_managed;
DROP TABLE sales_external;

Creates a managed and an external table, then drops both to show difference in data deletion.

Execution Table

Step	Command	Table Type	Data Location	Action on Drop	Result
1	CREATE TABLE sales_managed (id INT, amount FLOAT);	Managed	Warehouse default	Delete data	Table created, data stored in warehouse
2	CREATE EXTERNAL TABLE sales_external (id INT, amount FLOAT) LOCATION '/data/sales';	External	/data/sales	Keep data	Table created, data location external
3	DROP TABLE sales_managed;	Managed	Warehouse default	Delete data	Table dropped, data deleted from warehouse
4	DROP TABLE sales_external;	External	/data/sales	Keep data	Table dropped, data remains at external location

💡 All commands executed; difference in data deletion on drop shown

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4
sales_managed_table	None	Exists with data in warehouse	Exists with data in warehouse	Dropped, data deleted	Dropped
sales_external_table	None	None	Exists pointing to /data/sales	Exists pointing to /data/sales	Dropped, data remains at /data/sales

Key Moments - 2 Insights

Why does dropping a managed table delete the data but dropping an external table does not?

Because managed tables store data inside the warehouse and control it fully, so dropping deletes data (see Step 3). External tables only reference data outside, so dropping removes the table metadata but leaves data intact (see Step 4).

What happens if you create an external table without specifying LOCATION?

The system will throw an error or treat it as managed because external tables require a data location outside the warehouse (see Step 2 for correct syntax).

Visual Quiz - 3 Questions

Look at the execution table, what happens to the data of sales_managed after DROP TABLE?

A Data remains in warehouse

B Data moves to external location

C Data is deleted from warehouse

D Data is archived automatically

💡 Hint

Check Step 3 in execution_table under 'Result' and 'Action on Drop'

At which step does the external table get created with a specified data location?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Look at Step 2 command and 'Data Location' column

If you drop the external table, what happens to the data at '/data/sales/'?

A Data remains untouched

B Data is moved to warehouse

C Data is deleted

D Data is backed up automatically

💡 Hint

See Step 4 'Result' and 'Action on Drop' columns

Concept Snapshot

External vs Managed Tables in Hadoop:
- Managed tables store data inside warehouse.
- External tables point to data outside warehouse.
- Dropping managed table deletes data.
- Dropping external table keeps data intact.
- External tables require LOCATION clause.

Full Transcript

This visual execution shows how managed and external tables behave differently in Hadoop. When creating a managed table, data is stored inside the warehouse. Dropping this table deletes both metadata and data. For external tables, data lives outside the warehouse at a specified location. Dropping the external table removes only metadata, leaving data untouched. This difference is important for managing data lifecycle and storage control.

Practice

1. What is the main difference between a managed table and an external table in Hadoop?

easy

A. Managed tables do not store any data, external tables store all data inside Hadoop.

B. External tables store data inside Hadoop, managed tables store data outside Hadoop.

C. Managed tables store and manage data files within Hadoop, while external tables only manage metadata and keep data files outside Hadoop.

D. There is no difference; both manage data files the same way.

Solution

1. Step 1: Understand managed table behavior

   Managed tables store data files inside Hadoop's warehouse directory and Hadoop controls their lifecycle.

2. Step 2: Understand external table behavior

   External tables only manage metadata in Hadoop but keep data files at an external location you specify.

3. Final Answer:

   Managed tables store and manage data files within Hadoop, while external tables only manage metadata and keep data files outside Hadoop. -> Option C

4. Quick Check:

   Managed = data + metadata, External = metadata only [OK]

Hint: Managed = Hadoop controls data; External = you control data location [OK]

Common Mistakes:

• Confusing where data files are stored
• Thinking external tables delete data on drop
• Assuming managed tables don't store data

2. Which of the following is the correct HiveQL syntax to create an external table?

easy

A. CREATE TABLE my_table (id INT, name STRING) LOCATION '/user/data' EXTERNAL;

B. CREATE TABLE my_table EXTERNAL (id INT, name STRING) LOCATION '/user/data';

C. CREATE TABLE my_table (id INT, name STRING) EXTERNAL LOCATION '/user/data';

D. CREATE EXTERNAL TABLE my_table (id INT, name STRING) LOCATION '/user/data';

Solution

1. Step 1: Recall correct syntax for external table creation

   The keyword EXTERNAL comes immediately after CREATE and before TABLE.

2. Step 2: Check location clause placement

   The LOCATION clause specifies the external data path and comes after the column definitions.

3. Final Answer:

   CREATE EXTERNAL TABLE my_table (id INT, name STRING) LOCATION '/user/data'; -> Option D

4. Quick Check:

   EXTERNAL after CREATE, LOCATION after columns [OK]

Hint: EXTERNAL goes right after CREATE in HiveQL [OK]

Common Mistakes:

• Placing EXTERNAL after TABLE keyword
• Misplacing LOCATION clause
• Omitting EXTERNAL keyword for external tables

3. Consider the following Hive commands:

CREATE EXTERNAL TABLE ext_table (id INT) LOCATION '/data/ext';
CREATE TABLE man_table (id INT);

What happens if you drop ext_table and man_table respectively?

medium

A. Dropping ext_table deletes data files; dropping man_table deletes only metadata.

B. Dropping ext_table deletes only metadata; dropping man_table deletes both metadata and data files.

C. Both drops delete only metadata, data files remain.

D. Both drops delete metadata and data files.

Solution

1. Step 1: Understand drop behavior for external tables

   Dropping an external table removes only metadata; data files at '/data/ext' remain untouched.

2. Step 2: Understand drop behavior for managed tables

   Dropping a managed table removes metadata and deletes data files stored in Hadoop's warehouse.

3. Final Answer:

   Dropping ext_table deletes only metadata; dropping man_table deletes both metadata and data files. -> Option B

4. Quick Check:

   External drop = metadata only, Managed drop = metadata + data [OK]

Hint: External drop keeps data; managed drop deletes data [OK]

Common Mistakes:

• Assuming external drop deletes data files
• Thinking managed drop keeps data files
• Confusing metadata and data deletion

4. You created an external table but accidentally omitted the LOCATION clause. What will happen when you try to query this table?

medium

A. Query will fail because LOCATION is mandatory for external tables.

B. Query will succeed using default warehouse location, behaving like a managed table.

C. Query will succeed but return no data.

D. Hive will automatically assign a random external location.

Solution

1. Step 1: Check requirement of LOCATION for external tables

   External tables require LOCATION to specify where data files reside outside Hadoop's warehouse.

2. Step 2: Understand consequence of missing LOCATION

   Without LOCATION, Hive cannot find external data, so query fails with an error.

3. Final Answer:

   Query will fail because LOCATION is mandatory for external tables. -> Option A

4. Quick Check:

   External tables need LOCATION [OK]

Hint: External tables must have LOCATION clause [OK]

Common Mistakes:

• Assuming default location is used for external tables
• Thinking query returns empty data silently
• Believing Hive auto-assigns external location

5. You have important data files stored in HDFS at /user/hadoop/data. You want to create a table in Hive to query this data but ensure that deleting the table does NOT delete the data files. Which table type should you create and what is the correct approach?

hard

A. Create an external table with LOCATION '/user/hadoop/data' so Hive manages metadata only and leaves data intact on drop.

B. Create a managed table and copy data files into Hive warehouse; data will be deleted on drop.

C. Create a managed table with LOCATION '/user/hadoop/data' to keep data files safe on drop.

D. Create an external table without specifying LOCATION; Hive will manage data safely.

Solution

1. Step 1: Identify requirement to keep data files safe on table drop

   Only external tables keep data files untouched when the table is dropped.

2. Step 2: Use external table with explicit LOCATION

   Specify LOCATION '/user/hadoop/data' so Hive knows where data files are stored externally.

3. Step 3: Understand managed table behavior

   Managed tables store data in warehouse and delete data files on drop, so not suitable here.

4. Final Answer:

   Create an external table with LOCATION '/user/hadoop/data' so Hive manages metadata only and leaves data intact on drop. -> Option A

5. Quick Check:

   External + LOCATION = safe data on drop [OK]

Hint: Use external table with LOCATION to protect data files [OK]

Common Mistakes:

• Using managed table and losing data on drop
• Omitting LOCATION for external table
• Assuming managed table can protect external data