External vs managed tables in Hadoop

Introduction

We use tables to organize data in Hadoop. Managed and external tables help decide who controls the data files and what happens when we delete the table.

When you want Hadoop to fully manage your data files and clean them up automatically.

When you want to keep data files even after deleting the table in Hadoop.

When sharing data files across different Hadoop tools or clusters.

When you want to control where data files are stored outside Hadoop's default location.

Syntax

Hadoop

CREATE TABLE table_name (column1 TYPE, column2 TYPE, ...)
STORED AS file_format;
-- For external tables, add: LOCATION 'path_to_data';

Managed tables do not use the LOCATION clause; Hadoop stores data in its default folder.

External tables use LOCATION to point to data outside Hadoop's control.

Examples

This creates a managed table. Hadoop stores and manages the data files.

Hadoop

CREATE TABLE managed_table (
  id INT,
  name STRING
)
STORED AS PARQUET;

This creates an external table. Data files stay where you put them, outside Hadoop's default folder.

Hadoop

CREATE EXTERNAL TABLE external_table (
  id INT,
  name STRING
)
STORED AS PARQUET
LOCATION '/user/data/external_table/';

Sample Program

This example creates one managed and one external table, inserts data, and queries both. Managed table data is stored inside Hadoop's control. External table data is stored at the specified location.

Hadoop

CREATE TABLE managed_employees (
  emp_id INT,
  emp_name STRING
)
STORED AS TEXTFILE;

CREATE EXTERNAL TABLE external_employees (
  emp_id INT,
  emp_name STRING
)
STORED AS TEXTFILE
LOCATION '/user/hadoop/external_employees/';

-- After creating, insert some data
INSERT INTO managed_employees VALUES (1, 'Alice');
INSERT INTO external_employees VALUES (2, 'Bob');

-- Query both tables
SELECT * FROM managed_employees;
SELECT * FROM external_employees;

Important Notes

Deleting a managed table deletes its data files automatically.

Deleting an external table only deletes the table metadata, not the data files.

Use external tables to share data across different systems without moving files.

Summary

Managed tables let Hadoop control data storage and cleanup.

External tables keep data files where you choose and only manage metadata.

Choose based on whether you want Hadoop to manage your data files or not.

Practice

1. What is the main difference between a managed table and an external table in Hadoop?

easy

A. Managed tables do not store any data, external tables store all data inside Hadoop.

B. External tables store data inside Hadoop, managed tables store data outside Hadoop.

C. Managed tables store and manage data files within Hadoop, while external tables only manage metadata and keep data files outside Hadoop.

D. There is no difference; both manage data files the same way.

Solution

1. Step 1: Understand managed table behavior

   Managed tables store data files inside Hadoop's warehouse directory and Hadoop controls their lifecycle.

2. Step 2: Understand external table behavior

   External tables only manage metadata in Hadoop but keep data files at an external location you specify.

3. Final Answer:

   Managed tables store and manage data files within Hadoop, while external tables only manage metadata and keep data files outside Hadoop. -> Option C

4. Quick Check:

   Managed = data + metadata, External = metadata only [OK]

Hint: Managed = Hadoop controls data; External = you control data location [OK]

Common Mistakes:

• Confusing where data files are stored
• Thinking external tables delete data on drop
• Assuming managed tables don't store data

2. Which of the following is the correct HiveQL syntax to create an external table?

easy

A. CREATE TABLE my_table (id INT, name STRING) LOCATION '/user/data' EXTERNAL;

B. CREATE TABLE my_table EXTERNAL (id INT, name STRING) LOCATION '/user/data';

C. CREATE TABLE my_table (id INT, name STRING) EXTERNAL LOCATION '/user/data';

D. CREATE EXTERNAL TABLE my_table (id INT, name STRING) LOCATION '/user/data';

Solution

1. Step 1: Recall correct syntax for external table creation

   The keyword EXTERNAL comes immediately after CREATE and before TABLE.

2. Step 2: Check location clause placement

   The LOCATION clause specifies the external data path and comes after the column definitions.

3. Final Answer:

   CREATE EXTERNAL TABLE my_table (id INT, name STRING) LOCATION '/user/data'; -> Option D

4. Quick Check:

   EXTERNAL after CREATE, LOCATION after columns [OK]

Hint: EXTERNAL goes right after CREATE in HiveQL [OK]

Common Mistakes:

• Placing EXTERNAL after TABLE keyword
• Misplacing LOCATION clause
• Omitting EXTERNAL keyword for external tables

3. Consider the following Hive commands:

CREATE EXTERNAL TABLE ext_table (id INT) LOCATION '/data/ext';
CREATE TABLE man_table (id INT);

What happens if you drop ext_table and man_table respectively?

medium

A. Dropping ext_table deletes data files; dropping man_table deletes only metadata.

B. Dropping ext_table deletes only metadata; dropping man_table deletes both metadata and data files.

C. Both drops delete only metadata, data files remain.

D. Both drops delete metadata and data files.

Solution

1. Step 1: Understand drop behavior for external tables

   Dropping an external table removes only metadata; data files at '/data/ext' remain untouched.

2. Step 2: Understand drop behavior for managed tables

   Dropping a managed table removes metadata and deletes data files stored in Hadoop's warehouse.

3. Final Answer:

   Dropping ext_table deletes only metadata; dropping man_table deletes both metadata and data files. -> Option B

4. Quick Check:

   External drop = metadata only, Managed drop = metadata + data [OK]

Hint: External drop keeps data; managed drop deletes data [OK]

Common Mistakes:

• Assuming external drop deletes data files
• Thinking managed drop keeps data files
• Confusing metadata and data deletion

4. You created an external table but accidentally omitted the LOCATION clause. What will happen when you try to query this table?

medium

A. Query will fail because LOCATION is mandatory for external tables.

B. Query will succeed using default warehouse location, behaving like a managed table.

C. Query will succeed but return no data.

D. Hive will automatically assign a random external location.

Solution

1. Step 1: Check requirement of LOCATION for external tables

   External tables require LOCATION to specify where data files reside outside Hadoop's warehouse.

2. Step 2: Understand consequence of missing LOCATION

   Without LOCATION, Hive cannot find external data, so query fails with an error.

3. Final Answer:

   Query will fail because LOCATION is mandatory for external tables. -> Option A

4. Quick Check:

   External tables need LOCATION [OK]

Hint: External tables must have LOCATION clause [OK]

Common Mistakes:

• Assuming default location is used for external tables
• Thinking query returns empty data silently
• Believing Hive auto-assigns external location

5. You have important data files stored in HDFS at /user/hadoop/data. You want to create a table in Hive to query this data but ensure that deleting the table does NOT delete the data files. Which table type should you create and what is the correct approach?

hard

A. Create an external table with LOCATION '/user/hadoop/data' so Hive manages metadata only and leaves data intact on drop.

B. Create a managed table and copy data files into Hive warehouse; data will be deleted on drop.

C. Create a managed table with LOCATION '/user/hadoop/data' to keep data files safe on drop.

D. Create an external table without specifying LOCATION; Hive will manage data safely.

Solution

1. Step 1: Identify requirement to keep data files safe on table drop

   Only external tables keep data files untouched when the table is dropped.

2. Step 2: Use external table with explicit LOCATION

   Specify LOCATION '/user/hadoop/data' so Hive knows where data files are stored externally.

3. Step 3: Understand managed table behavior

   Managed tables store data in warehouse and delete data files on drop, so not suitable here.

4. Final Answer:

   Create an external table with LOCATION '/user/hadoop/data' so Hive manages metadata only and leaves data intact on drop. -> Option A

5. Quick Check:

   External + LOCATION = safe data on drop [OK]

Hint: Use external table with LOCATION to protect data files [OK]

Common Mistakes:

• Using managed table and losing data on drop
• Omitting LOCATION for external table
• Assuming managed table can protect external data