Migration from Hadoop to cloud-native - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Key advantage of cloud-native over Hadoop

Which of the following is the main advantage of migrating from Hadoop to a cloud-native data platform?

A Cloud-native platforms require manual resource allocation for each job to optimize costs.

B Cloud-native platforms provide automatic scaling and managed services reducing operational overhead.

C Hadoop clusters are easier to set up and require less maintenance than cloud-native platforms.

D Hadoop supports serverless computing natively, unlike cloud-native platforms.

💡 Hint

Think about how cloud platforms handle resources compared to traditional Hadoop clusters.

Cloud-native platforms offer automatic scaling and managed services, which reduce the need for manual cluster management, unlike Hadoop which requires manual setup and maintenance.

❓ data_output

intermediate

Output of data migration script

Given the following Python pseudocode for migrating data from HDFS to cloud storage, what will be the output summary?

Hadoop

files = ['file1.csv', 'file2.csv', 'file3.csv']
migrated = []
for f in files:
    if f.endswith('.csv'):
        migrated.append(f)
print(f'Migrated {len(migrated)} files: {migrated}')

A Migrated 2 files: ['file1.csv', 'file2.csv']

B Migrated 0 files: []

C Migrated 3 files: ['file1.csv', 'file2.csv', 'file3.csv']

D Migrated 3 files: ['file1.csv', 'file2.csv']

💡 Hint

Check which files end with '.csv' in the list.

All three files end with '.csv', so all are appended to the migrated list, resulting in 3 files migrated.

🔧 Debug

advanced

Identify error in cloud migration script

What error will the following Python code raise during migration?

Hadoop

def migrate_data(files):
    migrated = {}
    for f in files:
        migrated[f] = migrated.get(f, 0) + 1
    return migrated

files = ['data1', 'data2', 'data1']
result = migrate_data(files)
print(result['data3'])

A KeyError because 'data3' is not in the dictionary.

B SyntaxError due to missing colon in function definition.

C No error, output will be {'data1': 2, 'data2': 1, 'data3': 0}.

D TypeError because dictionary keys must be integers.

💡 Hint

Check what happens when accessing a dictionary key that does not exist.

Accessing result['data3'] raises a KeyError because 'data3' was never added to the dictionary.

❓ visualization

advanced

Interpreting migration progress chart

A line chart shows the number of files migrated over time during a Hadoop to cloud migration. The line starts slow, then sharply rises, then plateaus. What does this pattern most likely indicate?

A Files were deleted instead of migrated, causing the plateau.

B Constant migration speed with no changes in resource allocation.

C Migration failed early and restarted multiple times.

D Initial setup delays, followed by fast migration, then resource limits causing slowdown.

💡 Hint

Think about common phases in large data migrations and resource usage.

The slow start is typical setup time, the sharp rise shows active migration, and the plateau suggests hitting resource or quota limits.

🚀 Application

expert

Choosing the best cloud-native tool for batch processing

You need to migrate a Hadoop batch processing job to a cloud-native environment. The job processes large datasets with complex transformations and requires fault tolerance. Which cloud-native tool is best suited?

A Apache Spark on managed Kubernetes with autoscaling and checkpointing.

B Simple cloud storage buckets with manual script execution.

C Serverless functions with 1-minute execution limits.

D Relational database triggers for batch processing.

💡 Hint

Consider fault tolerance, scalability, and batch processing capabilities.

Apache Spark on managed Kubernetes supports complex batch jobs, fault tolerance via checkpointing, and autoscaling, making it ideal for migrating Hadoop batch jobs.

Practice

1. What is the main benefit of migrating Hadoop workloads to cloud-native platforms?

easy

A. Better scalability and easier management

B. Reduced data storage capacity

C. Elimination of all data processing

D. Increased hardware maintenance

Solution

1. Step 1: Understand Hadoop migration goals

   Migration aims to improve how Hadoop workloads scale and are managed.

2. Step 2: Identify benefits of cloud-native platforms

   Cloud platforms offer flexible scaling and easier cluster management compared to on-premises Hadoop.

3. Final Answer:

   Better scalability and easier management -> Option A

4. Quick Check:

   Cloud migration = scalability + management [OK]

Hint: Think why cloud is popular for big data [OK]

Common Mistakes:

• Confusing storage reduction with scalability
• Assuming migration removes data processing
• Thinking hardware maintenance increases

2. Which of the following is the correct syntax to start a Hadoop job on AWS EMR using the AWS CLI?

easy

A. aws emr run-job --id j-2AXXXXXXGAPLF --type Spark --file myapp.jar

B. aws emr start-job --cluster j-2AXXXXXXGAPLF --job Spark --args s3://mybucket/myapp.jar

C. aws emr add-steps --cluster-id j-2AXXXXXXGAPLF --steps Type=Spark,Name="Spark Program",ActionOnFailure=CONTINUE,Args=[--class,org.apache.spark.examples.SparkPi,s3://mybucket/myapp.jar,10]

D. aws emr submit-step --cluster j-2AXXXXXXGAPLF --step Spark --jar myapp.jar

Solution

1. Step 1: Review AWS EMR CLI syntax for adding steps

   The correct command uses aws emr add-steps with detailed step parameters.

2. Step 2: Match syntax with options

   aws emr add-steps --cluster-id j-2AXXXXXXGAPLF --steps Type=Spark,Name="Spark Program",ActionOnFailure=CONTINUE,Args=[--class,org.apache.spark.examples.SparkPi,s3://mybucket/myapp.jar,10] matches the official syntax with cluster ID, step type, name, failure action, and arguments.

3. Final Answer:

   aws emr add-steps --cluster-id j-2AXXXXXXGAPLF --steps Type=Spark,Name="Spark Program",ActionOnFailure=CONTINUE,Args=[--class,org.apache.spark.examples.SparkPi,s3://mybucket/myapp.jar,10] -> Option C

4. Quick Check:

   Use add-steps with detailed args [OK]

Hint: Remember AWS EMR uses add-steps command [OK]

Common Mistakes:

• Using non-existent commands like start-job
• Omitting required step parameters
• Incorrect option names or flags

3. Given the following Python snippet using Google Cloud Dataproc API to submit a Hadoop job, what will be the output if the job completes successfully?

from google.cloud import dataproc_v1
client = dataproc_v1.JobControllerClient()
job = {
  "placement": {"cluster_name": "my-cluster"},
  "hadoop_job": {"main_jar_file_uri": "gs://my-bucket/my-job.jar"}
}
response = client.submit_job(project_id="my-project", region="us-central1", job=job)
print(response.status.state)

medium

A. JobStatus.State.DONE

B. JobStatus.State.ERROR

C. JobStatus.State.RUNNING

D. JobStatus.State.PENDING

Solution

1. Step 1: Understand the job submission and response

   The submit_job method returns a job response with a status field indicating job state.

2. Step 2: Interpret the status.state value after submission

   The submit_job returns immediately with initial state JobStatus.State.PENDING. The print occurs right after submission, before completion, even if the job succeeds later.

3. Final Answer:

   JobStatus.State.PENDING -> Option D

4. Quick Check:

   submit_job initial state = PENDING [OK]

Hint: submit_job returns PENDING immediately [OK]

Common Mistakes:

• Thinking the code waits for DONE on completion
• Confusing initial PENDING with final state
• Assuming RUNNING or ERROR on submission

4. You tried migrating a Hadoop job to AWS EMR but got an error: AccessDeniedException. What is the most likely cause?

medium

A. The EMR cluster is not running

B. The IAM role lacks permissions to access S3 bucket

C. The Hadoop job jar file is corrupted

D. The AWS CLI version is outdated

Solution

1. Step 1: Analyze the error type AccessDeniedException

   This error usually means permission issues, especially with AWS IAM roles.

2. Step 2: Identify common permission problems in EMR migration

   AccessDeniedException often occurs if the EMR cluster's IAM role cannot read the S3 bucket containing job files.

3. Final Answer:

   The IAM role lacks permissions to access S3 bucket -> Option B

4. Quick Check:

   AccessDenied = permission issue [OK]

Hint: AccessDenied usually means permission problem [OK]

Common Mistakes:

• Blaming corrupted jar without checking permissions
• Assuming cluster state causes access errors
• Ignoring IAM role permissions

5. You want to migrate a large Hadoop dataset to a cloud-native platform but want to minimize downtime. Which approach is best?

hard

A. Use incremental data replication with tools like DistCp to sync data continuously

B. Copy all data once during a maintenance window and stop all jobs

C. Manually upload data files via FTP to cloud storage

D. Run Hadoop jobs locally without migration

Solution

1. Step 1: Understand downtime minimization in migration

   Minimizing downtime requires continuous syncing rather than one-time full copy.

2. Step 2: Identify best tool for incremental replication

   DistCp supports incremental copying of Hadoop data to cloud storage, enabling near-zero downtime migration.

3. Final Answer:

   Use incremental data replication with tools like DistCp to sync data continuously -> Option A

4. Quick Check:

   Incremental sync = less downtime [OK]

Hint: Incremental sync tools reduce downtime [OK]

Common Mistakes:

• Doing one-time full copy causing long downtime
• Using manual FTP uploads for big data
• Avoiding migration by running locally