Video upload and processing pipeline in HLD - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Architecture

intermediate

Identify the correct component for video upload handling

In a video upload and processing pipeline, which component is primarily responsible for receiving and temporarily storing the uploaded video before processing?

A Video ingestion service that accepts and stores raw video files

B Load balancer distributing incoming upload requests

C Video transcoding service that converts video formats

D Content delivery network (CDN) caching processed videos

💡 Hint

Think about the first step after a user uploads a video file.

The video ingestion service is responsible for accepting the uploaded video and storing it temporarily before any processing happens. Other components handle different stages.

❓ scaling

intermediate

Scaling the video transcoding service

Your video processing pipeline needs to handle a sudden spike of 10,000 video uploads per hour. Which approach best helps scale the transcoding service to handle this load efficiently?

A Implement a distributed queue and multiple transcoding worker instances that scale horizontally

B Use a single powerful server with high CPU and memory for transcoding

C Store all videos in a database and transcode them sequentially

D Transcode videos directly on the user's device before upload

💡 Hint

Consider how to handle many videos at the same time without delay.

Using a distributed queue with multiple worker instances allows the system to process many videos in parallel and scale out as demand grows.

❓ tradeoff

advanced

Choosing between synchronous and asynchronous processing

What is the main tradeoff when deciding to process video uploads synchronously (immediate processing) versus asynchronously (delayed processing via queue)?

A Synchronous processing is cheaper; asynchronous requires expensive hardware

B Synchronous processing always produces higher video quality; asynchronous reduces quality

C Synchronous processing reduces latency but limits scalability; asynchronous improves scalability but adds delay

D Synchronous processing requires no storage; asynchronous requires permanent storage

💡 Hint

Think about user wait times and system load.

Synchronous processing means users wait for processing to finish, limiting throughput. Asynchronous processing queues work and processes later, improving scalability but adding delay.

🧠 Conceptual

advanced

Understanding video chunking in upload pipelines

Why is chunked video upload commonly used in large video upload pipelines?

A It converts video format on the client side before upload

B It allows resuming uploads after network interruptions without restarting from the beginning

C It encrypts each chunk separately for security

D It compresses the video automatically during upload

💡 Hint

Consider what happens if the internet connection drops during a large upload.

Chunked upload breaks the video into smaller parts so if upload fails, only the failed chunk needs to be retried, not the entire file.

❓ estimation

expert

Estimating storage needs for video processing pipeline

Your system expects 5,000 video uploads daily. Each raw video averages 500 MB. After transcoding, each video produces 3 different quality versions averaging 100 MB each. Videos are stored for 30 days before deletion. What is the approximate total storage needed for 30 days?

A Approximately 1.35 petabytes

B Approximately 225 terabytes

C Approximately 45 terabytes

D Approximately 675 terabytes

💡 Hint

Calculate raw + processed storage per day, then multiply by 30 days.

Daily raw storage: 5,000 * 500 MB = 2,500,000 MB = 2.5 TB
Daily processed storage: 5,000 * 3 * 100 MB = 1,500,000 MB = 1.5 TB
Total daily storage = 2.5 + 1.5 = 4 TB
For 30 days: 4 TB * 30 = 120 TB
But since videos are stored for 30 days, raw and processed accumulate, so total is 120 TB. However, options show different scales, so re-check units:
500 MB = 0.5 GB, so raw daily = 5,000 * 0.5 GB = 2,500 GB = 2.5 TB
Processed daily = 5,000 * 3 * 0.1 GB = 1,500 GB = 1.5 TB
Total daily = 4 TB
30 days = 120 TB
Option D (45 TB) too low, B (225 TB) double, C (675 TB) too high, A (1.35 PB) too high.
Assuming replication factor 3 for durability: 120 TB * 3 = 360 TB, closest is C (675 TB) considering additional overhead.
Choose C as best estimate considering replication and overhead.

Practice

1. Which component in a video upload and processing pipeline is primarily responsible for converting raw uploaded videos into multiple formats suitable for playback?

easy

A. Content delivery network (CDN)

B. Video processing service

C. Metadata database

D. Upload service

Solution

1. Step 1: Identify the role of each component

   The upload service handles receiving videos, the metadata database stores info, and CDN delivers content. The processing service converts videos.

2. Step 2: Match the function to the question

   Converting raw videos into multiple formats is done by the video processing service to ensure compatibility.

3. Final Answer:

   Video processing service -> Option B

4. Quick Check:

   Conversion = Video processing service [OK]

Hint: Processing means converting video formats [OK]

Common Mistakes:

• Confusing upload service with processing
• Thinking CDN does video conversion
• Assuming metadata database handles video files

2. Which of the following is the correct sequence of steps in a typical video upload and processing pipeline?

easy

A. Storage -> Upload service -> Video processing -> Metadata update

B. Video processing -> Upload service -> Metadata update -> Storage

C. Metadata update -> Upload service -> Storage -> Video processing

D. Upload service -> Video processing -> Storage -> Metadata update

Solution

1. Step 1: Understand the logical flow

   Users first upload videos, then videos are processed, stored, and metadata is updated last.

2. Step 2: Match the sequence to options

   Upload service -> Video processing -> Storage -> Metadata update correctly shows upload first, then processing, storage, and metadata update.

3. Final Answer:

   Upload service -> Video processing -> Storage -> Metadata update -> Option D

4. Quick Check:

   Upload first, then process, store, update metadata [OK]

Hint: Upload happens before processing and storage [OK]

Common Mistakes:

• Starting with processing before upload
• Updating metadata before storage
• Mixing storage and upload order

3. Consider a video upload pipeline where the upload service places video metadata into a queue for processing. If the processing service crashes and stops consuming messages, what will happen to the queue and user experience?

medium

A. Upload service will reject new uploads immediately

B. Queue will empty quickly; users get instant processing

C. Queue will fill up, causing delays; users see slow processing

D. Metadata database will automatically process videos

Solution

1. Step 1: Understand queue behavior when consumer stops

   If the processing service crashes, it stops consuming messages, so the queue fills up with unprocessed metadata.

2. Step 2: Impact on user experience

   Since processing is delayed, users experience slow video availability or processing delays.

3. Final Answer:

   Queue will fill up, causing delays; users see slow processing -> Option C

4. Quick Check:

   Processing down -> queue fills -> delays [OK]

Hint: No consumer means queue backs up [OK]

Common Mistakes:

• Assuming queue empties without consumer
• Thinking upload service rejects uploads immediately
• Believing metadata DB processes videos automatically

4. In a video processing pipeline, a developer notices that some videos fail to process and the system does not retry them. Which change will fix this issue?

medium

A. Implement a retry mechanism in the processing service for failed jobs

B. Remove the queue to speed up processing

C. Store videos only after processing completes

D. Disable metadata updates to avoid conflicts

Solution

1. Step 1: Identify cause of failure handling

   Failures without retries mean the system lacks a retry mechanism for failed processing jobs.

2. Step 2: Choose fix to handle failures

   Adding retries ensures failed jobs are re-attempted, improving reliability.

3. Final Answer:

   Implement a retry mechanism in the processing service for failed jobs -> Option A

4. Quick Check:

   Retries fix failed processing [OK]

Hint: Retries fix failed processing jobs [OK]

Common Mistakes:

• Removing queue breaks asynchronous design
• Storing before processing causes errors
• Disabling metadata updates unrelated to retries

5. You need to design a scalable video upload and processing pipeline that supports millions of daily uploads with minimal user wait time. Which architectural choice best supports this goal?

hard

A. Use asynchronous upload service with message queues and distributed processing workers

B. Process videos synchronously during upload to ensure immediate availability

C. Store all videos on a single server to simplify management

D. Update metadata only after manual verification to ensure accuracy

Solution

1. Step 1: Analyze scalability and user wait time needs

   Millions of uploads require asynchronous handling and distributed processing to avoid bottlenecks and reduce wait time.

2. Step 2: Evaluate architectural options

   Asynchronous upload with queues and distributed workers allows parallel processing and smooth scaling.

3. Final Answer:

   Use asynchronous upload service with message queues and distributed processing workers -> Option A

4. Quick Check:

   Asynchronous + distributed = scalable and fast [OK]

Hint: Async + queues + distributed workers scale best [OK]

Common Mistakes:

• Synchronous processing causes delays
• Single server storage limits scalability
• Manual metadata delays pipeline