File download automation in Bash Scripting - Mini Project: Build & Apply

File download automation

📖 Scenario: You work in an office where you often need to download files from the internet. Doing this manually every time wastes time. Automating file downloads with a simple script can save you effort and avoid mistakes.

🎯 Goal: You will create a bash script that downloads a list of files from given URLs automatically. The script will check if the files already exist before downloading to avoid duplicates.

📋 What You'll Learn

Create a bash array variable with exact URLs

Create a variable for the download folder path

Use a for loop to iterate over the URLs

Check if the file already exists before downloading

Use wget command to download files

Print the download status for each file

💡 Why This Matters

🌍 Real World

Automating file downloads saves time and reduces errors when you need to get many files regularly from the internet.

💼 Career

Many IT and DevOps roles require scripting to automate repetitive tasks like downloading files, making this skill valuable for efficiency and reliability.

1

Set up the list of file URLs

Create a bash array called file_urls with these exact URLs: https://example.com/file1.txt, https://example.com/file2.txt, and https://example.com/file3.txt.

Bash Scripting

# Create the bash array file_urls with the 3 URLs
# Your code here

Hint

Use parentheses () to create a bash array and double quotes "" around each URL.

2

Create the download folder variable

Create a variable called download_folder and set it to downloads (a folder name).

Bash Scripting

file_urls=("https://example.com/file1.txt" "https://example.com/file2.txt" "https://example.com/file3.txt")
# Create the variable download_folder and set it to downloads
# Your code here

Hint

Use = without spaces to assign the folder name as a string.

3

Write the loop to download files if missing

Write a for loop using url as the iterator to go through file_urls. Inside the loop, extract the filename from the URL using basename. Then check if the file $download_folder/$filename exists. If it does not exist, use wget to download the file into $download_folder. If it exists, print "File $filename already exists."

Bash Scripting

file_urls=("https://example.com/file1.txt" "https://example.com/file2.txt" "https://example.com/file3.txt")
download_folder="downloads"

# Write a for loop to iterate over file_urls
# Extract filename with basename
# Check if file exists in download_folder
# Download if missing, else print message
# Your code here

Hint

Use mkdir -p to create the folder if it does not exist. Use basename to get the filename from the URL.

4

Print a final message after downloads

Add a echo command after the loop to print exactly: All downloads complete.

Bash Scripting

file_urls=("https://example.com/file1.txt" "https://example.com/file2.txt" "https://example.com/file3.txt")
download_folder="downloads"

mkdir -p "$download_folder"
for url in "${file_urls[@]}"; do
    filename=$(basename "$url")
    if [ ! -f "$download_folder/$filename" ]; then
        wget -P "$download_folder" "$url"
    else
        echo "File $filename already exists."
    fi
done

# Print the final message
# Your code here

Hint

Use echo to print the message exactly as shown.

Practice

1. What is the main purpose of automating file downloads in bash scripting?

easy

A. To slow down the internet speed

B. To make files larger

C. To save time and reduce manual errors

D. To delete files automatically

Solution

1. Step 1: Understand automation benefits

   Automating repetitive tasks like file downloads saves time and reduces mistakes.

2. Step 2: Identify the correct purpose

   The main goal is efficiency and accuracy, not file size or deletion.

3. Final Answer:

   To save time and reduce manual errors -> Option C

4. Quick Check:

   Automation = Save time & reduce errors [OK]

Hint: Automation means doing tasks faster and safer [OK]

Common Mistakes:

• Thinking automation changes file size
• Confusing automation with file deletion
• Assuming automation slows internet

2. Which of the following is the correct syntax to download a file using wget in bash?

easy

A. curl -download http://example.com/file.txt

B. download http://example.com/file.txt

C. fetch http://example.com/file.txt

D. wget http://example.com/file.txt

Solution

1. Step 1: Recall wget command syntax

   The basic wget command to download a file is 'wget URL'.

2. Step 2: Compare options

   Only wget http://example.com/file.txt uses the correct wget syntax; others use invalid commands.

3. Final Answer:

   wget http://example.com/file.txt -> Option D

4. Quick Check:

   wget URL = correct syntax [OK]

Hint: Use 'wget URL' to download files simply [OK]

Common Mistakes:

• Using non-existent commands like 'download' or 'fetch'
• Mixing curl options with wget
• Forgetting to specify the URL

3. What will be the output of this bash command?

curl -O https://example.com/sample.txt
ls | grep sample.txt

medium

A. sample.txt

B. No output

C. curl: command not found

D. Error: file not found

Solution

1. Step 1: Understand curl -O behavior

   The command downloads 'sample.txt' to the current directory.

2. Step 2: Check the ls and grep output

   Listing files and filtering for 'sample.txt' will show the filename if download succeeded.

3. Final Answer:

   sample.txt -> Option A

4. Quick Check:

   curl -O downloads file, ls | grep shows filename [OK]

Hint: curl -O saves file; grep finds it in list [OK]

Common Mistakes:

• Assuming curl prints file content
• Expecting no output after download
• Confusing curl errors with normal output

4. Identify the error in this script snippet for downloading a file:

curl -o https://example.com/data.csv

medium

A. URL is incorrect

B. Missing filename after -o option

C. curl cannot download CSV files

D. No error, script is correct

Solution

1. Step 1: Check curl -o usage

   The '-o' option requires a filename to save the download.

2. Step 2: Analyze the command

   The command provides URL but no filename after '-o', causing error.

3. Final Answer:

   Missing filename after -o option -> Option B

4. Quick Check:

   curl -o needs filename [OK]

Hint: Always give filename after curl -o [OK]

Common Mistakes:

• Forgetting filename after -o
• Thinking URL is wrong format
• Believing curl can't download certain file types

5. You want to automate downloading multiple files listed in a text file named urls.txt. Which bash script snippet correctly downloads each file using wget?

hard

A. while read url; do wget "$url"; done < urls.txt

B. for url in urls.txt; do wget $url; done

C. wget -i urls.txt -O output.log

D. cat urls.txt | wget

Solution

1. Step 1: Understand reading lines from file

   The 'while read url; do ... done < urls.txt' reads each line as a URL.

2. Step 2: Check wget usage inside loop

   Using wget "$url" downloads each URL correctly.

3. Step 3: Analyze other options

   'for url in urls.txt; do wget $url; done' treats filename as URL, 'wget -i urls.txt -O output.log' misuses -O option causing all files to overwrite as 'output.log', 'cat urls.txt | wget' pipes incorrectly.

4. Final Answer:

   while read url; do wget "$url"; done < urls.txt -> Option A

5. Quick Check:

   Loop reading lines + wget = correct [OK]

Hint: Use while read loop to process URLs line by line [OK]

Common Mistakes:

• Using for loop directly on filename
• Misusing wget options
• Piping file content directly to wget