Loading datasets (CSV, built-in datasets) in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Loading datasets (CSV, built-in datasets)

Which metric matters for Loading Datasets and WHY

When loading datasets, the main goal is to ensure data quality and correctness before training a model. Metrics like data completeness (no missing rows), data consistency (correct format and types), and data integrity (no corrupted or duplicated entries) matter most. These ensure the model learns from accurate and reliable data. Without clean data, even the best model will perform poorly.

Confusion Matrix or Equivalent Visualization

Loading datasets itself does not produce a confusion matrix, but errors in loading can be visualized as:

    +-------------------------+
    | Dataset Loading Status  |
    +-------------------------+
    | Total rows expected: 1000 |
    | Rows loaded: 980          |
    | Missing rows: 20          |
    | Format errors: 5          |
    | Duplicates found: 10      |
    +-------------------------+
    

This helps identify issues before model training.

Tradeoff: Data Completeness vs Data Quality

Sometimes you can load all data but with errors, or load less data but clean. For example:

• Load all rows quickly: Might include corrupted or missing values, hurting model accuracy.
• Load only clean rows: Might lose some data but improve model training quality.

Choosing the right balance depends on your project needs.

What Good vs Bad Looks Like for Dataset Loading

• Good: 100% rows loaded, no missing or corrupted data, correct data types, no duplicates.
• Bad: Missing rows, wrong data types (e.g., text in numeric columns), corrupted entries, many duplicates.

Good data loading leads to reliable model training and trustworthy results.

Common Pitfalls When Loading Datasets

• Data leakage: Accidentally including test data in training data during loading.
• Incorrect parsing: Misreading CSV delimiters or encoding causing wrong data.
• Ignoring missing values: Not handling empty cells can cause errors later.
• Overfitting due to duplicates: Duplicate rows can bias the model.

Self Check

Your model has 98% accuracy but only 12% recall on fraud cases. Is it good for production?

Answer: No. This means the model misses most fraud cases (low recall). Even if accuracy is high, it fails to catch fraud, which is critical. You need to improve recall before using it in production.

Key Result

Ensuring complete and clean data loading is key to reliable model training and accurate evaluation.

Practice

1. What is the main purpose of loading datasets in machine learning?

easy

A. To get data from files or built-in sources for training models

B. To save the model after training

C. To visualize the model's predictions

D. To delete unused data from memory

Solution

1. Step 1: Understand dataset loading

   Loading datasets means bringing data into your program so you can work with it.

2. Step 2: Identify the purpose in machine learning

   Data is needed to train and test machine learning models, so loading datasets is the first step.

3. Final Answer:

   To get data from files or built-in sources for training models -> Option A

4. Quick Check:

   Loading datasets = Getting data [OK]

Hint: Loading datasets means getting data ready for models [OK]

Common Mistakes:

• Confusing loading data with saving models
• Thinking loading means visualizing results
• Mixing loading with deleting data

2. Which Python code correctly loads a CSV file named data.csv using pandas?

easy

A. import pandas as pd
df = pd.csv_read('data.csv')

B. import pandas as pd
df = pd.load_csv('data.csv')

C. import pandas as pd
df = pd.read('data.csv')

D. import pandas as pd
df = pd.read_csv('data.csv')

Solution

1. Step 1: Recall pandas CSV loading syntax

   The correct function to load CSV files in pandas is read_csv().

2. Step 2: Check each option's function name

   Only import pandas as pd df = pd.read_csv('data.csv') uses pd.read_csv('data.csv'), which is correct syntax.

3. Final Answer:

   import pandas as pd\ndf = pd.read_csv('data.csv') -> Option D

4. Quick Check:

   pandas.read_csv() loads CSV files [OK]

Hint: Use pandas.read_csv() to load CSV files [OK]

Common Mistakes:

• Using wrong function names like load_csv or csv_read
• Forgetting to import pandas
• Using read() instead of read_csv()

3. What will be the output type of the following code?

from sklearn.datasets import load_iris
iris = load_iris()
print(type(iris.data))

medium

A. <class 'numpy.ndarray'>

B. <class 'list'>

C. <class 'pandas.DataFrame'>

D. <class 'dict'>

Solution

1. Step 1: Understand sklearn's load_iris output

   The load_iris() function returns a Bunch object with data as a NumPy array.

2. Step 2: Check the type of iris.data

   Printing type(iris.data) shows it is a numpy.ndarray.

3. Final Answer:

   <class 'numpy.ndarray'> -> Option A

4. Quick Check:

   sklearn data attribute = numpy.ndarray [OK]

Hint: Sklearn datasets return numpy arrays for data [OK]

Common Mistakes:

• Thinking iris.data is a list or dict
• Confusing with pandas DataFrame
• Not knowing sklearn returns numpy arrays

4. Identify the error in this code snippet that tries to load a CSV file:

import pandas as pd
df = pd.read_csv('data.csv', delimiter=';')
print(df.head())

Assuming the CSV uses commas, what is wrong?

medium

A. Missing import statement for csv module

B. The delimiter should be ',' not ';'

C. The function read_csv does not exist

D. print(df.head()) is incorrect syntax

Solution

1. Step 1: Check delimiter parameter usage

   The code uses delimiter=';' but the CSV file uses commas, so this is wrong.

2. Step 2: Understand correct delimiter for comma CSV

   For comma-separated files, the delimiter should be ',' or omitted (default).

3. Final Answer:

   The delimiter should be ',' not ';' -> Option B

4. Quick Check:

   Delimiter must match CSV separator [OK]

Hint: Match delimiter to CSV separator character [OK]

Common Mistakes:

• Using wrong delimiter for CSV files
• Thinking read_csv is missing
• Confusing print syntax errors

5. You want to load the built-in digits dataset from sklearn and convert it to a pandas DataFrame with feature columns named pixel_0 to pixel_63. Which code correctly does this?

hard

A. import pandas as pd
df = pd.read_csv('digits.csv')
df.columns = [f'pixel_{i}' for i in range(64)]

B. from sklearn.datasets import load_digits
import pandas as pd
digits = load_digits()
df = pd.DataFrame(digits, columns=[f'pixel_{i}' for i in range(64)])

C. from sklearn.datasets import load_digits
import pandas as pd
digits = load_digits()
df = pd.DataFrame(digits.data, columns=[f'pixel_{i}' for i in range(64)])

D. from sklearn.datasets import load_digits
import pandas as pd
digits = load_digits()
df = pd.DataFrame(digits.target, columns=[f'pixel_{i}' for i in range(64)])

Solution

1. Step 1: Load digits dataset correctly

   Use load_digits() to get the dataset object with data attribute for features.

2. Step 2: Create DataFrame with correct data and columns

   Pass digits.data to pd.DataFrame() and set columns to pixel_0 to pixel_63.

3. Final Answer:

   from sklearn.datasets import load_digits\nimport pandas as pd\ndigits = load_digits()\ndf = pd.DataFrame(digits.data, columns=[f'pixel_{i}' for i in range(64)]) -> Option C

4. Quick Check:

   Use digits.data for features in DataFrame [OK]

Hint: Use digits.data for features, not digits or digits.target [OK]

Common Mistakes:

• Passing digits object directly to DataFrame
• Using digits.target as data
• Trying to load digits from CSV without file