Incrementality testing in Digital Marketing - Mini Project: Build & Apply

Incrementality Testing in Digital Marketing

📖 Scenario: You work as a digital marketer for an online store. Your manager wants to understand if a recent advertising campaign actually increased sales or if sales would have happened anyway. To find out, you will set up a simple incrementality test.

🎯 Goal: Build a step-by-step plan to conduct an incrementality test by creating groups, setting a baseline, measuring results, and concluding the campaign's impact.

📋 What You'll Learn

Create two groups: a control group and a test group

Set a baseline sales number for comparison

Calculate the difference in sales between test and control groups

Conclude if the campaign caused an increase in sales

💡 Why This Matters

🌍 Real World

Incrementality testing helps marketers understand if their campaigns truly drive extra sales or actions beyond what would have happened anyway.

💼 Career

Digital marketers, analysts, and campaign managers use incrementality testing to optimize advertising budgets and improve campaign effectiveness.

1

Create Customer Groups

Create two lists called control_group and test_group. Add these exact customer IDs to control_group: 101, 102, 103. Add these exact customer IDs to test_group: 201, 202, 203.

Digital Marketing

# Create control_group and test_group lists with specified customer IDs
# Your code here

Hint

Use square brackets to create lists and separate customer IDs with commas.

2

Set Baseline Sales

Create a variable called baseline_sales and set it to the exact value 1500. This represents sales before the campaign.

Digital Marketing

control_group = [101, 102, 103]
test_group = [201, 202, 203]
# Set baseline_sales to 1500
# Your code here

Hint

Use a simple assignment to create the baseline_sales variable.

3

Calculate Sales Difference

Create two variables: control_sales with value 1400 and test_sales with value 1800. Then create a variable called incremental_sales that calculates the difference between test_sales and control_sales.

Digital Marketing

control_group = [101, 102, 103]
test_group = [201, 202, 203]
baseline_sales = 1500
# Define control_sales and test_sales, then calculate incremental_sales
# Your code here

Hint

Subtract control_sales from test_sales to find incremental_sales.

4

Conclude Campaign Impact

Create a variable called campaign_effect and set it to True if incremental_sales is greater than zero, otherwise False. Use an if-else statement.

Digital Marketing

control_group = [101, 102, 103]
test_group = [201, 202, 203]
baseline_sales = 1500
control_sales = 1400
test_sales = 1800
incremental_sales = test_sales - control_sales
# Set campaign_effect to True if incremental_sales > 0, else False
# Your code here

Hint

Use a simple if-else to assign True or False based on incremental_sales.

Practice

1. What is the main purpose of incrementality testing in digital marketing?

easy

A. To track social media likes and shares

B. To count the total number of website visitors

C. To measure if marketing actions truly increase results

D. To create new marketing content

Solution

1. Step 1: Understand the goal of incrementality testing

   Incrementality testing aims to find out if marketing efforts cause a real increase in results, not just measure activity.

2. Step 2: Compare with other options

   Counting visitors, tracking likes, or creating content are different marketing tasks, not incrementality testing.

3. Final Answer:

   To measure if marketing actions truly increase results -> Option C

4. Quick Check:

   Incrementality testing = measure true impact [OK]

Hint: Incrementality testing checks real impact, not just activity [OK]

Common Mistakes:

• Confusing incrementality with simple tracking metrics
• Thinking it measures total audience size
• Assuming it creates marketing content

2. Which of the following is the correct setup for an incrementality test?

easy

A. Send the same ad to everyone without grouping

B. Use random ads without measuring results

C. Only track clicks without comparing groups

D. Divide audience into test and control groups

Solution

1. Step 1: Identify the key setup for incrementality testing

   Incrementality testing requires splitting the audience into a test group (receives marketing) and a control group (does not).

2. Step 2: Evaluate other options

   Sending the same ad to all, tracking clicks only, or using random ads without measurement do not allow measuring true impact.

3. Final Answer:

   Divide audience into test and control groups -> Option D

4. Quick Check:

   Test and control groups = correct setup [OK]

Hint: Test vs control groups are key to incrementality [OK]

Common Mistakes:

• Not creating a control group
• Measuring only clicks without comparison
• Assuming random ads show incrementality

3. In an incrementality test, the test group had 120 conversions and the control group had 80 conversions. What is the incrementality percentage increase?

medium

A. 50%

B. 40%

C. 33%

D. 25%

Solution

1. Step 1: Calculate the difference in conversions

   120 (test) - 80 (control) = 40 additional conversions due to marketing.

2. Step 2: Calculate incrementality percentage

   (40 / 80) * 100 = 50% increase over control group.

3. Final Answer:

   50% -> Option A

4. Quick Check:

   (120-80)/80*100 = 50% [OK]

Hint: Subtract control from test, divide by control, multiply by 100 [OK]

Common Mistakes:

• Dividing by test group instead of control
• Forgetting to multiply by 100 for percentage
• Subtracting test from control instead of reverse

4. A marketer ran an incrementality test but forgot to exclude the control group from receiving ads. What is the likely problem?

medium

A. The control group will have zero conversions

B. The test results will underestimate the true impact

C. The test group will have no conversions

D. The test will measure only ad impressions

Solution

1. Step 1: Understand the control group role

   The control group should not receive ads to serve as a baseline for comparison.

2. Step 2: Analyze the effect of ads reaching control group

   If control group sees ads, their conversions increase, reducing difference with test group, causing underestimation of true impact.

3. Final Answer:

   The test results will underestimate the true impact -> Option B

4. Quick Check:

   Control group exposed = impact underestimated [OK]

Hint: Control group must not see ads to measure true lift [OK]

Common Mistakes:

• Assuming control group conversions become zero
• Thinking test group loses conversions
• Confusing impressions with conversions

5. A company wants to test if a new email campaign increases sales. They split customers into two groups: Group A receives the email, Group B does not. After one month, Group A had 200 sales and Group B had 150 sales. However, Group A also had a higher average purchase value. How should the company calculate the true incremental revenue from the campaign?

hard

A. Calculate the difference in total sales revenue between Group A and Group B

B. Only count the number of sales in Group A

C. Multiply Group B sales by Group A average purchase value

D. Ignore purchase value and compare sales counts only

Solution

1. Step 1: Understand revenue impact requires both sales count and purchase value

   Incremental revenue depends on how many sales increased and their value, not just counts.

2. Step 2: Calculate total revenue difference

   Calculate total revenue for each group (sales x average purchase value), then subtract Group B revenue from Group A revenue to find true incremental revenue.

3. Final Answer:

   Calculate the difference in total sales revenue between Group A and Group B -> Option A

4. Quick Check:

   Incremental revenue = revenue difference [OK]

Hint: Use total revenue difference, not just sales count [OK]

Common Mistakes:

• Ignoring purchase value differences
• Using only sales count without revenue
• Incorrectly multiplying sales by wrong average value