Incrementality testing in Digital Marketing - Step-by-Step Execution

Concept Flow - Incrementality testing

Start with Audience

↓

Split into Test & Control Groups

↓

Apply Marketing Campaign to Test Group Only

↓

Measure Outcomes in Both Groups

↓

Compare Results: Test vs Control

↓

Calculate Incremental Impact

↓

Decide on Campaign Effectiveness

Incrementality testing splits an audience into two groups, applies a campaign to one, then compares results to find the true impact.

Execution Sample

Digital Marketing

Audience = 1000 people
Split: Test=500, Control=500
Apply campaign to Test only
Measure sales Test=150, Control=100
Increment = 150 - 100 = 50

This example shows splitting an audience, running a campaign on one group, and calculating the incremental sales caused by the campaign.

Analysis Table

Step	Action	Test Group Outcome	Control Group Outcome	Incremental Impact Calculation
1	Split audience into Test and Control	500 people	500 people	N/A
2	Apply campaign to Test group only	Campaign active	No campaign	N/A
3	Measure sales in Test group	150 sales	N/A	N/A
4	Measure sales in Control group	N/A	100 sales	N/A
5	Calculate incremental impact	N/A	N/A	150 - 100 = 50 sales
6	Interpret result	N/A	N/A	Campaign caused 50 extra sales

💡 Test and Control groups measured; incremental impact calculated as difference in outcomes.

State Tracker

Variable	Start	After Split	After Campaign	After Measurement	Final
Audience Size	1000	Test=500, Control=500	Same	Same	Same
Campaign Applied	No	No	Test=Yes, Control=No	Same	Same
Sales	0	0	0	Test=150, Control=100	Increment=50

Key Insights - 3 Insights

Why do we need a control group that does not see the campaign?

The control group shows what would have happened without the campaign, so comparing it with the test group reveals the true incremental effect (see execution_table rows 4 and 5).

Can we just look at sales in the test group to know the campaign's success?

No, because sales might have increased anyway. The control group's sales provide a baseline to subtract from test group sales (see execution_table rows 3 and 4).

What if the test and control groups are not similar?

Differences in groups can bias results. Proper random splitting ensures groups are alike, making the incremental impact valid (see variable_tracker 'After Split').

Visual Quiz - 3 Questions

Looking at the execution_table, what is the sales number for the control group at step 4?

A 150 sales

B 50 sales

C 100 sales

D 0 sales

💡 Hint

Check the 'Control Group Outcome' column at step 4 in the execution_table.

At which step in the execution_table is the incremental impact calculated?

A Step 5

B Step 3

C Step 2

D Step 6

💡 Hint

Look for the step where the difference between test and control sales is computed.

If the control group sales were 120 instead of 100, how would the incremental impact change?

A It would increase to 70 sales

B It would decrease to 30 sales

C It would stay 50 sales

D It would become negative

💡 Hint

Use the formula Increment = Test sales - Control sales with new control sales from variable_tracker.

Concept Snapshot

Incrementality testing splits an audience into test and control groups.
Only the test group sees the campaign.
Measure outcomes in both groups.
Subtract control results from test results.
The difference shows the campaign's true impact.

Full Transcript

Incrementality testing is a method in digital marketing to find out if a campaign truly causes more sales or actions. First, you split your audience into two groups: test and control. The test group gets the campaign, the control group does not. Then you measure the results in both groups. By subtracting the control group's results from the test group's results, you find the incremental impact caused by the campaign. This helps marketers understand if their efforts are effective or if results would have happened anyway.

Practice

1. What is the main purpose of incrementality testing in digital marketing?

easy

A. To track social media likes and shares

B. To count the total number of website visitors

C. To measure if marketing actions truly increase results

D. To create new marketing content

Solution

1. Step 1: Understand the goal of incrementality testing

   Incrementality testing aims to find out if marketing efforts cause a real increase in results, not just measure activity.

2. Step 2: Compare with other options

   Counting visitors, tracking likes, or creating content are different marketing tasks, not incrementality testing.

3. Final Answer:

   To measure if marketing actions truly increase results -> Option C

4. Quick Check:

   Incrementality testing = measure true impact [OK]

Hint: Incrementality testing checks real impact, not just activity [OK]

Common Mistakes:

• Confusing incrementality with simple tracking metrics
• Thinking it measures total audience size
• Assuming it creates marketing content

2. Which of the following is the correct setup for an incrementality test?

easy

A. Send the same ad to everyone without grouping

B. Use random ads without measuring results

C. Only track clicks without comparing groups

D. Divide audience into test and control groups

Solution

1. Step 1: Identify the key setup for incrementality testing

   Incrementality testing requires splitting the audience into a test group (receives marketing) and a control group (does not).

2. Step 2: Evaluate other options

   Sending the same ad to all, tracking clicks only, or using random ads without measurement do not allow measuring true impact.

3. Final Answer:

   Divide audience into test and control groups -> Option D

4. Quick Check:

   Test and control groups = correct setup [OK]

Hint: Test vs control groups are key to incrementality [OK]

Common Mistakes:

• Not creating a control group
• Measuring only clicks without comparison
• Assuming random ads show incrementality

3. In an incrementality test, the test group had 120 conversions and the control group had 80 conversions. What is the incrementality percentage increase?

medium

A. 50%

B. 40%

C. 33%

D. 25%

Solution

1. Step 1: Calculate the difference in conversions

   120 (test) - 80 (control) = 40 additional conversions due to marketing.

2. Step 2: Calculate incrementality percentage

   (40 / 80) * 100 = 50% increase over control group.

3. Final Answer:

   50% -> Option A

4. Quick Check:

   (120-80)/80*100 = 50% [OK]

Hint: Subtract control from test, divide by control, multiply by 100 [OK]

Common Mistakes:

• Dividing by test group instead of control
• Forgetting to multiply by 100 for percentage
• Subtracting test from control instead of reverse

4. A marketer ran an incrementality test but forgot to exclude the control group from receiving ads. What is the likely problem?

medium

A. The control group will have zero conversions

B. The test results will underestimate the true impact

C. The test group will have no conversions

D. The test will measure only ad impressions

Solution

1. Step 1: Understand the control group role

   The control group should not receive ads to serve as a baseline for comparison.

2. Step 2: Analyze the effect of ads reaching control group

   If control group sees ads, their conversions increase, reducing difference with test group, causing underestimation of true impact.

3. Final Answer:

   The test results will underestimate the true impact -> Option B

4. Quick Check:

   Control group exposed = impact underestimated [OK]

Hint: Control group must not see ads to measure true lift [OK]

Common Mistakes:

• Assuming control group conversions become zero
• Thinking test group loses conversions
• Confusing impressions with conversions

5. A company wants to test if a new email campaign increases sales. They split customers into two groups: Group A receives the email, Group B does not. After one month, Group A had 200 sales and Group B had 150 sales. However, Group A also had a higher average purchase value. How should the company calculate the true incremental revenue from the campaign?

hard

A. Calculate the difference in total sales revenue between Group A and Group B

B. Only count the number of sales in Group A

C. Multiply Group B sales by Group A average purchase value

D. Ignore purchase value and compare sales counts only

Solution

1. Step 1: Understand revenue impact requires both sales count and purchase value

   Incremental revenue depends on how many sales increased and their value, not just counts.

2. Step 2: Calculate total revenue difference

   Calculate total revenue for each group (sales x average purchase value), then subtract Group B revenue from Group A revenue to find true incremental revenue.

3. Final Answer:

   Calculate the difference in total sales revenue between Group A and Group B -> Option A

4. Quick Check:

   Incremental revenue = revenue difference [OK]

Hint: Use total revenue difference, not just sales count [OK]

Common Mistakes:

• Ignoring purchase value differences
• Using only sales count without revenue
• Incorrectly multiplying sales by wrong average value