A/B testing landing pages in Digital Marketing - Step-by-Step Execution

Concept Flow - A/B testing landing pages

Create two versions: A and B

↓

Split visitors randomly

↓

Visitors see A

↓

Collect data on

↓

visitor actions

↓

Compare results

↓

Choose better page

Visitors are randomly split to see two page versions; their actions are tracked and compared to find the better performing page.

Execution Sample

Digital Marketing

Version A: Blue button
Version B: Red button
Split 1000 visitors evenly
Track clicks on button
Compare click rates

This test compares two landing page versions with different button colors by measuring which gets more clicks.

Analysis Table

Step	Action	Visitors in A	Visitors in B	Clicks in A	Clicks in B	Result
1	Split visitors	500	500	0	0	Visitors split evenly
2	Visitors see pages	500	500	0	0	Pages shown to visitors
3	Track clicks	500	500	120	150	Clicks recorded
4	Calculate click rate	500	500	24%	30%	Click rates calculated
5	Compare rates	500	500	24%	30%	Version B performs better
6	Choose winner	500	500	24%	30%	Select Version B for launch

💡 Test ends after comparing click rates and selecting the better performing page

State Tracker

Variable	Start	After Step 1	After Step 3	After Step 5	Final
Visitors in A	0	500	500	500	500
Visitors in B	0	500	500	500	500
Clicks in A	0	0	120	120	120
Clicks in B	0	0	150	150	150
Click rate A	0%	0%	24%	24%	24%
Click rate B	0%	0%	30%	30%	30%

Key Insights - 3 Insights

Why do we split visitors evenly between versions A and B?

Splitting evenly ensures a fair comparison by giving both versions the same chance to be seen, as shown in Step 1 of the execution_table.

What does the click rate tell us in this test?

Click rate shows the percentage of visitors who clicked the button, helping us measure which page is more effective, as calculated in Step 4.

Why do we choose the version with the higher click rate?

The higher click rate means more visitors took the desired action, indicating better performance, as concluded in Step 5 and 6.

Visual Quiz - 3 Questions

Look at the execution_table at Step 3: How many clicks did Version B receive?

A 120

B 150

C 500

D 30%

💡 Hint

Check the 'Clicks in B' column at Step 3 in the execution_table.

At which step does the test decide which version performs better?

A Step 2

B Step 4

C Step 5

D Step 6

💡 Hint

Look for the step where click rates are compared in the execution_table.

If Version A had 200 clicks instead of 120, what would happen to its click rate?

A It would increase to 40%

B It would stay at 24%

C It would decrease

D It would be 30%

💡 Hint

Calculate click rate as clicks divided by visitors (200/500) from variable_tracker.

Concept Snapshot

A/B testing landing pages:
- Create two versions (A and B)
- Split visitors randomly to each
- Track key actions (e.g., clicks)
- Compare results to find better version
- Use winner to improve conversions

Full Transcript

A/B testing landing pages means creating two different versions of a page and showing each to a random half of visitors. We track how visitors act on each page, like clicking a button. Then we compare the results to see which page works better. The better page is chosen to improve overall success. This process helps make decisions based on real visitor behavior.

Practice

1. What is the main purpose of A/B testing landing pages?

easy

A. To compare two versions and find which performs better

B. To create multiple unrelated web pages

C. To increase website loading speed

D. To design logos for the website

Solution

1. Step 1: Understand the goal of A/B testing

   A/B testing is used to compare two versions of a webpage to see which one works better for visitors.

2. Step 2: Identify the correct purpose

   Among the options, only comparing two versions to find the better one matches the goal of A/B testing.

3. Final Answer:

   To compare two versions and find which performs better -> Option A

4. Quick Check:

   A/B testing purpose = Compare versions [OK]

Hint: Remember: A/B testing compares two versions only [OK]

Common Mistakes:

• Thinking A/B testing creates many unrelated pages
• Confusing A/B testing with website speed optimization
• Assuming A/B testing is for design tasks like logos

2. Which of the following is a correct step in setting up an A/B test for landing pages?

easy

A. Change the entire website design during the test

B. Test multiple changes at once to save time

C. Randomly split visitors between two page versions

D. Ignore visitor data and guess the better page

Solution

1. Step 1: Review proper A/B testing setup

   A/B testing requires splitting visitors randomly to fairly compare two versions.

2. Step 2: Identify the correct step

   Only randomly splitting visitors between two versions is a correct and essential step.

3. Final Answer:

   Randomly split visitors between two page versions -> Option C

4. Quick Check:

   Visitor split = Random between versions [OK]

Hint: Always split visitors randomly for fair comparison [OK]

Common Mistakes:

• Testing many changes at once causing unclear results
• Changing whole website instead of just landing pages
• Ignoring real visitor data during the test

3. If version A of a landing page has a 5% conversion rate and version B has a 7% conversion rate after testing with equal visitors, which version should you choose?

medium

A. Version A because it was tested first

B. Version B because it has a higher conversion rate

C. Neither, because conversion rates are too close

D. Both, by showing them randomly forever

Solution

1. Step 1: Compare conversion rates of both versions

   Version A has 5% and version B has 7%, so B performs better.

2. Step 2: Decide based on performance

   Choose the version with the higher conversion rate to improve results.

3. Final Answer:

   Version B because it has a higher conversion rate -> Option B

4. Quick Check:

   Higher conversion rate = Better version [OK]

Hint: Pick the version with the higher conversion rate [OK]

Common Mistakes:

• Choosing version tested first instead of better performing
• Ignoring small but meaningful conversion differences
• Continuing to show both versions without decision

4. A marketer ran an A/B test but changed the headline and the call-to-action button at the same time. What is the main problem with this approach?

medium

A. It makes it impossible to know which change affected results

B. It speeds up the test and gives clearer results

C. It reduces the number of visitors needed

D. It improves the website loading speed

Solution

1. Step 1: Understand the problem with multiple changes

   Changing more than one element at once confuses which change caused the result.

2. Step 2: Identify the main issue

   The main problem is losing clarity on which change improved or hurt performance.

3. Final Answer:

   It makes it impossible to know which change affected results -> Option A

4. Quick Check:

   Multiple changes = unclear results [OK]

Hint: Test one change at a time for clear results [OK]

Common Mistakes:

• Believing multiple changes speed up or clarify tests
• Thinking fewer visitors are needed with many changes
• Confusing test changes with website speed improvements

5. You want to improve a landing page using A/B testing. You have two versions: Version A with a blue button and Version B with a red button. After 1000 visitors each, Version A has 50 conversions and Version B has 55 conversions. What should you do next?

hard

A. Declare Version B the winner and switch all traffic to it immediately

B. Change both button color and headline and test again

C. Ignore the test and pick a new design randomly

D. Run the test longer to collect more data before deciding

Solution

1. Step 1: Analyze the conversion difference

   Version A has 5% conversion (50/1000), Version B has 5.5% (55/1000). The difference is small.

2. Step 2: Decide based on data size and difference

   Small differences with limited visitors need more data for reliable results.

3. Final Answer:

   Run the test longer to collect more data before deciding -> Option D

4. Quick Check:

   Small difference + limited data = test longer [OK]

Hint: Small differences need more visitors before deciding [OK]

Common Mistakes:

• Choosing winner too soon with small data
• Changing multiple elements before finalizing test
• Ignoring test results and guessing