How to Handle Infinite Scroll in Selenium: Fix and Best Practices

To handle infinite scroll in Selenium, you need to scroll the page down repeatedly until no new content loads. Use JavaScript execution with window.scrollTo inside a loop and wait for new elements to appear before continuing. This ensures Selenium loads all dynamic content before assertions.

Why This Happens

Infinite scroll pages load more content dynamically as you scroll down. If you try to test such pages without scrolling, Selenium only sees the initially loaded content. This causes tests to miss elements or fail when trying to interact with content that isn't loaded yet.

Many beginners write code that tries to find elements immediately without scrolling, which leads to incomplete or failed tests.

python

from selenium import webdriver
from selenium.webdriver.common.by import By

# Open browser and go to infinite scroll page
browser = webdriver.Chrome()
browser.get('https://example.com/infinite-scroll')

# Try to find all items without scrolling
items = browser.find_elements(By.CLASS_NAME, 'item')
print(f'Items found: {len(items)}')

browser.quit()

Output

Items found: 10  # Only initial items, missing dynamically loaded ones

The Fix

To fix this, scroll down the page step-by-step using JavaScript until no new content loads. After each scroll, wait briefly for new elements to appear. This way, Selenium can load and detect all items on the page.

python

import time
from selenium import webdriver
from selenium.webdriver.common.by import By

browser = webdriver.Chrome()
browser.get('https://example.com/infinite-scroll')

last_height = browser.execute_script('return document.body.scrollHeight')

while True:
    # Scroll down to bottom
    browser.execute_script('window.scrollTo(0, document.body.scrollHeight);')
    time.sleep(2)  # Wait for new content to load

    new_height = browser.execute_script('return document.body.scrollHeight')
    if new_height == last_height:
        break  # No more new content
    last_height = new_height

items = browser.find_elements(By.CLASS_NAME, 'item')
print(f'Items found after scrolling: {len(items)}')

browser.quit()

Output

Items found after scrolling: 50  # All items loaded

Prevention

To avoid issues with infinite scroll in the future, always:

• Detect if the page uses infinite scroll before writing tests.
• Use scrolling loops with checks for new content loading.
• Include waits after scrolling to let content load fully.
• Use explicit waits or expected conditions for better reliability.
• Keep your locators specific and stable to avoid flaky tests.

Related Errors

Common related errors include:

• NoSuchElementException: Happens when elements are not loaded yet because scrolling was not done.
• StaleElementReferenceException: Occurs if the page reloads or updates elements during scrolling.
• TimeoutException: When waits are too short or missing after scrolling.

Quick fixes: add scrolling loops, use explicit waits, and handle exceptions gracefully.

Key Takeaways

Always scroll down incrementally to load all dynamic content on infinite scroll pages.

Use JavaScript execution with window.scrollTo and wait after each scroll.

Check if the page height changes to know when to stop scrolling.

Use explicit waits to ensure elements are loaded before interacting.

Handle common exceptions by adding retries and waits in your test code.