How to Automate Pagination in Selenium: Step-by-Step Guide

To automate pagination in Selenium, locate and click the pagination controls like 'Next' buttons or page numbers inside a loop until no more pages exist. Use WebDriverWait to wait for page elements to load after each click, ensuring smooth navigation through pages.

Syntax

Here is the basic pattern to automate pagination in Selenium:

• while loop to continue until no next page
• Find the pagination control element (e.g., 'Next' button)
• Click the pagination control
• Wait for the new page content to load
• Extract or test page data

python

while True:
    try:
        next_button = driver.find_element(By.LINK_TEXT, 'Next')
        next_button.click()
        WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.CSS_SELECTOR, 'selector-for-page-content'))
        )
        # Extract or test page content here
    except NoSuchElementException:
        break  # No more pages, exit loop

Example

This example shows how to automate pagination on a website with a 'Next' button. It clicks through pages until the button disappears, printing the page number each time.

python

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException

# Setup WebDriver (Chrome example)
driver = webdriver.Chrome()
driver.get('https://example.com/pagination')

page_number = 1
while True:
    print(f'Processing page {page_number}')
    # Wait for page content to load
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, '.page-content'))
    )
    try:
        next_button = driver.find_element(By.LINK_TEXT, 'Next')
        next_button.click()
        page_number += 1
    except NoSuchElementException:
        print('No more pages to process.')
        break

driver.quit()

Output

Processing page 1
Processing page 2
Processing page 3
No more pages to process.

Common Pitfalls

• Not waiting for the page to load after clicking 'Next' can cause stale element errors.
• Using incorrect locators for pagination controls leads to NoSuchElementException.
• Infinite loops if the exit condition is not properly handled when no next page exists.
• Pagination controls might be disabled or hidden on the last page, so check for element presence carefully.

python

try:
    # Wrong: Clicking without waiting
    next_button = driver.find_element(By.LINK_TEXT, 'Next')
    next_button.click()
    # Immediately try to find content - may fail
    driver.find_element(By.CSS_SELECTOR, '.page-content')
except Exception as e:
    print('Error:', e)

# Correct approach
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

next_button = driver.find_element(By.LINK_TEXT, 'Next')
next_button.click()
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, '.page-content'))
)

Quick Reference

• Use WebDriverWait to wait for page content after pagination clicks.
• Locate pagination controls with stable locators like By.LINK_TEXT or CSS selectors.
• Handle exceptions like NoSuchElementException to detect last page.
• Use loops to automate clicking through all pages.

Key Takeaways

Always wait for page content to load after clicking pagination controls to avoid errors.

Use reliable locators to find 'Next' buttons or page numbers for pagination.

Handle exceptions to detect when no more pages exist and stop the loop.

Automate pagination by looping clicks until the last page is reached.

Test your pagination automation on the actual website to adjust locators and waits.