Build a High-Performance Web Scraper with Playwright and Google Sheets

ByMohammad Farhad

Build a high-performance Python web scraper with Playwright. Learn to handle infinite-scroll, bypass JS-rendering, and sync data to Google Sheets with deduplication.

Introduction

Build a scraper that handles JavaScript-rendered, infinitely-scrolling pages and writes the results straight into a Google Sheet — no database, no export scripts, no manual copy-pasting. You'll need Python, Playwright for browser automation, and a Google service account for Sheets access.

Table of Contents

  • [Prerequisites & Tech Stack](#prerequisites–tech-stack)
  • [Architecture Overview](#architecture-overview)
  • [Step 1: Set Up Google Sheets API Access](#step-1-set-up-google-sheets-api-access)
  • [Step 2: Connect to the Sheet and Centralize Config](#step-2-connect-to-the-sheet-and-centralize-config)
  • [Step 3: Launch a Headless Browser and Load the Page](#step-3-launch-a-headless-browser-and-load-the-page)
  • [Step 4: Handle Infinite Scroll Without a Fixed Wait](#step-4-handle-infinite-scroll-without-a-fixed-wait)
  • [Step 5: Extract Data With Error Logging](#step-5-extract-data-with-error-logging)
  • [Step 6: Write to Google Sheets — Batched and Deduplicated](#step-6-write-to-google-sheets–batched-and-deduplicated)
  • [Step 7: Scrape Multiple URLs Concurrently](#step-7-scrape-multiple-urls-concurrently)
  • [Verification & Testing](#verification–testing)
  • [Troubleshooting & Common Errors](#troubleshooting–common-errors)
  • [FAQ](#faq)
  • [Conclusion & Next Steps](#conclusion–next-steps)

Prerequisites & Tech Stack

| Tool | Version | Why it is needed | |——|———|——————-| | Python | 3.14.3 | Runtime for the script | | playwright | 1.60.0 | Drives a real headless browser to render JavaScript-heavy pages | | gspread | 6.2.1 | Python client for the Google Sheets API | | google-auth | 2.54.0 | Authenticates the script against Google using a service account | | beautifulsoup4 (dev only) | 4.15.0 | Used only while you're inspecting saved HTML to find CSS selectors — not part of the running scraper | | lxml (dev only) | 6.1.1 | Fast parser backend for BeautifulSoup, also dev-only |

Install everything in one go:

pip install playwright gspread google-auth beautifulsoup4 lxml
playwright install chromium

beautifulsoup4 and lxml never appear inside the production scraper — they're used once, in a throwaway helper script, to inspect a saved page and confirm your selectors before you commit to them. If you already know your selectors, you can skip installing them.

The playwright install chromium step is easy to forget — Playwright ships as a Python package, but the actual browser binary is downloaded separately.

Target site for this tutorial: quotes.toscrape.com/scroll — a sandbox site built specifically for practicing scraping. It loads quotes via infinite scroll, the same pattern used by many real product/listing pages, so what you learn here transfers directly.

Architecture Overview

┌──────────────┐   ┌───────────────────┐   ┌──────────────────┐   ┌─────────────────┐   ┌────────────────┐
│  Target Pages  │-->│ Playwright Browser │-->│ Extraction Logic  │-->│ Dedup + Batching  │-->│  Google Sheet   │
│  (N URLs,      │   │ (concurrent pages, │   │ (locate cards,     │   │ (skip existing,    │   │  (via gspread + │
│   infinite      │   │  condition-based   │   │  log failures)     │   │  flush every N)    │   │   service acct, │
│   scroll)       │   │  waits)            │   │                     │   │                     │   │   lock-guarded) │
└──────────────┘   └───────────────────┘   └──────────────────┘   └─────────────────┘   └────────────────┘

What this looks like in the browser:

┌──────────────────────────────────────┐
│  "The world as we have created..."     │
│  — Albert Einstein   [tags: change]     │ ← one .quote card
├──────────────────────────────────────┤
│  "It is our choices, Harry..."          │
│  — J.K. Rowling   [tags: choices]       │ ← another .quote card
├──────────────────────────────────────┤
│           (more load as you scroll)      │
└──────────────────────────────────────┘

What the destination Google Sheet looks like after a run:

| Quote | Author | Tags | |—|—|—| | The world as we have created it… | Albert Einstein | change, deep-thoughts | | It is our choices, Harry… | J.K. Rowling | abilities, choices |

  • Target Pages — one or more JS-rendered, infinite-scroll URLs, scraped concurrently instead of one at a time.
  • Playwright Browser — a real (headless) Chromium instance per page, waiting on actual conditions instead of fixed sleeps.
  • Extraction Logic — CSS selectors (centralized in one config block) that pull text fields out of each card, logging any failure instead of swallowing it.
  • Dedup + Batching — every batch is checked against rows already in the sheet before writing, and writes happen in fixed-size chunks instead of holding the whole page in memory.
  • Google Sheet — the final destination, written to under a lock so concurrent pages never write at the same instant.

Step 1: Set Up Google Sheets API Access

What we are doing: Creating a Google Cloud service account so the script can write to Sheets without a human logging in interactively.

  1. Go to the Google Cloud Console, create a project, and enable the Google Sheets API and Google Drive API.
  2. Create a Service Account under "IAM & Admin" → "Service Accounts."
  3. Generate a JSON key for that account and save it as credentials.json in your project folder.
  4. Open the target Google Sheet and share it with the service account's email (found inside credentials.json, field client_email), giving it Editor access.

Key insight: Creating the credentials file alone grants nothing — the sheet only becomes writable once you explicitly share it with the service account's email. This is the most common setup mistake.

Security rule: Never commit credentials.json to version control. Add it to .gitignore immediately.

Step 2: Connect to the Sheet and Centralize Config

What we are doing: Authenticating with the service account, and putting every site-specific value — selectors, batch size, concurrency limit — in one place instead of scattered through the script.

import sys
import asyncio
import gspread
from google.oauth2.service_account import Credentials

sys.stdout.reconfigure(encoding='utf-8')

SHEET_LINK = "https://docs.google.com/spreadsheets/d/YOUR_SHEET_ID/edit"

print("Connecting to Google Sheets...")
SCOPES = ['https://www.googleapis.com/auth/spreadsheets', 'https://www.googleapis.com/auth/drive']
creds = Credentials.from_service_account_file('credentials.json', scopes=SCOPES)
client = gspread.authorize(creds)

try:
    sheet = client.open_by_url(SHEET_LINK).sheet1
except Exception as e:
    print("Error connecting to Google Sheet! Details:", e)
    sys.exit(1)

# Every selector lives here. If the target site's markup changes,
# this is the only block you need to touch.
SELECTORS = {
    "card": ".quote",
    "text": ".text",
    "author": ".author",
    "tags": ".tags .tag",
}

URLS_TO_SCRAPE = [
    "https://quotes.toscrape.com/scroll",
]

BATCH_SIZE = 50            # flush to the sheet every N scraped cards
MAX_CONCURRENT_PAGES = 3   # how many URLs to scrape at the same time

sheet_lock = asyncio.Lock()  # prevents two concurrent pages from writing at once

Key insight: open_by_url(...).sheet1 grabs the first tab of the spreadsheet, and the try/except fails loudly at startup if the URL is wrong or the share step was skipped. Pulling SELECTORS, BATCH_SIZE, and MAX_CONCURRENT_PAGES to the top means adapting this script to a new site — or tuning its behavior — never means hunting through the scraping logic itself.

Step 3: Launch a Headless Browser and Load the Page

What we are doing: Opening a real browser instance and navigating to the page you want to scrape.

from playwright.async_api import async_playwright

async def scrape_url(browser, url, seen_keys, semaphore):
    async with semaphore:
        page = await browser.new_page()
        print(f"\nLoading page: {url}")
        await page.goto(url, wait_until="networkidle")

Key insight: chromium.launch(headless=True) (shown in Step 7) starts a browser with no visible window, which is what makes this runnable on a server. wait_until="networkidle" waits until the page stops firing background requests — important for sites where content loads via API calls rather than appearing in the raw HTML. The async with semaphore: block is what will let multiple URLs run at once in Step 7, without overwhelming the target site or your machine.

Test it now:

python scraper.py

Expected output:

Connecting to Google Sheets...
Loading page: https://quotes.toscrape.com/scroll

Step 4: Handle Infinite Scroll Without a Fixed Wait

What we are doing: Scrolling repeatedly until no new cards load — but instead of guessing a fixed delay, waiting on the actual condition: more cards existing in the DOM.

async def wait_for_new_cards(page, previous_count, timeout=10000):
    """Wait until more cards appear, or give up after `timeout` ms.
    Replaces a fixed sleep with a condition the page itself reports."""
    try:
        await page.wait_for_function(
            f"document.querySelectorAll('{SELECTORS['card']}').length > {previous_count}",
            timeout=timeout,
        )
    except Exception:
        pass  # no new cards within timeout -- the caller re-checks the count itself

# inside scrape_url, after page.goto(...):
        previous_count = 0
        batch = []
        skipped = 0

        while True:
            await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
            await wait_for_new_cards(page, previous_count)

            cards = await page.locator(SELECTORS["card"]).all()
            current_count = len(cards)

            if current_count == previous_count:
                await wait_for_new_cards(page, previous_count, timeout=3000)
                cards = await page.locator(SELECTORS["card"]).all()
                if len(cards) == previous_count:
                    break

Key insight: A flat page.wait_for_timeout(2000) is a guess — too short on a slow connection, wasted time on a fast one. wait_for_function polls the page's own DOM and returns the moment new cards actually appear, capped by a timeout so a page that's genuinely done loading doesn't hang forever. The double-check before breaking (if len(cards) == previous_count) still guards against false stops mid-render.

If your target site uses a "Load More" button instead of pure infinite scroll, click it inside the same loop, then reuse wait_for_new_cards instead of a fixed sleep:

            load_more_btn = page.locator("text='Load More'")
            if await load_more_btn.is_visible():
                try:
                    await load_more_btn.click()
                    await wait_for_new_cards(page, previous_count)
                except Exception:
                    pass

Finding the right selector (.quote here) is a manual step: open the page in a browser, press F12, and inspect the repeating element wrapping each item. To confirm a selector without re-launching a browser every time:

# inspect_selectors.py — development only, not part of the production scraper
import asyncio
from playwright.async_api import async_playwright

async def inspect():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto("https://quotes.toscrape.com/scroll", wait_until='networkidle')
        html = await page.content()
        with open('page_snapshot.html', 'w', encoding='utf-8') as f:
            f.write(html)
        await browser.close()

asyncio.run(inspect())

Then test selectors against page_snapshot.html with BeautifulSoup locally — much faster than re-running a live browser on every attempt.

Step 5: Extract Data With Error Logging

What we are doing: Looping over newly-loaded cards and pulling out text fields — and when a card fails, recording why instead of silently dropping it.

            new_cards = cards[previous_count:current_count]
            for card in new_cards:
                try:
                    text = await card.locator(SELECTORS["text"]).inner_text()
                    author = await card.locator(SELECTORS["author"]).inner_text()
                    tag_elements = await card.locator(SELECTORS["tags"]).all()
                    tags = ", ".join([await t.inner_text() for t in tag_elements])

                    batch.append([text.strip(), author.strip(), tags])
                except Exception as e:
                    skipped += 1
                    print(f"  [skip] Could not extract a card: {e}")

            previous_count = current_count
            print(f"Loaded {current_count} cards so far...")

Key insight: cards[previous_count:current_count] only processes cards loaded in this pass — already-extracted cards aren't re-scraped on every scroll. The except block no longer just swallows the error: it counts the skip and prints the actual exception, so a run that drops 12 cards tells you so instead of leaving you to notice the gap later in the sheet.

Step 6: Write to Google Sheets — Batched and Deduplicated

What we are doing: Flushing rows to the sheet in fixed-size batches instead of holding an entire page in memory, and skipping any row whose unique key is already in the sheet so re-running the script doesn't create duplicates.

def load_existing_keys():
    """Read rows already in the sheet so re-runs don't duplicate them."""
    existing_rows = sheet.get_all_values()[1:]  # skip header row
    return {row[0] for row in existing_rows if row}  # using the quote text as the unique key

async def flush_batch(batch, seen_keys):
    """Write a batch to the sheet, skipping duplicates. The duplicate check,
    the write, and the seen_keys update all happen inside one lock so two
    concurrent pages can't both decide the same row is new."""
    if not batch:
        return
    async with sheet_lock:
        new_rows = [row for row in batch if row[0] not in seen_keys]
        if not new_rows:
            print(f"Skipped {len(batch)} duplicate row(s), nothing new to write.")
            return
        sheet.append_rows(new_rows)
        seen_keys.update(row[0] for row in new_rows)
    print(f"Wrote {len(new_rows)} new row(s) ({len(batch) - len(new_rows)} duplicate(s) skipped).")

Call it from inside the scroll loop whenever the batch fills up, and once more after the loop ends to flush any remainder:

            if len(batch) >= BATCH_SIZE:
                await flush_batch(batch, seen_keys)
                batch = []

        # after the while loop:
        await flush_batch(batch, seen_keys)
        if skipped:
            print(f"Finished {url}: {skipped} card(s) skipped due to extraction errors.")
        await page.close()

Key insight: BATCH_SIZE = 50 means the script never holds more than 50 unwritten rows in memory, no matter how large the page is. The duplicate check, the append_rows() call, and updating seen_keys all happen inside sheet_lock — if those three steps weren't atomic, two concurrent pages could each check the key before either had written, and both would append the same row.

Step 7: Scrape Multiple URLs Concurrently

What we are doing: Running several URLs through the same browser at once, instead of finishing one fully before starting the next.

async def scrape_all():
    async with async_playwright() as p:
        print("Starting browser...")
        browser = await p.chromium.launch(headless=True)

        if not sheet.get_all_values():
            sheet.append_row(["Quote", "Author", "Tags"])

        seen_keys = load_existing_keys()
        print(f"Found {len(seen_keys)} existing row(s) in the sheet -- duplicates will be skipped.")

        semaphore = asyncio.Semaphore(MAX_CONCURRENT_PAGES)
        tasks = [scrape_url(browser, url, seen_keys, semaphore) for url in URLS_TO_SCRAPE]
        await asyncio.gather(*tasks)

        await browser.close()
        print("\nAll pages scraped successfully!")

if __name__ == "__main__":
    asyncio.run(scrape_all())

Key insight: asyncio.Semaphore(MAX_CONCURRENT_PAGES) caps how many scrape_url tasks run at once — asyncio.gather() launches all of them, but each waits at async with semaphore: (from Step 3) until a slot is free. This turns a script that scrapes 10 URLs one after another into one that scrapes 3 at a time, without rewriting the per-page logic at all. The header row is only written once, and only if the sheet is currently empty — re-running the script never re-clears existing data, which is what makes the deduplication in Step 6 meaningful.

Verification & Testing

Run the full script end-to-end:

python scraper.py

Expected terminal output:

Connecting to Google Sheets...
Starting browser...
Found 0 existing row(s) in the sheet -- duplicates will be skipped.

Loading page: https://quotes.toscrape.com/scroll
Loaded 10 cards so far...
Loaded 20 cards so far...
Loaded 30 cards so far...
Wrote 30 new row(s) (0 duplicate(s) skipped).

All pages scraped successfully!

Run it a second time without changing anything, and confirm the duplicate handling works:

Found 30 existing row(s) in the sheet -- duplicates will be skipped.
...
Skipped 30 duplicate row(s), nothing new to write.

Open the target Google Sheet — you should see a header row (Quote, Author, Tags) followed by exactly one row per quote, no matter how many times you run the script.

Troubleshooting & Common Errors

| Error Message | Cause | Fix | |—————-|——-|—–| | UnicodeEncodeError | Windows terminal codepage can't print non-ASCII scraped text | Add sys.stdout.reconfigure(encoding='utf-8') at the top of the script | | 403 Permission Denied (Sheets) | The target sheet isn't shared with the service account | Share the sheet with the client_email from credentials.json, with Editor access | | gspread.exceptions.SpreadsheetNotFound | Wrong sheet URL, or Drive API not enabled | Double-check SHEET_LINK, confirm both Sheets and Drive APIs are enabled in Google Cloud Console | | Cards stay at 0 / selector matches nothing | CSS selector doesn't match the live site's structure | Re-inspect the page with the snapshot technique from Step 4 — class names change between site versions | | Script hangs at page.goto(...) | wait_until="networkidle" never resolves on sites with persistent background requests (ads, analytics pings) | Switch to wait_until="domcontentloaded" and rely on wait_for_new_cards for the actual content wait | | APIError: Quota exceeded | Too many write calls to the Sheets API in a short window | Increase BATCH_SIZE so fewer, larger append_rows() calls are made | | Same rows appear twice after re-running | seen_keys built from the wrong column, or the unique key isn't actually unique | Make sure load_existing_keys() reads the same column you compare against in flush_batch, and pick a key that's truly unique per item | | Target site starts returning errors / blocking requests | Too many concurrent pages, or scraping faster than a real user would | Lower MAX_CONCURRENT_PAGES, and confirm wait_for_new_cards isn't being bypassed |

FAQ

Can this approach work on any website? Yes, as long as the site doesn't actively block headless browsers. Update the SELECTORS dict and the pagination mechanism (infinite scroll vs. a "Load More" button, shown in Step 4) per site — the rest of the pipeline (dedup, batching, concurrency) stays the same.

Why use Playwright instead of requests and BeautifulSoup? requests only fetches the raw HTML returned by the server. If a site loads content via JavaScript after the initial page load, that content won't exist in the response. Playwright runs an actual browser engine, so it sees the page exactly as a user would.

Is it safe to scrape any site this way? Always check a site's terms of service and robots.txt, and keep MAX_CONCURRENT_PAGES and the scroll throttling reasonable. quotes.toscrape.com is explicitly built for scraping practice; most production sites are not, and need their own review before scraping.

What happens if the script is interrupted halfway through? Because writes happen every BATCH_SIZE rows rather than once at the very end, an interruption loses at most one unflushed batch — and because every row is deduplicated against the sheet, simply re-running the script picks up where it left off without creating repeats.

Conclusion & Next Steps

What you've built:

  • A headless browser pipeline that renders JavaScript-heavy, infinite-scroll pages
  • A condition-based pagination loop that waits on actual page state instead of a fixed delay
  • Per-card extraction that logs and counts failures instead of silently dropping them
  • Batched, deduplicated writes to Google Sheets, safe to re-run without creating duplicate rows
  • Concurrent scraping across multiple URLs, guarded by a semaphore and a write lock

From here, the natural extensions are: retrying a failed page navigation automatically instead of letting it end the task, persisting seen_keys to disk so a restart doesn't need to re-read the entire sheet, and running the whole script on a schedule.

The next part of this series covers exactly that — adding retries, scheduling, and basic monitoring so this scraper can run unattended. [Read Part 2 →](#)

Similar Posts