Web scraping

Scrape websites using cloud browsers that handle JavaScript rendering, bot protection, and dynamic content. Browserbase gives you reliable infrastructure for web scraping workflows, whether you’re using Stagehand or Playwright.

Scale scraping across concurrent sessions without managing infrastructure
Browse protected sites with Browserbase’s Verified
Rotate IPs and geolocations with proxies
Debug and monitor scraping runs with session replays and live views

Need scheduled scraping or webhook-triggered data collection? Functions provide serverless scraping that can be invoked on-demand or on a schedule—perfect for building data pipelines and monitoring workflows.

Template

Get started quickly with a ready-to-use web scraping template.

Company Value Prop Generator

Clone, configure, and run in minutes

Example: Scraping a book catalog

To demonstrate web scraping with Browserbase, this example extracts book titles, prices, and availability from a sample catalog site.

Code example

Node.js
Python

import { Stagehand } from "@browserbasehq/stagehand";
import { z } from "zod";
import dotenv from "dotenv";
dotenv.config();

const stagehand = new Stagehand({
    env: "BROWSERBASE",
    verbose: 0,
});

async function scrapeBooks() {
    await stagehand.init();
    const page = stagehand.context.pages()[0];

    await page.goto("https://books.toscrape.com/");

    const scrape = await stagehand.extract({
        instruction: "Extract the books from the page",
        schema: z.object({
            books: z.array(z.object({
                title: z.string(),
                price: z.string(),
                image: z.string(),
                inStock: z.string(),
                link: z.string(),
            }))
        }),
    });

    console.log(scrape.books);

    await stagehand.close();
}

scrapeBooks().catch(console.error);

import os
import asyncio
from stagehand import AsyncStagehand
from dotenv import load_dotenv

load_dotenv()

async def main():
    client = AsyncStagehand(
        browserbase_api_key=os.environ["BROWSERBASE_API_KEY"],
        browserbase_project_id=os.environ["BROWSERBASE_PROJECT_ID"],
        model_api_key=os.environ["MODEL_API_KEY"],
    )
    session = await client.sessions.create(model_name="google/gemini-2.5-flash")

    # Navigate to the site
    await session.navigate(url="https://books.toscrape.com/")

    # Extract structured data from the page
    extract_response = await session.extract(
        instruction="Extract the books from the page including title, price, and stock status",
        schema={
            "type": "object",
            "properties": {
                "books": {
                    "type": "array",
                    "items": {
                        "type": "object",
                        "properties": {
                            "title": {"type": "string"},
                            "price": {"type": "string"},
                            "inStock": {"type": "string"},
                        },
                    },
                },
            },
        },
    )

    print(extract_response.data.result)

    await session.end()

if __name__ == "__main__":
    asyncio.run(main())

Example output

[
  {
    title: 'A Light in the Attic',
    price: '£51.77',
    image: 'https://books.toscrape.com/media/cache/2c/da/2cdad67c44b002e7ead0cc35693c0e8b.jpg',
    inStock: 'In stock',
    link: 'catalogue/a-light-in-the-attic_1000/index.html'
  },
  ...
]

Best practices for web scraping

Follow these best practices to build reliable, efficient, and ethical web scrapers with Browserbase.

Ethical scraping

Respect robots.txt: Check the website’s robots.txt file for crawling guidelines
Rate limiting: Implement reasonable delays between requests (2-5 seconds)
Terms of Service: Review the website’s terms of service before scraping
Data usage: Only collect and use data in accordance with the website’s policies

Performance optimization

Batch processing: Process multiple pages in batches with concurrent sessions
Selective scraping: Only extract the data you need
Resource management: Close browser sessions promptly after use
Connection reuse: Reuse browsers for sequential scraping tasks

Protected Sites

Enable Browserbase Verified: Recognized by bot protection partners
Randomize behavior: Add variable delays between actions
Use proxies: Rotate IPs to distribute requests
Mimic human interaction: Add realistic mouse movements and delays
Handle CAPTCHAs: Enable Browserbase’s automatic CAPTCHA solving

Next steps

Verified

Configure fingerprinting and CAPTCHA solving

Browser Contexts

Persist cookies and session data

Proxies

Configure IP rotation and geolocation

Browserbase Functions

Deploy scraping workflows as serverless functions

Welcome

Platform

Use cases

Optimization

Account

Template

Company Value Prop Generator

Example: Scraping a book catalog

Code example

Example output

Best practices for web scraping

Ethical scraping

Performance optimization

Protected Sites

Next steps

Verified

Browser Contexts

Proxies

Browserbase Functions

Welcome

Platform

Use cases

Optimization

Account

​Template

Company Value Prop Generator

​Example: Scraping a book catalog

​Code example

​Example output

​Best practices for web scraping

​Ethical scraping

​Performance optimization

​Protected Sites

​Next steps

Verified

Browser Contexts

Proxies

Browserbase Functions

Template

Example: Scraping a book catalog

Code example

Example output

Best practices for web scraping

Ethical scraping

Performance optimization

Protected Sites

Next steps