Crawl4AI

Crawl4AI turns the web into clean, LLM-ready Markdown for RAG, agents, and data pipelines. Combine it with Driver hosted sessions to crawl from managed Chrome infrastructure.

Installation

# Create environment with uv (Python >= 3.11)
uv init
uv add crawl4ai requests
uv sync

Setup

Get your API key from the Driver dashboard:

# .env
DRIVER_API_KEY=your-driver-key

Quick Start

Python

import os
import asyncio
import requests
from crawl4ai import AsyncWebCrawler, BrowserConfig


async def main():
    # Create Driver session
    resp = requests.post(
        "https://api.driver.dev/v1/browser/session",
        headers={
            "Authorization": f"Bearer {os.getenv('DRIVER_API_KEY')}",
            "Content-Type": "application/json",
        },
        json={"type": "hosted"},
    )
    resp.raise_for_status()
    data = resp.json()

    # Configure Crawl4AI to use the Driver browser
    browser_config = BrowserConfig(
        cdp_url=data["cdpUrl"],
        use_managed_browser=True,
    )

    # Run the crawler
    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(
            url="https://www.nbcnews.com/business",
        )
        print(result.markdown)

    # Cleanup
    requests.delete(
        "https://api.driver.dev/v1/browser/session",
        headers={"Authorization": f"Bearer {os.getenv('DRIVER_API_KEY')}"},
        params={"sessionId": data["sessionId"]},
    )


if __name__ == "__main__":
    asyncio.run(main())

uv run main.py

With Session Options

Python

resp = requests.post(
    "https://api.driver.dev/v1/browser/session",
    headers={
        "Authorization": f"Bearer {os.getenv('DRIVER_API_KEY')}",
        "Content-Type": "application/json",
    },
    json={
        "country": "US",
        "type": "hosted",
        "windowSize": "1920x1080",
    },
)

With Persistent Profiles

Keep cookies and session data across crawl runs:

Python

import os
import requests

headers = {
    "Authorization": f"Bearer {os.getenv('DRIVER_API_KEY')}",
    "Content-Type": "application/json",
}

# First run — establish session
resp = requests.post(
    "https://api.driver.dev/v1/browser/session",
    headers=headers,
    json={
        "type": "hosted",
        "profile": {"name": "my-crawler-profile", "persist": True}
    },
)

# Crawler runs and stores cookies...

# Later runs — cookies and data preserved
resp = requests.post(
    "https://api.driver.dev/v1/browser/session",
    headers=headers,
    json={
        "type": "hosted",
        "profile": {"name": "my-crawler-profile", "persist": True}
    },
)

Why Driver + Crawl4AI?

Hosted browsers — Crawl with managed Chrome sessions
LLM-ready output — Clean Markdown for RAG and AI pipelines
Global reach — Crawl from different countries
Persistent state — Maintain sessions across crawl runs
Scalability — No local browser management

Start

Browser Sessions

Session Options

Integrations

Installation

Setup

Quick Start

With Session Options

With Persistent Profiles

Why Driver + Crawl4AI?

​Installation

​Setup

​Quick Start

​With Session Options

​With Persistent Profiles

​Why Driver + Crawl4AI?

Installation

Setup

Quick Start

With Session Options

With Persistent Profiles

Why Driver + Crawl4AI?