spagents

Give your AI agents eyes for the modern web.

Installation · Quick Start · MCP Server · Python SDK · How It Works

AI agents are blind to Single Page Applications. Tools like fetch and requests return empty HTML shells — no articles, no content, no interactive elements. SPAs render everything via JavaScript after the page loads.

spagents fixes this. It launches a real browser, intelligently waits for SPAs to finish rendering, and returns structured, agent-friendly data — articles, actions, inputs, navigation — ready for your agent to use.

How Claude handles dynamic content before spagents

How Claude handles dynamic content after spagents

Key features

• Smart content detection — Knows when a SPA is done rendering (not just sleep(5))
• Structured extraction — Returns articles, links, metadata as typed Pydantic models
• Full interaction — Click, type, scroll, press keys, navigate — like a real user
• Action discovery — Finds every interactive element: buttons, inputs, ARIA roles, custom components
• Session persistence — Cookies, localStorage, and auth state preserved across navigations
• Three interfaces — Python SDK, CLI, and MCP server for Claude Desktop / AI agents

Installation

pip install spagents
playwright install chromium

From source

git clone https://github.com/your-username/spagents.git
cd spagents
uv sync
uv run playwright install chromium

Quick start

CLI

# Structured JSON output (default)
spagents browse "https://news.kagi.com"

# Human-readable text
spagents browse "https://news.kagi.com" --format text

# Interactive REPL session
spagents interactive "https://news.kagi.com"

Python SDK

import asyncio
from spagents import BrowserManager

async def main():
    async with BrowserManager() as browser:
        session = await browser.new_session()

        # Browse a SPA — content is fully rendered
        state = await session.navigate("https://news.kagi.com")
        for article in state.content.articles:
            print(f"{article.category}: {article.headline}")

        # Interact with the page
        for action in state.actions:
            if "Technology" in action.description:
                state = await session.click(action.selector)
                break

        # Type into inputs, press keys
        state = await session.type_text("#search", "climate change")
        state = await session.press_key("Enter")

        await session.close()

asyncio.run(main())

MCP Server

Connect spagents to Claude Desktop or any MCP-compatible AI agent:

spagents mcp

Add to your Claude Desktop config (claude_desktop_config.json):

{
  "mcpServers": {
    "spagents": {
      "command": "spagents",
      "args": ["mcp"]
    }
  }
}

Claude Code

Add the MCP server to your project or user settings:

claude mcp add spagents -- spagents mcp

Or add it directly to your .claude/settings.json:

{
  "mcpServers": {
    "spagents": {
      "command": "spagents",
      "args": ["mcp"]
    }
  }
}

Now Claude can browse any SPA:

You: "What's the top story on Kagi News today?"

Claude: uses browse tool → reads structured articles → gives you the answer

MCP tools

Interactive REPL commands

click <n>                  Click action by number
click <selector>           Click by CSS selector
type <n> <text>            Type text into an input by action number
type "<selector>" <text>   Type text into an input by CSS selector
press <key>                Press a key (Enter, Escape, Tab, ArrowDown, etc.)
select <n> <value>         Select dropdown option by action number
scroll [down|up]           Scroll the page
actions                    List all interactive elements
extract                    Re-extract page content
navigate <url>             Navigate to a new URL
json                       Dump current state as JSON
quit                       Exit the session

How it works

spagents wraps Playwright and adds three intelligent layers:

1. Content ready detection

A multi-signal detector that knows when a SPA has actually finished rendering:

This replaces naive approaches like sleep(5) or Playwright's networkidle (which breaks on long-polling and WebSocket connections).

2. Content extraction

Extracts structured data from the rendered DOM:

• Articles with headlines, summaries, sources, highlights, quotes, and sections
• Links with surrounding context (which heading or section they're under)
• Metadata from OG tags, meta descriptions, and page title

3. Action discovery

Finds every interactive element on the page through four phases:

1. Semantic HTML — <a>, <button>, <input>, <select>
2. ARIA roles — role="button", role="tab", role="listitem", etc.
3. Custom components — tabindex, onclick, cursor: pointer
4. Disambiguation — Duplicate labels get context from parent containers

CLI reference

spagents browse <url> [OPTIONS]
  --format, -f    Output format: json (default) or text
  --timeout, -t   Content detection timeout in ms (default: 15000)
  --no-headless   Run browser with visible window

spagents interactive <url> [OPTIONS]
  --timeout, -t   Content detection timeout in ms (default: 15000)
  --no-headless   Run browser with visible window

spagents mcp [OPTIONS]
  --transport     Transport: stdio (default) or sse
  --port, -p      Port for SSE transport (default: 8000)

License

MIT with an amended community contribution requirement for organizations with more than 100 employees. See LICENSE.md for details.