[!WARNING] Yosoi is currently in Alpha. The API is expected to change significantly. We do not expect a stable API until we are out of Beta.

Yosoi - You Only Scrape Once (iteratively)

Discover once, scrape forever

[!WARNING] Yosoi is research tooling for API design and web reverse engineering. You assume all legal risk for how you use it. Respect robots.txt, rate limits, and IP bans; and please don't bypass them with Tor or a VPN. Read DISCLAIMER.md before pointing it at anything.

Give Yosoi a URL, domain, or group of URLs, and it uses AI to automatically discover the best selectors for structured content.

Installation

# Install yosoi using uv
uv add yosoi

Browser Fetcher (JavaScript-heavy pages)

Yosoi uses VoidCrawl, its Rust-native Chrome DevTools Protocol backend, for rendered headless, headful, and waterfall acquisition. The pinned VoidCrawl wheel is installed with Yosoi; building from source is optional.

from yosoi.core.fetcher import create_fetcher


async def fetch_rendered():
    fetcher = create_fetcher('headless', no_sandbox=True)
    async with fetcher:
        result = await fetcher.fetch('https://example.com')
        print(result.html)

For direct VoidCrawl usage, use its current pool API:

from voidcrawl import BrowserPool, PoolConfig


async def fetch_directly():
    async with BrowserPool(PoolConfig()) as pool:
        async with pool.acquire() as tab:
            response = await tab.goto('https://example.com', capture_endpoints=True)
            print(response.html, response.endpoints)

See docs/voidcrawl.md, the official VoidCrawl documentation, and docs/fingerprinting-stack.md.

Typed JavaScript and handwritten browser flows

Use ys.Executor.js(...) for browser-computed contract values. Larger evaluators can be imported from confined local .js/.mjs module trees. When content must be revealed first, a typed ys.Flow class compiles directly into Yosoi's existing A3Node assess/act/expect replay model:

import yosoi as ys


class ItemsOpen(ys.State):
    condition = ys.css('[role="menu"]')


class OpenItems(ys.Flow):
    open_panel: ys.Expect[ItemsOpen] = ys.click(ys.role('tab', name='Items'))
    title: str = ys.Executor.js('document.title')


result = await OpenItems.run('https://example.com', fetcher_type='headless')

These APIs are experimental during alpha. See docs/executor-js-flow.md.

Deterministic extractor fields

Use fluent selector plans or ys.Extractor() callbacks for async, per-row scraper logic that consumes already-acquired evidence without an LLM:

import yosoi as ys


class Company(ys.Contract):
    # Without a root, the full page is one row. Collection plans naturally return [].
    name: str = ys.css('h1').text()
    links: list[str] = ys.css('a[href]').attr('href')


records = await ys.extract(html, Company, url='https://example.com/')

The annotation supplies cardinality and remains the model value type. @ys.extraction(field) binds custom logic without a naming convention; @ys.extractions(...) executes one callback for several fields. Declare decorated callbacks with @staticmethod so editors and Pyrefly recognize their row-only signature. Extractor fingerprints contain strategy/structure evidence, never extracted values. See docs/extractors.md and examples/extractor_fields.py.

Portable recipes

Recipes package a contract, verified selectors, optional A3Node browser actions, and validation evidence into deterministic JSON for review and replay:

uv run yosoi recipe mint --contract @Product --from-cache https://example.com/product/1 --out .yosoi/recipes/ --yes
uv run yosoi recipe validate .yosoi/recipes/product.recipe.json --url https://example.com/product/1 --write
uv run yosoi scrape https://example.com/product/2 --recipe .yosoi/recipes/product.recipe.json --recipe-id v1:sha256:...

Remote recipes are pin-required and trust-gated. See docs/recipes.md.

Agent workflows

Install Yosoi fetch/search/crawl/research skills into supported coding agents:

uvx yosoi agents install --target pi
uvx yosoi agents install --target agents

See docs/agent-workflows.md. For direct, bounded multi-URL page acquisition, see docs/fetch.md.

Read-only QA index

Existing observation indexes can be exposed through a bounded Python API or an injected MCP server. The standalone CLI and MCP launcher report their unwired state rather than starting capture or a provider:

uvx yosoi qa status --json
uvx yosoi agents install --target pi

See docs/qa-index.md for Python, MCP, CLI, skill, and security boundaries.

Quick Start

API Key

Export your API Key or create a .env file

# Set keys for whichever providers you want to use
<PROVIDER_NAME>_KEY=your_api_key_here
GROQ_API_KEY=your_groq_key_here               # groq/...
GEMINI_API_KEY=your_gemini_api_key_here       # gemini/...
OPENAI_API_KEY=your_openai_api_key_here       # openai/...
CEREBRAS_API_KEY=your_cerebras_api_key_here   # cerebras/...
OPENROUTER_API_KEY=your_openrouter_key_here  # openrouter/...

See the full list of supported providers

Basic Usage

CLI Usage

# Specify model explicitly with -m provider:model-name
uv run yosoi -m groq:llama-3.3-70b-versatile --url https://qscrape.dev/l1/eshop/catalog/?cat=Forge%20%26%20Smithing --contract Product

You can then find your scraped content, selectors and logs in ./.yosoi relative to the directory you run the CLI command from.

Python Usage

We also have example scripts, you can find them in our example docs

Citation

If you use yosoi in your research or projects, please cite it using the metadata provided in the CITATION.cff file.

Community

• Responsible use: see DISCLAIMER.md

Contact

contact@cascadinglabs.com