yosoi 0.0.2a19

AI-Powered Selector Discovery - Discover once, scrape forever

[!WARNING] Yosoi is currently in Alpha. The API is expected to change significantly. We do not expect a stable API until we are out of Beta.

Yosoi - You Only Scrape Once (iteratively)

Discover once, scrape forever

Give Yosoi a URL, domain, or group of URLs, and it uses AI to automatically discover the best selectors for structured content.

Installation

# Install yosoi using uv
uv add yosoi

Browser Fetcher (JavaScript-heavy pages)

Yosoi supports a browser-based HTML fetcher powered by void_crawl — a Rust-native CDP client that renders JavaScript-heavy pages via Chrome DevTools Protocol.

Install void_crawl from the Void-Crawl repo (requires Rust ≥ 1.86, maturin ≥ 1.7, Chrome/Chromium), then use it via the create_fetcher API or the yosoi.vc convenience module:

# Fetcher interface
from yosoi.core.fetcher import create_fetcher

async def scrape():
    fetcher = create_fetcher("browser", no_sandbox=True)
    async with fetcher:
        result = await fetcher.fetch("https://example.com")
        print(result.html)

# vc convenience module — pool-based (recommended)
from yosoi import vc

async def scrape():
    async with vc.pool() as pool:
        async with await pool.acquire() as tab:
            await tab.navigate("https://example.com")
            html = await tab.content()

See examples/pool_usage.py and examples/pool_fetch_cleaned.py for full working examples.

Quick Start

API Key

Export your API Key or create a .env file

# Set keys for whichever providers you want to use
<PROVIDER_NAME>_KEY=your_api_key_here
GROQ_API_KEY=your_groq_key_here               # groq/...
GEMINI_API_KEY=your_gemini_api_key_here       # gemini/...
OPENAI_API_KEY=your_openai_api_key_here       # openai/...
CEREBRAS_API_KEY=your_cerebras_api_key_here   # cerebras/...
OPENROUTER_API_KEY=your_openrouter_key_here  # openrouter/...

See the full list of supported providers

Basic Usage

CLI Usage

# Specify model explicitly with -m provider:model-name
uv run yosoi -m groq:llama-3.3-70b-versatile --url https://qscrape.dev/l1/eshop/catalog/?cat=Forge%20%26%20Smithing --contract Product

You can then find your scraped content, selectors and logs in ./.yosoi relative to the directory you run the CLI command from.

Python Usage

We also have example scripts, you can find them in our example docs

Citation

If you use yosoi in your research or projects, please cite it using the metadata provided in the CITATION.cff file.

Contact

[e-mail]