Onecrawler

Crawling hard? Not anymore.

Async Python crawling library with sitemap discovery, browser-based extraction, and LLM-powered structured output — one Settings object drives it all.

Installation · Quick Start · CLI · Documentation

Overview

Onecrawler helps you build maintainable crawling and extraction workflows without turning every project into a custom scraping script. It provides a shared configuration model, async execution, sitemap discovery, browser-backed link extraction, heuristic content extraction, and optional GenAI extraction for typed outputs.

Recommended workflow:

1. Use sitemaps first whenever possible.
2. Fall back to browser link extraction when sitemap coverage is missing or dynamic.
3. Scrape the final URL list with heuristic extraction by default.
4. Use GenAI extraction when you need structured output in a Pydantic schema.

async with LinkExtractor(settings) as link_engine:
    links = await link_engine.run("https://example.com")

async with Scraper(settings) as scraper_engine:
    records = await scraper_engine.run(links)  # [{"url": ..., "result": ...}, ...]

Features

When To Use What

Installation

pip install onecrawler

Install Playwright browser binaries when you use browser-backed crawling or scraping:

python -m playwright install chromium

For local development:

git clone https://github.com/sayedshaun/onecrawler.git
cd onecrawler
python -m pip install -e ".[dev]"
python -m playwright install chromium

Docker Support

OneCrawler provides an optimized Docker image that includes all necessary browser dependencies. This is the recommended way to run onecrawler in production or CI/CD environments.

Build the Image

docker pull ghcr.io/sayedshaun/onecrawler:latest

You can rename it for convenience

docker tag ghcr.io/sayedshaun/onecrawler:latest onecrawler

Run a Script with Docker

docker run -it --rm -v $(pwd):/app onecrawler python your_script.py

Run the Built-in CLI with Docker

docker run --rm onecrawler python -m onecrawler crawl https://example.com --limit 20

Command-Line Interface

For a quick one-off crawl without writing a script:

python -m onecrawler crawl https://example.com --limit 20 --concurrency 5

Each extracted page is printed to stdout as a JSON line. Run python -m onecrawler crawl --help for the full flag list (--strategy, --format, --include, --exclude). The CLI covers heuristic-strategy crawling only — proxies, GenAI extraction, and human-behavior simulation need the Python API below.

Quick Start

from onecrawler import Settings, Crawler
from onecrawler.utils import writter

async def main():
    settings = Settings(link_extraction_limit=100, concurrency=5)

    async with Crawler(settings) as engine:
        results = await engine.run("https://www.prothomalo.com/")

    writter.dump_json(results, "output.json")


if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

Separate Workflow

from onecrawler import HumanBehaviorSettings, LinkExtractor, Scraper, Settings
from onecrawler.utils import writter

async def main():
    settings = Settings(
        link_extraction_strategy="deep",
        link_extraction_limit=10,
        concurrency=7,
        scraping_strategy="heuristic",
        scraping_output_format="json",
        human_behavior_settings=HumanBehaviorSettings(),
    )

    async with LinkExtractor(settings) as link_engine:
        links = await link_engine.run("https://www.example.com/")

    async with Scraper(settings) as scraper_engine:
        results = await scraper_engine.run(links)  # [{"url": ..., "result": ...}, ...]

    writter.dump_json(results, "output.json")


if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

Browser Link Extraction

Use browser extraction when sitemaps are incomplete, unavailable, or unable to expose JavaScript-rendered links.

import asyncio
from onecrawler import Settings, LinkExtractor


async def main():
    settings = Settings(
        link_extraction_strategy="deep",
        link_extraction_limit=250,
        include_link_patterns=["/news/*"],
        concurrency=5,
    )

    async with LinkExtractor(settings) as engine:
        links = await engine.run("https://example.com/news")

    print(f"Collected {len(links)} links")


if __name__ == "__main__":
    asyncio.run(main())

Content Filtering

Filter crawled results by date, keywords, file type, or semantic similarity. Filters are passed to Crawler.run() or Crawler.stream() and applied after content extraction.

import asyncio
from onecrawler import Crawler, Settings
from onecrawler.filters import AND, by_date, by_keywords


async def main():
    settings = Settings(
        link_extraction_limit=50,
        concurrency=5,
    )

    # Keep only pages from 2025 that mention "python" or "async"
    content_filter = AND(
        by_date(start="2025-01-01", end="2025-12-31"),
        by_keywords(["python", "async"]),
    )

    async with Crawler(settings) as engine:
        results = await engine.run(
            "https://example.com/blog",
            filters=content_filter,
        )

    print(f"Matched {len(results)} pages")


if __name__ == "__main__":
    asyncio.run(main())

Available Filters

Composing Filters

Use AND, OR, and NOT from onecrawler.filters to combine filters:

from onecrawler.filters import AND, OR, NOT, by_date, by_keywords, by_files

# Pages from 2025 that mention "python" but are not PDFs
f = AND(
    by_date(start="2025-01-01"),
    by_keywords(["python"]),
    NOT(by_files(["pdf"])),
)

# Pages that mention "AI" or are from 2025
f = OR(
    by_keywords(["AI"]),
    by_date(start="2025-01-01", end="2025-12-31"),
)

Streaming With Filters

Filters work with Crawler.stream() for real-time filtered output:

async with Crawler(settings) as engine:
    async for item in engine.stream(
        "https://example.com/news",
        filters=by_cosine_similarity("climate policy", threshold=0.3),
    ):
        print(item["title"])

GenAI Extraction With a Schema

Use GenAI extraction when you need a strongly typed response shape instead of plain content.

import asyncio
from typing import Optional
from pydantic import BaseModel
from onecrawler import Settings, LLMSettings, Scraper


class ArticleSummary(BaseModel):
    title: str
    author: Optional[str] = None
    published_at: Optional[str] = None
    summary: str
    topics: list[str]


async def main():
    settings = Settings(
        scraping_strategy="genai",
        scraping_output_format="json",
        genai=LLMSettings(
            provider="openai",
            model_name="gpt-4o-mini",
            api_key="YOUR_API_KEY",
            output_schema=ArticleSummary,
        ),
        concurrency=2,
        request_timeout=30,
    )

    async with Scraper(settings) as scraper:
        item = await scraper.run("https://example.com/articles/story")

    result = item["result"]
    print(result.model_dump() if hasattr(result, "model_dump") else result)


if __name__ == "__main__":
    asyncio.run(main())

Supported Providers

Ollama Example

settings = Settings(
    scraping_strategy="genai",
    genai=LLMSettings(
        provider="ollama",
        model_name="llama3:8b",
        base_url="http://localhost:11434/",
        output_schema=ArticleSummary,
    ),
)

Proxy Support

Attach one proxy or a rotating proxy pool directly to Settings.

from onecrawler import Settings, ProxySettings


settings = Settings(
    proxies=[
        ProxySettings(server="http://proxy-1.example:8080"),
        ProxySettings(
            server="http://proxy-2.example:8080",
            username="user",
            password="pass",
        ),
    ],
    proxy_rotation_method="round_robin",
)

Use proxies=[ProxySettings(...)] for a single proxy (a one-element list), or add more entries with proxy_rotation_method for a pool.

Production Tips

License

Released under the MIT License. See LICENSE for full terms.