ergane 0.6.0

High-performance async web scraper with selectolax parsing

Ergane

High-performance async web scraper with HTTP/2 support, built with Python.

Named after Ergane, Athena's title as goddess of crafts and weaving in Greek mythology.

Features

• Programmatic API — Crawler, crawl(), and stream() let you embed scraping in any Python application
• Hook System — Intercept requests and responses with the CrawlHook protocol
• HTTP/2 & Async — Fast concurrent connections with per-domain rate limiting and retry logic
• Fast Parsing — Selectolax HTML parsing (16x faster than BeautifulSoup)
• Built-in Presets — Pre-configured schemas for popular sites (no coding required)
• Custom Schemas — Define Pydantic models with CSS selectors and type coercion
• Multi-Format Output — Export to CSV, Excel, Parquet, JSON, JSONL, or SQLite
• Response Caching — SQLite-based caching for faster development and debugging
• Production Ready — robots.txt compliance, graceful shutdown, checkpoints, proxy support

Installation

pip install ergane

Quick Start

CLI — run from your terminal

# Use a built-in preset (no code needed)
ergane --preset quotes -o quotes.csv

# Crawl a custom URL
ergane -u https://example.com -n 100 -o data.parquet

# List available presets
ergane --list-presets

Python — embed in your application

import asyncio
from ergane import Crawler

async def main():
    async with Crawler(
        urls=["https://quotes.toscrape.com"],
        max_pages=20,
    ) as crawler:
        async for item in crawler.stream():
            print(item.url, item.title)

asyncio.run(main())

Python Library

Ergane's engine is a pure async library. The CLI is a thin wrapper around it — everything the CLI can do, your code can do too.

Crawler

The main entry point. Use it as an async context manager:

from ergane import Crawler

async with Crawler(
    urls=["https://example.com"],
    max_pages=50,
    concurrency=10,
    rate_limit=5.0,
) as crawler:
    results = await crawler.run()      # collect all items

Key parameters:

Parameter	Default	Description
urls	(required)	Seed URL(s) to start crawling
schema	None	Pydantic model for typed extraction
concurrency	10	Number of concurrent workers
max_pages	100	Maximum pages to crawl
max_depth	3	Maximum link-follow depth
rate_limit	10.0	Requests per second per domain
timeout	30.0	HTTP request timeout (seconds)
same_domain	True	Only follow links on the seed domain
hooks	None	List of CrawlHook instances
output	None	File path to write results
output_format	"auto"	csv, excel, parquet, json, jsonl, sqlite
cache	False	Enable SQLite response caching

run()

Executes the crawl and returns all extracted items as a list:

async with Crawler(urls=["https://example.com"], max_pages=10) as c:
    results = await c.run()
    print(f"Got {len(results)} items")

stream()

Yields items as they arrive — memory-efficient for large crawls:

async with Crawler(urls=["https://example.com"], max_pages=500) as c:
    async for item in c.stream():
        process(item)  # handle each item immediately

crawl()

One-shot convenience function — creates a Crawler, runs it, returns results:

from ergane import crawl

results = await crawl(
    urls=["https://example.com"],
    max_pages=10,
    concurrency=5,
)

Typed Extraction with Schemas

Pass a Pydantic model with CSS selectors to extract structured data:

from datetime import datetime
from pydantic import BaseModel
from ergane import Crawler, selector

class Quote(BaseModel):
    url: str
    crawled_at: datetime
    text: str = selector("span.text")
    author: str = selector("small.author")
    tags: list[str] = selector("div.tags a.tag")

async with Crawler(
    urls=["https://quotes.toscrape.com"],
    schema=Quote,
    max_pages=50,
) as crawler:
    for quote in await crawler.run():
        print(f"{quote.author}: {quote.text}")

The selector() helper supports:

Argument	Description
css	CSS selector string
attr	Extract an attribute instead of text (e.g. "href", "src")
coerce	Aggressive type coercion ("$19.99" → 19.99)
default	Default value if selector matches nothing

Hooks

Hooks let you intercept and modify requests before they're sent, and responses after they're received. They follow the CrawlHook protocol:

from ergane import CrawlHook, CrawlRequest, CrawlResponse

class CrawlHook(Protocol):
    async def on_request(self, request: CrawlRequest) -> CrawlRequest | None: ...
    async def on_response(self, response: CrawlResponse) -> CrawlResponse | None: ...

Return the (possibly modified) object to continue, or None to skip/discard.

BaseHook

Subclass BaseHook and override only the methods you need:

from ergane import BaseHook, CrawlRequest

class SkipAdminPages(BaseHook):
    async def on_request(self, request: CrawlRequest) -> CrawlRequest | None:
        if "/admin" in request.url:
            return None  # skip this URL
        return request

Built-in Hooks

Hook	Purpose
LoggingHook()	Logs requests and responses at DEBUG level
AuthHeaderHook(headers)	Injects custom headers (e.g. {"Authorization": "Bearer ..."})
StatusFilterHook(allowed)	Discards responses outside allowed status codes (default: {200})

Using Hooks

from ergane import Crawler
from ergane.crawler.hooks import LoggingHook, AuthHeaderHook

async with Crawler(
    urls=["https://api.example.com"],
    hooks=[
        AuthHeaderHook({"Authorization": "Bearer token123"}),
        LoggingHook(),
    ],
) as crawler:
    results = await crawler.run()

Hooks run in order: for requests, each hook receives the output of the previous one. The same applies for responses.

Built-in Presets

Preset	Site	Fields Extracted
hacker-news	news.ycombinator.com	title, link, score, author, comments
github-repos	github.com/search	name, description, stars, language, link
reddit	old.reddit.com	title, subreddit, score, author, comments, link
quotes	quotes.toscrape.com	quote, author, tags
amazon-products	amazon.com	title, price, rating, reviews, link
ebay-listings	ebay.com	title, price, condition, shipping, link
wikipedia-articles	en.wikipedia.org	title, link
bbc-news	bbc.com/news	title, summary, link

Custom Schemas

Define extraction rules in a YAML schema file:

# schema.yaml
name: ProductItem
fields:
  name:
    selector: "h1.product-title"
    type: str
  price:
    selector: "span.price"
    type: float
    coerce: true  # "$19.99" -> 19.99
  tags:
    selector: "span.tag"
    type: list[str]
  image_url:
    selector: "img.product"
    attr: src
    type: str

ergane -u https://example.com --schema schema.yaml -o products.parquet

Type coercion (coerce: true) handles common patterns: "$19.99" → 19.99, "1,234" → 1234, "yes" → True.

Supported types: str, int, float, bool, datetime, list[T].

You can also load YAML schemas programmatically:

from ergane import Crawler, load_schema_from_yaml

ProductItem = load_schema_from_yaml("schema.yaml")

async with Crawler(
    urls=["https://example.com"],
    schema=ProductItem,
) as crawler:
    results = await crawler.run()

Output Formats

Output format is auto-detected from file extension:

ergane --preset quotes -o quotes.csv      # CSV
ergane --preset quotes -o quotes.xlsx     # Excel
ergane --preset quotes -o quotes.parquet  # Parquet (default)
ergane --preset quotes -o quotes.json     # JSON array
ergane --preset quotes -o quotes.jsonl    # JSONL (one object per line)
ergane --preset quotes -o quotes.sqlite   # SQLite database

You can also force a format with --format/-f regardless of file extension:

ergane --preset quotes -f jsonl -o output.dat

import polars as pl
df = pl.read_parquet("output.parquet")

Architecture

Ergane separates the engine (pure async library) from the CLI (Rich progress bars, signal handling). Hooks plug into the pipeline at two points: after scheduling and after fetching.

         CLI (main.py)                          Python Library
    ┌──────────────────────┐             ┌──────────────────────────┐
    │  Click options        │             │  from ergane import ...   │
    │  Rich progress bar    │             │  Crawler / crawl()       │
    │  Signal handling      │             │  stream()                │
    │  Config file merge    │             │                          │
    └──────────┬───────────┘             └────────────┬─────────────┘
               │                                      │
               └──────────────┬───────────────────────┘
                              │
                              ▼
               ┌──────────────────────────────────┐
               │         Crawler  (engine)         │
               │    Pure async · no I/O concerns   │
               │    Spawns N worker coroutines     │
               └──────────────┬───────────────────┘
                              │
              ┌───────────────┼───────────────────┐
              │               │                   │
              ▼               ▼                   ▼
      ┌──────────────┐ ┌───────────┐   ┌──────────────┐
      │  Scheduler   │ │  Fetcher  │   │   Pipeline   │
      │  URL frontier│ │  HTTP/2   │   │  Batch write │
      │  dedup queue │ │  retries  │   │  multi-format│
      └──────┬───────┘ └─────┬─────┘   └──────────────┘
             │               │
             ▼               ▼
  ┌──────────────────────────────────────────────────┐
  │                Worker loop (× N)                  │
  │                                                   │
  │  1. Scheduler.get()   → CrawlRequest              │
  │  2. hooks.on_request  → modify / skip             │
  │  3. Fetcher.fetch()   → CrawlResponse             │
  │  4. hooks.on_response → modify / discard          │
  │  5. Parser.extract()  → Pydantic model            │
  │  6. Pipeline.add()    → buffered output           │
  │  7. extract_links()   → new URLs → Scheduler      │
  └──────────────────────────────────────────────────┘

  ┌──────────────────────────────────────────────────┐
  │               Cross-cutting concerns              │
  │                                                   │
  │  Cache ─── SQLite response cache with TTL         │
  │  Checkpoint ─ periodic JSON snapshots for resume  │
  │  Schema ── YAML → dynamic Pydantic model + coerce │
  └──────────────────────────────────────────────────┘

CLI Reference

Common Options

Option	Short	Default	Description
--url	-u	none	Start URL(s), can specify multiple
--output	-o	output.parquet	Output file path
--max-pages	-n	100	Maximum pages to crawl
--max-depth	-d	3	Maximum crawl depth
--concurrency	-c	10	Concurrent requests
--rate-limit	-r	10.0	Requests per second per domain
--schema	-s	none	YAML schema file for custom extraction
--preset	-p	none	Use a built-in preset
--format	-f	auto	Output format: csv, excel, parquet, json, jsonl, sqlite
--timeout	-t	30	Request timeout in seconds
--proxy	-x	none	HTTP/HTTPS proxy URL
--same-domain/--any-domain		--same-domain	Restrict crawling to seed domain
--ignore-robots		false	Ignore robots.txt
--cache		false	Enable response caching
--cache-dir		.ergane_cache	Cache directory
--cache-ttl		3600	Cache TTL in seconds
--resume			Resume from checkpoint
--checkpoint-interval		100	Save checkpoint every N pages
--log-level		INFO	DEBUG, INFO, WARNING, ERROR
--log-file		none	Write logs to file
--no-progress			Disable progress bar
--config	-C	none	Config file path

Run ergane --help for the full list.

Advanced CLI Examples

# Crawl with a proxy
ergane -u https://example.com -o data.csv --proxy http://localhost:8080

# Resume an interrupted crawl (requires prior checkpoint)
ergane -u https://example.com -n 500 --resume

# Save checkpoints every 50 pages with debug logging
ergane -u https://example.com -n 500 --checkpoint-interval 50 \
    --log-level DEBUG --log-file crawl.log

# Use a YAML config file and override concurrency from CLI
ergane -u https://example.com -C config.yaml -c 20

# Combine preset with custom URL and explicit format
ergane --preset hacker-news -u https://news.ycombinator.com/newest \
    -f csv -o newest.csv -n 200

Configuration

Ergane looks for a config file in these locations (first match wins):

1. Explicit path via --config/-C
2. ~/.ergane.yaml
3. ./.ergane.yaml
4. ./ergane.yaml

crawler:
  max_pages: 100
  max_depth: 3
  concurrency: 10
  rate_limit: 10.0

defaults:
  output_format: "csv"
  checkpoint_interval: 100

logging:
  level: "INFO"
  file: null

CLI flags override config file values.

Troubleshooting

Getting empty or partial output

• Check --max-depth: depth 0 means only the seed URL is crawled. Increase with -d 3 to follow links.
• Same-domain filtering: by default Ergane only follows links on the same domain as the seed URL. Use --any-domain to crawl cross-domain.
• Selector mismatch: if using a custom schema, verify your CSS selectors match the actual site HTML (sites change frequently).

Blocked by robots.txt

If a target site disallows your user-agent in robots.txt, Ergane will return 403 for those URLs. Options:

# Ignore robots.txt (use responsibly)
ergane -u https://example.com --ignore-robots -o data.csv

Rate limiting or 429 responses

Lower the request rate and concurrency:

ergane -u https://example.com -r 2 -c 3 -o data.csv

The built-in per-domain token-bucket rate limiter (-r) controls requests per second. Reducing concurrency (-c) also lowers overall load.

Timeouts and connection errors

Increase the request timeout and enable retries (3 retries is the default):

ergane -u https://slow-site.com -t 60 -o data.csv

Resuming after a crash

Ergane periodically saves checkpoints (default: every 100 pages). To resume:

ergane -u https://example.com -n 1000 --resume

The checkpoint file is automatically deleted after a successful crawl.

License

MIT