ergane 0.3.0

High-performance async web scraper with selectolax parsing

Ergane

High-performance async web scraper with HTTP/2 support, built with Python.

Named after Ergane, Athena's title as goddess of crafts and weaving in Greek mythology.

Features

• HTTP/2 Support - Fast concurrent connections via httpx
• Rate Limiting - Per-domain token bucket throttling
• Retry Logic - Exponential backoff (max 3 attempts)
• robots.txt Compliance - Respects crawler directives by default
• Fast HTML Parsing - Selectolax with CSS selector extraction (16x faster than BeautifulSoup)
• Smart Scheduling - Priority queue with URL deduplication
• Multi-Format Output - Export to CSV, Excel, or Parquet
• Built-in Presets - Pre-configured schemas for common sites (no coding required)
• Graceful Shutdown - Clean termination on SIGINT/SIGTERM
• Custom Schemas - Define Pydantic models with CSS selectors for type-safe extraction
• Native Types - Lists and nested objects stored as native Parquet types (not JSON strings)
• Type Coercion - Extract "$19.99" as float(19.99), "1,234" as int(1234)

Installation

pip install ergane

For development:

pip install ergane[dev]

Quick Start

Using Presets (Easiest)

# Use a preset - no schema needed!
ergane --preset quotes -o quotes.csv

# Export to Excel
ergane --preset hacker-news -o stories.xlsx

# List available presets
ergane --list-presets

Manual Crawling

# Crawl a single site
ergane -u https://example.com -n 100

# Crawl multiple start URLs
ergane -u https://site1.com -u https://site2.com -n 500

# Custom output and settings
ergane -u https://docs.python.org -n 50 -c 20 -r 5 -o python_docs.parquet

Built-in Presets

Presets provide pre-configured schemas for popular websites:

Preset	Site	Fields Extracted
hacker-news	news.ycombinator.com	title, link, score, author, comments
github-repos	github.com/search	name, description, stars, language, link
reddit	old.reddit.com	title, subreddit, score, author, comments, link
quotes	quotes.toscrape.com	quote, author, tags

# See all presets
ergane --list-presets

# Use preset with custom settings
ergane --preset hacker-news -n 100 -o hn.xlsx

CLI Options

Option	Short	Default	Description
--url	-u	none	Start URL(s), can specify multiple
--output	-o	output.parquet	Output file path
--max-pages	-n	100	Maximum pages to crawl
--max-depth	-d	3	Maximum crawl depth from start URLs
--concurrency	-c	10	Concurrent requests
--rate-limit	-r	10.0	Requests per second per domain
--timeout	-t	30.0	Request timeout in seconds
--same-domain		true	Stay on same domain as start URLs
--any-domain		false	Follow links to any domain
--ignore-robots		false	Ignore robots.txt restrictions
--schema	-s	none	YAML schema file for custom output fields
--format	-f	auto	Output format: auto, csv, excel, parquet
--preset	-p	none	Use a built-in preset
--list-presets			Show available presets and exit

Custom Schemas

Define your own output schema with CSS selectors for type-safe extraction:

Programmatic Usage

from pydantic import BaseModel
from datetime import datetime
from ergane.schema import selector

class ProductItem(BaseModel):
    url: str                    # Auto-populated from crawled URL
    crawled_at: datetime        # Auto-populated timestamp

    name: str = selector("h1.product-title")
    price: float = selector("span.price", coerce=True)  # "$19.99" -> 19.99
    tags: list[str] = selector("span.tag")              # Native list type
    image_url: str = selector("img.product", attr="src")
    in_stock: bool = selector("span.availability")

# Use with Crawler
from ergane import Crawler, CrawlConfig

config = CrawlConfig(output_schema=ProductItem)
crawler = Crawler(
    config=config,
    start_urls=["https://example.com/products"],
    output_path="products.parquet",
    max_pages=100,
    max_depth=2,
    same_domain=True,
)
await crawler.run()

YAML Schema (CLI)

Create a schema file schema.yaml:

name: ProductItem
fields:
  name:
    selector: "h1.product-title"
    type: str
  price:
    selector: "span.price"
    type: float
    coerce: true
  tags:
    selector: "span.tag"
    type: list[str]
  image_url:
    selector: "img.product"
    attr: src
    type: str

Then run:

ergane -u https://example.com --schema schema.yaml -o products.parquet

Type Coercion

The coerce=true option enables smart type conversion:

Input	Target Type	Result
"$19.99"	float	19.99
"1,234"	int	1234
"yes" / "true" / "1"	bool	True
"2024-01-15"	datetime	datetime(2024, 1, 15)

Supported Types

Python Type	Parquet Type	Example
str	Utf8	"Hello"
int	Int64	42
float	Float64	3.14
bool	Boolean	True
datetime	Datetime	datetime.now()
list[T]	List(T)	["a", "b"]
BaseModel	Struct	Nested object

Output Formats

Ergane supports multiple output formats, auto-detected from file extension:

Extension	Format	Best For
.csv	CSV	Universal compatibility, spreadsheets
.xlsx	Excel	Business users, non-technical stakeholders
.parquet	Parquet	Large datasets, data pipelines

# Auto-detect from extension
ergane --preset quotes -o quotes.csv
ergane --preset quotes -o quotes.xlsx
ergane --preset quotes -o quotes.parquet

# Or explicitly specify format
ergane --preset quotes -o data.out --format csv

Default Schema (without custom schema)

Column	Type	Description
url	string	Page URL
title	string	Page title
text	string	Extracted text content (max 10k chars)
links	string	JSON array of extracted links
extracted_data	string	JSON object of custom extractions
crawled_at	string	ISO timestamp

Reading Results

import polars as pl

# Read any format
df = pl.read_parquet("output.parquet")
df = pl.read_csv("output.csv")
df = pl.read_excel("output.xlsx")

print(df.head())

Benchmarks

Ergane uses selectolax for HTML parsing, which is significantly faster than BeautifulSoup:

Operation	Selectolax	BS4 + lxml	Speedup
Parse (small)	0.05ms	0.11ms	2.0x
Parse (large)	0.19ms	6.05ms	31.1x
Extract title	0.20ms	6.06ms	30.7x
Extract links	0.25ms	6.73ms	27.3x
Extract text	0.29ms	7.03ms	24.5x
CSS selector	0.20ms	7.25ms	35.7x

Average: 16x faster (1000 iterations, 34KB HTML)

Run the benchmark:

pip install beautifulsoup4 lxml
python benchmarks/parse_benchmark.py

License

MIT