hikugen 0.1.0

AI-powered web scraping library that generates Python code to extract structured data into Pydantic schemas

Hikugen

AI-powered web scraping library that generates Python code to extract structured data into Pydantic schemas.

Overview

Hikugen uses LLMs (via OpenRouter) to generate extraction code for any website. Given a URL, HTML content, and a Pydantic schema, Hikugen:

1. Generates extraction code via LLM
2. Validates code security (AST-based, whitelist imports)
3. Executes with timeout protection
4. Validates extracted data against your schema
5. Caches generated code for reuse

No manual parsing code neededjust define your schema, and Hikugen handles the rest.

Quick Start

Installation

# Clone the repository
git clone <repo-url>
cd hikugen

# Install dependencies
uv sync

Basic Usage

from hikugen import HikuExtractor
from pydantic import BaseModel

# Define your extraction schema
class Article(BaseModel):
    title: str
    author: str
    published_date: str
    content: str

# Initialize extractor
extractor = HikuExtractor(
    api_key="your-openrouter-api-key",
    model="google/gemini-2.5-flash"  # Optional, this is default
)

# Extract data from a URL
result = extractor.extract(
    url="https://example.com/article",
    schema=Article
)

print(result.title)
print(result.author)

Parameters

• url - URL to extract from
• schema - Pydantic BaseModel defining the extraction structure
• use_cached_code - Use cached extraction code if available (default: True)
• cookies_path - Path to cookies.txt for authenticated requests (optional)
• max_regenerate_attempts - Max regeneration attempts on failure (default: 1)
• validate_quality - Run LLM quality check on fresh code (default: True)

Key Features

• LLM-powered extraction - Automatically generates extraction code via OpenRouter LLMs
• Security validated - AST-based validation ensures no dangerous imports or code patterns
• Intelligent caching - Generated code cached by URL + schema hash for reuse
• Auto-regeneration - Automatically regenerates on extraction failures
• Quality validation - Optional LLM quality check for extracted data
• Timeout protected - 30-second execution timeout prevents infinite loops
• Type safe - Full Pydantic integration for schema validation

Development

Running Tests

# All tests
uv run pytest

# Specific test file
uv run pytest tests/test_code_generator.py

# With coverage
uv run pytest --cov=hikugen --cov-report=term-missing

Code Quality

# Lint and format
uv run ruff check src/hikugen/
uv run ruff format src/hikugen/

# Run all checks
uv run ruff check src/hikugen/ && uv run ruff format --check src/hiku/

Architecture

Layered Design

• Layer 1 (Foundation) - Database caching and HTTP client
• Layer 2 (Code Intelligence) - Prompts, validation, and generation
• Layer 3 (Main API) - HikuExtractor with auto-regeneration and quality validation

Workflow

URL + Schema
    �
Cache Check
    �
[Cache Hit] � Cached Code
[Cache Miss] � LLM generates code
    �
AST Validation
    �
Execute with Timeout
    �
Pydantic Validation
    �
LLM Quality Check (optional)
    �
Return Result or Regenerate

Constraints

Generated Code Requirements

Generated extraction functions must:

• Have exact signature: def extract_data(html_content):
• Return a dict (validated against schema afterward)
• Only import from allowed modules (stdlib + requests + bs4 + pydantic)
• NOT import lxml or system modules (os, subprocess, sys, etc.)

Execution

• 30-second timeout per extraction
• Runs in isolated namespace
• Returns data as dict (not BaseModel instance)

See Also

• CLAUDE.md - Development guide for contributors

License

See LICENSE file for details.