py-quarry 2.0.9

An intuitive web scraping toolkit with intelligent templates and interactive workflows

Quarry 🪨⛏️

A modern Python toolkit for web data extraction with robust support for React, Vue, and other JavaScript frameworks.

---

🌟 What is Quarry?

Quarry provides two powerful approaches for web scraping:

1. ⚒️ Quarry Tools - Interactive CLI pipeline for building extraction workflows
2. ⛏️ Miner Mode - Interactive pipeline assistant with Scout recommendations and templates

Key Feature: Resilient selectors that survive CSS framework updates (React CSS-in-JS, Vue scoped styles, etc.)

---

🚀 Quick Start

Installation

pip install py-quarry

Requirements: Python 3.11+

New to Quarry? Try the Guided Tutorial

quarry foreman

The Foreman walks you through the complete 5-tool pipeline in ~2-3 minutes using Hacker News as a demo.

Your First Extraction

# Analyze a webpage
quarry scout https://example.com

# Extract data
quarry excavate schema.yml --url https://example.com

# Export results
quarry ship output.jsonl results.csv

📖 Full guide: USAGE_GUIDE.md | INSTALLATION.md

---

⚒️ Quarry Tools

5 integrated tools for complete extraction pipelines:

Tool	Purpose	Example
📡 Scout	Analyze HTML & detect patterns	quarry scout <url>
📐 Survey	Design extraction schemas	quarry survey create schema.yml
🔨 Excavate	Execute data extraction	quarry excavate schema.yml --url <url>
✨ Polish	Transform & clean data	quarry polish data.jsonl --dedupe
📦 Ship	Export to CSV/JSON/SQLite/PostgreSQL	quarry ship data.jsonl output.csv

Complete pipeline:

quarry scout <url> | quarry excavate | quarry polish --dedupe | quarry ship results.csv

📚 Detailed docs: docs/QUARRY_COMPLETE.md

---

⛏️ Miner Mode

Guided extraction pipeline with intelligent recommendations:

# Launch interactive miner
quarry miner

# Miner will:
# 1. Analyze page with Scout
# 2. Recommend selectors or offer templates  
# 3. Extract data with pagination
# 4. Clean and deduplicate
# 5. Export to your format (CSV, PostgreSQL, etc.)

Choose your approach: Scout recommendations (fastest), templates (guided), or custom selectors (full control)

📚 Miner guide: USAGE_GUIDE.md

---

🎯 Modern Framework Support

The Problem: React/Vue sites use dynamic CSS classes (.css-17p10p8) that change with every build.

Quarry's Solution: Structural selectors that survive CSS changes:

# ❌ Brittle - breaks on rebuild
title: h3.css-17p10p8 a

# ✅ Resilient - structural hierarchy
title: h3 a

Tools for Modern Sites

1. Selector Audit Tool

python scripts/audit_schema_selectors.py my_schema.yml
# Detects brittle selectors, suggests fixes

2. Selector Utilities

from quarry.lib.selectors import build_robust_selector

robust = build_robust_selector('h3.css-xyz a', ['tag'])
# Returns: 'h3 a'

3. Framework Detection

from quarry.framework_profiles import detect_framework

framework = detect_framework(html, soup, url)
# Automatically detects React, Vue, WordPress, etc.

📚 Complete guide: docs/MODERN_FRAMEWORKS.md

---

📊 Features

• ✅ Framework Detection - Automatic detection of 9+ frameworks
• ✅ Resilient Selectors - Survive CSS framework updates
• ✅ Rate Limiting - Token bucket with exponential backoff
• ✅ Robots.txt - Automatic parsing and compliance
• ✅ State Management - SQLite-based deduplication
• ✅ Multiple Exports - CSV, JSON, SQLite, Parquet, PostgreSQL
• ✅ Validation - Schema validation with Pydantic
• ✅ Guided Tutorial - quarry foreman for interactive learning
• ✅ Testing - 217 tests, 100% passing

---

📁 Project Structure

quarry/
├── lib/                 # Core utilities
│   ├── selectors.py    # CSS selector utilities
│   ├── http.py         # HTTP client with rate limiting
│   └── robots.py       # Robots.txt parser
├── tools/              # Quarry suite
│   ├── scout/          # HTML analysis
│   ├── survey/         # Schema designer
│   ├── excavate/       # Extraction engine
│   ├── polish/         # Data transformation
│   └── ship/           # Data export
├── framework_profiles/ # Framework detection
├── connectors/         # Data source connectors
├── transforms/         # Data transformations
└── sinks/              # Output writers

---

📖 Documentation

• Installation - Setup and requirements
• Usage Guide - Complete usage documentation
• Modern Frameworks - React/Vue/Next.js guide
• Selector Reference - Quick selector patterns
• Wizard Guide - YAML-based extraction
• Architecture - System design
• Testing - Running tests
• Troubleshooting - Common issues

---

⚙️ Configuration

The CLI and HTTP client can be tuned via environment variables:

Logging & HTTP:

• QUARRY_LOG_LEVEL: Logging level (DEBUG, INFO, WARNING, ERROR). Default INFO.
• QUARRY_LOG_JSON: Set to 1 to emit JSON logs to stderr.
• QUARRY_OUTPUT_DIR: Base directory for schemas, extraction output, caches, and the Foreman tutorial. Defaults to the current working directory. When set, Quarry skips save-location prompts and writes all artifacts inside this directory.
• QUARRY_DEFAULT_RPS: Default requests-per-second per domain (float). Default 1.0.
• QUARRY_HTTP_TIMEOUT: Default request timeout in seconds (int). Default 30.
• QUARRY_HTTP_MAX_RETRIES: Default HTTP retries (int). Default 3.
• PROXY_URL: HTTP/HTTPS proxy URL (also honors standard HTTP(S)_PROXY).
• QUARRY_MAX_CONTENT_MB: Max response size in MB (int). Rejects larger payloads.
• QUARRY_INTERACTIVE: 1 to prompt when robots.txt blocks (ethical default is non-interactive).
• QUARRY_IGNORE_ROBOTS: 1 to ignore robots.txt (testing only).

Database Exports:

• QUARRY_POSTGRES_URL: PostgreSQL connection URL for database exports.

Examples:

export QUARRY_LOG_LEVEL=INFO
export QUARRY_DEFAULT_RPS=0.5
export QUARRY_HTTP_TIMEOUT=60
export QUARRY_HTTP_MAX_RETRIES=5
export QUARRY_MAX_CONTENT_MB=10
export PROXY_URL=http://proxy.internal:8080
export QUARRY_OUTPUT_DIR=/tmp/quarry_outputs
export QUARRY_POSTGRES_URL=postgresql://user:pass@localhost:5432/extractions

quarry excavate schemas/example.yml --url https://example.com -o out.jsonl
quarry ship out.jsonl postgresql://localhost/db  # Export to PostgreSQL

---

🧪 Development

# Run tests
pytest                          # All tests
pytest tests/test_scout.py -v   # Specific tool

# Code quality
ruff format .                   # Format code
ruff check .                    # Lint code

# Quick commands
make test                       # Run tests
make format                     # Format code
make check                      # Lint code

---

🤝 Contributing

Contributions welcome! See CONTRIBUTING.md for guidelines.

To add a new framework profile:

# quarry/framework_profiles/frameworks/my_framework.py
class MyFrameworkProfile(FrameworkProfile):
    name = "MyFramework"
    
    def detect(self, html: str, soup: BeautifulSoup, url: str) -> int:
        score = 0
        if 'framework-marker' in html:
            score += 50
        return score

---

📄 License

MIT License - see LICENSE

---

🙏 Acknowledgments

• Beautiful Soup - HTML parsing
• Pyarrow - Parquet support
• Click - CLI framework
• Rich - Terminal formatting
• Pydantic - Data validation

---

Happy Scraping! 🎉