Chronicle

The web scraping library that remembers.

Chronicle is a data-science-native web scraping framework with temporal versioning and drift detection built in. Every scrape is stored, profiled, and compared to history — so you know exactly when your data changes and why.

Why Chronicle?

Existing scraping libraries hand you raw data and walk away. Chronicle remembers.

Chronicle sits on top of these tools and adds the layer data scientists actually need:

• 🕐 Temporal versioning — every scrape is stored, versioned, and replayable
• 🔍 Drift detection — catch broken selectors before they poison your pipeline
• 📊 DataFrame-native — one line from URL to analysis-ready data
• 📜 Reproducible — every run produces a self-contained artifact
• ✅ Schema validation — data quality enforced at scrape time
• 🧠 DS-first design — built for Jupyter, pandas, and real pipelines

Install

pip install chronicle-ds

Chronicle requires Python 3.10+.

Quickstart

from chronicle import Scrape

df = Scrape("https://example.com/products").to_dataframe()
print(df.head())

Note on auto-detection: The one-liner works best on pages with HTML <table> elements or simple repeated list structures. It can pick the wrong structure on pages with navigation chrome (news sites, SPAs). For those, pass selectors={...} — see "More Control" below.

More Control

from chronicle import Scrape

df = Scrape(
    url="https://example.com/products",
    selectors={
        "_container": "div.product",
        "name": "h2",
        "price": "span.price",
        "rating": "span.rating",
    },
    pages=5,
    rate_limit=2.0,
).to_dataframe()

The Core Idea

Chronicle treats every scrape as a versioned event. This unlocks two things no other scraping library provides:

1. Temporal Versioning

Every run is stored with:

• Raw HTML
• Parsed data (Parquet)
• Statistical profile
• Run metadata (timestamps, config, library version)

# Query the full history of a scraper
history = Scrape.history("https://example.com/products")
history.drift_timeline()

2. Drift Detection

Chronicle compares each new run against a baseline and detects:

• Schema drift — columns added, removed, or renamed
• Null drift — fields suddenly returning None
• Distribution drift — mean, median, or variance shifts
• Range drift — values outside historical bounds
• Selector health — selectors returning zero elements

result = Scrape("https://example.com/products").run()

if result.drift_detected:
    print(result.drift_report())
    # ⚠️ price null% jumped from 0.02 → 0.98 (selector likely broken)

Status

🚧 Early development. v0.1.0 in progress.

What's built

• HTTP layer (fetch.py) — retries, rate limiting, caching
• HTML parsing (parse.py) — selector + auto table/list detection
• Schema validation (normalize.py) — pydantic + quality reports
• Scrape class (core.py) — one-line API
• Storage + versioning (storage.py) — versioned artifacts + SQLite
• Profile generation (profile.py) — statistical fingerprints
• Drift detection (drift.py) — null / distribution / range / schema
• CLI (cli.py) — version, scrape, runs commands
• 18 passing unit tests
• Published to PyPI as chronicle-ds

Roadmap

• Data contracts (schema + quality guarantees)
• Reproducibility artifacts
• Self-healing selectors
• Playwright support for JS-rendered pages
• CLI + recipe system

Contributing

Chronicle is in early, fast-moving development. Issues, ideas, and PRs are welcome.

git clone git@github.com:TheDataNormad/chronicle.git
cd chronicle
python -m venv .venv
source .venv/Scripts/activate  # Windows
pip install -e ".[dev]"
pytest

License

MIT — see LICENSE.

Author

Built by @TheDataNormad.