Chronicle

The web scraping library that remembers.

Chronicle is a data-science-native web scraping framework with temporal versioning and drift detection built in. Every scrape is stored, profiled, and compared to history — so you know exactly when your data changes and why.

Why Chronicle?

Existing scraping libraries hand you raw data and walk away. Chronicle remembers.

Chronicle sits on top of these tools and adds the layer data scientists actually need:

• 🕐 Temporal versioning — every scrape is stored, versioned, and replayable
• 🔍 Drift detection — catch broken selectors before they poison your pipeline
• 📊 DataFrame-native — one line from URL to analysis-ready data
• 📜 Reproducible — every run produces a self-contained artifact
• ✅ Schema validation — data quality enforced at scrape time
• 🌐 Two fetchers — fast HTTP for static pages, headless Chromium for JS-rendered SPAs
• 🧠 DS-first design — built for Jupyter, pandas, and real pipelines

Install

pip install chronicle-ds

Chronicle requires Python 3.10+.

For JS-rendered pages (Playwright):

pip install "chronicle-ds[browser]"
playwright install chromium

Quickstart

from chronicle import Scrape

# Works best on pages with a clear HTML <table>
df = Scrape(
    "https://en.wikipedia.org/wiki/List_of_most-viewed_YouTube_videos"
).to_dataframe()

print(df.head())

Note on auto-detection: The one-liner works on pages with clear HTML <table> elements. For anything else — product grids, news sites, SPAs, JSON APIs — pass selectors={...}. See "More Control" below.

More Control

Selector mode works on any HTML page, no matter how it's structured:

from chronicle import Scrape

result = Scrape(
    url="http://books.toscrape.com/",
    selectors={
        "_container": "article.product_pod",
        "title": "h3 > a@title",
        "price": "p.price_color",
        "stock": "p.instock.availability",
        "link":  "h3 > a@href",
    },
    pages=5,
    rate_limit=2.0,
).run()

df = result.to_dataframe()
print(df.head())

JS-rendered pages

Modern sites build pages client-side — a plain HTTP fetch gets you an empty shell. Chronicle ships a Playwright-based fetcher that launches headless Chromium, waits for the content to actually render, and returns the final HTML.

from chronicle import Scrape

result = Scrape(
    url="https://quotes.toscrape.com/js/",
    selectors={
        "_container": "div.quote",
        "text":   "span.text",
        "author": "small.author",
    },
    use_playwright=True,
    wait_for="div.quote",   # wait for this selector before parsing
).run()

Key parameters:

Known limitations: Some sites (YouTube, Twitter, LinkedIn) actively detect headless browsers and serve reduced content. For those, use their official APIs. Chronicle is honest about what it can and can't do.

The Core Idea

Chronicle treats every scrape as a versioned event. Every run writes an artifact to .chronicle/:

.chronicle/projects/<project_id>/
├── baseline.json           # reference profile for drift
└── runs/<run_id>/
    ├── manifest.json       # config, timestamps, library version
    ├── profile.json        # statistical fingerprint of each column
    ├── data.parquet        # the validated DataFrame
    ├── invalid.parquet     # quarantined rows (if any)
    └── raw/page_1.html     # exact HTML received

Drift Detection

On the second and subsequent runs, Chronicle compares the new profile to the baseline and flags:

• Schema drift — columns added, removed, or renamed
• Null drift — fields suddenly returning None
• Distribution drift — mean, median, or variance shifts
• Range drift — values outside historical bounds
• Selector health — a column that used to be full is now mostly empty

from chronicle import Scrape

result = Scrape("https://en.wikipedia.org/wiki/List_of_most-viewed_YouTube_videos").run()

if result.has_drift():
    print(result.drift_report.pretty())

Output:

======================================================================
DRIFT REPORT   severity: SEVERE
======================================================================

🔴 [SEVERE] Views (null)
    Null % jumped from 0.00% to 97.00% — selector likely broken
    baseline: 0.0  →  current: 0.97

Quality Reports

Every ScrapeResult carries a quality_report() — row counts, success rate, per-column null %, dtype, and stats for numeric columns.

result.quality_report()
# {
#   "rows_total": 100,
#   "rows_valid": 98,
#   "rows_invalid": 2,
#   "success_rate": 0.98,
#   "columns": {
#     "price": {"dtype": "float64", "null_pct": 0.02, "mean": 145.3, ...},
#     ...
#   }
# }

Schema Validation

Pass a pydantic model and Chronicle coerces types, validates each row, and quarantines the ones that fail — instead of silently dropping them.

from pydantic import BaseModel
from chronicle import Scrape

class Video(BaseModel):
    title: str
    views: float | None = None

result = Scrape(
    url="https://en.wikipedia.org/wiki/List_of_most-viewed_YouTube_videos",
    schema=Video,
).run()

print(result.to_dataframe().head())
print(result.to_invalid_dataframe().head())   # rows that failed

Status

🚧 Alpha — v0.2.0. Actively developed. APIs may shift before v1.0.

What's built

• HTTP layer (fetch.py) — retries, exponential backoff, rate limiting, file cache
• Playwright fetcher (playwright_fetcher.py) — headless Chromium for JS-rendered pages
• HTML parsing (parse.py) — selector mode + auto table detection
• Schema validation (normalize.py) — pydantic-based, quarantines invalid rows
• Scrape class (core.py) — one-line API, pagination, ScrapeResult
• Storage (storage.py) — versioned artifacts + SQLite run index
• Profile generation (profile.py) — statistical fingerprint of each run
• Drift detection (drift.py) — null / distribution / range / schema / selector
• CLI (cli.py) — version, scrape, runs commands
• 18 unit tests, CI across Ubuntu / Windows / macOS × Python 3.10 / 3.11 / 3.12
• Published to PyPI as chronicle-ds

Roadmap

• JSON API mode — first-class handling of sites that serve data via internal JSON APIs
• History API — Scrape.history(url) and drift_timeline(url) for time-series queries
• Data contracts — declare "good data" (completeness, ranges, uniqueness) and fail the run when violated
• Self-healing selectors — auto-detect broken selectors and suggest alternatives
• Playwright stealth — evasion for sites that block headless browsers

Contributing

Chronicle is in early, fast-moving development. Issues, ideas, and PRs are welcome.

git clone git@github.com:TheDataNormad/chronicle.git
cd chronicle
python -m venv .venv
source .venv/Scripts/activate   # Windows Git Bash
# or: .\.venv\Scripts\Activate.ps1   # Windows PowerShell
pip install -e ".[dev]"
pytest

License

MIT — see LICENSE.

Author

Built by @TheDataNormad.