Chronicle

The web scraping library that remembers.

Chronicle is a data-science-native web scraping library. It stores every run, profiles the result, and compares it to history — so silent changes in a page's structure or content are visible before they poison your pipeline.

⚠️ Alpha software. v0.3.0 is the current release on PyPI. It is stable enough for research and internal pipelines, but APIs may still shift before v1.0. See Status & Limitations before relying on it in production.

New in v0.3.0

Run-to-run diff, a four-step date confidence ladder, a JSON API connector, robots.txt and Retry-After support, and fixes for every numeric-handling bug in v0.2.1. Jump to what's new →

Why Chronicle?

• Reproducible: every run keeps raw HTML, parsed data, config, and a statistical profile
• Two comparisons: a row-by-row diff vs. the last run, and a statistical drift check vs. a baseline
• Honest data: ambiguous values are flagged with the original text, never guessed
• DataFrame-native: one line from URL to a pandas DataFrame
• Local-first: files plus SQLite, no server and no account

Quick Start

1. Install — see Installation for all options.

pip install chronicle-ds

2. Scrape a page into a DataFrame

from chronicle import Scrape

df = Scrape(
    "https://en.wikipedia.org/wiki/List_of_most-viewed_YouTube_videos"
).to_dataframe()

print(df.head())

Auto-detection supports HTML <table> elements only. For product grids, news lists, or SPAs, pass selectors={...} (see Usage Examples).

3. Run it again and see what changed

result = Scrape(url, selectors={...}, key="title").run()

if result.has_changes():
    print(result.changes.pretty())       # +3 added, -1 removed, ~5 changed
if result.has_drift():
    print(result.drift_report.pretty())  # SEVERE: price null % jumped

4. Use the command line

chronicle version
chronicle scrape <url>
chronicle runs

The CLI is still thin (no --selectors flag yet). Use the Python API for selector mode.

How It Works

 URL ──► Acquire ──► Refine ──► Remember ──► ScrapeResult
         fetch.py    parse.py    storage.py   .to_dataframe()
         playwright  clean.py    profile.py   .invalid_rows
         jsonapi.py  normalize   diff.py      .changes  .drift_report
                                 drift.py     .cleaning .errors

Five layers, each depending only on the ones below it: Foundation → Acquisition → Refinement → Memory → Orchestration → Interface. Full details in ARCHITECTURE.md.

Features

🌐 Acquisition

• HTTP fetcher: retries, exponential backoff, per-host rate limiting, Retry-After, robots.txt (RFC 9309), opt-in cache
• Headless Chromium via Playwright for JavaScript-rendered pages
• JSON connector for API endpoints (Shopify and World Bank shapes included)

🧹 Refinement

• Selector mode and auto table detection for HTML
• Date cleaning with a four-step confidence ladder; ambiguous dates become NaT with the original text preserved
• Schema validation with pydantic: coerces types, quarantines invalid rows instead of dropping them
• Fail-fast selector validation

🕐 Memory

• Versioned runs: raw HTML, parsed data, config, profile, diff, and cleaning report per run
• Run-to-run diff: keyed and keyless comparison with honest warnings
• Drift detection: zero-row, schema, null, mean, range, and selector-health checks
• Quality reports: row counts, success rate, per-column null %, dtypes, numeric stats
• URL redaction and microsecond-precision run IDs

📊 Developer experience

• DataFrame-native; built for Jupyter, pandas, and real pipelines
• 147 tests; CI on Ubuntu, Windows, and macOS across Python 3.10 to 3.12

Installation

Chronicle requires Python 3.10+.

Displaying Output

Chronicle returns DataFrames (which Jupyter renders as rich tables) and strings (which it does not). The correct display form depends on your environment and the type.

The rule

Bare expression when Jupyter has a rich display for the type. print() when it does not.

Jupyter and Colab render certain Python objects as formatted HTML (tables, key/value views). For everything else, they fall back to repr() — which shows escaped characters like \n literally. print() uses str() instead and renders newlines properly.

By return type

Concrete examples

Jupyter / Colab — DataFrames render as tables:

from chronicle import Scrape

result = Scrape("https://en.wikipedia.org/wiki/List_of_most-viewed_YouTube_videos").run()

# Bare expression → rich HTML table
result.to_dataframe().head(10)

Jupyter / Colab — strings need print():

# ❌ Bare expression — shows repr with literal \n
result.drift_report.pretty()

# ✅ print() — renders the formatted report
print(result.drift_report.pretty())

Drift needs two runs

.drift_report is None on the first run of a URL — there is no baseline to compare against yet. Guard with has_drift(), which is None-safe:

# First run — no baseline
Scrape(url).run()

# Second run — comparison available
result = Scrape(url).run()

if result.has_drift():
    print(result.drift_report.pretty())
else:
    print(f"No drift detected. Severity: {result.drift_report.severity}")

Same rule for .changes: use result.has_changes() before calling result.changes.pretty().

Colab persistence

Colab wipes its filesystem on every session restart. Point storage_root at Google Drive to keep run history across sessions:

from google.colab import drive
drive.mount("/content/drive")

result = Scrape(
    url,
    storage_root="/content/drive/MyDrive/chronicle",
).run()

Without this, every Colab session starts fresh — the first run in each session is always a "first run" with no baseline.

Usage Examples

from chronicle import Scrape

result = Scrape(
    url="http://books.toscrape.com/",
    selectors={
        "_container": "article.product_pod",
        "title": "h3 > a@title",
        "price": "p.price_color",
        "stock": "p.instock.availability",
        "link":  "h3 > a@href",
    },
    rate_limit=2.0,
).run()

df = result.to_dataframe()

result = Scrape(
    url="https://quotes.toscrape.com/js/",
    selectors={
        "_container": "div.quote",
        "text":   "span.text",
        "author": "small.author",
    },
    use_playwright=True,
    wait_for="div.quote",
).run()

import threading
out = {}
def run():
    out["r"] = Scrape(url, selectors={...}, use_playwright=True).run()
t = threading.Thread(target=run); t.start(); t.join()
df = out["r"].to_dataframe()

result = Scrape(
    url="https://api.example.com/products.json",
    records_path="products",          # omit if only one list exists
    fields={"name": "title", "price": "variants.0.price"},
).run()

result = Scrape(url, selectors={...}, key="title").run()

if result.has_changes():
    print(result.changes.pretty())
    result.changes.to_dataframe()   # one row per change

result = Scrape(url, selectors={...}, dayfirst=True).run()
print(result.cleaning.pretty())
result.cleaning.to_dataframe()   # every flagged value, original preserved

from pydantic import BaseModel
from chronicle import Scrape

class Video(BaseModel):
    title: str
    views: float | None = None

result = Scrape(
    url="https://en.wikipedia.org/wiki/List_of_most-viewed_YouTube_videos",
    schema=Video,
).run()

result.to_dataframe()
result.to_invalid_dataframe()   # quarantined rows

result.quality_report()
# rows_total, rows_valid, rows_invalid, success_rate,
# per-column dtype / null_pct / numeric stats

The Core Idea: Every Run Is an Artifact

Every Scrape(...).run() writes to .chronicle/:

.chronicle/projects/<project_id>/
├── baseline.json           # reference profile for drift
└── runs/<run_id>/
    ├── manifest.json       # config, timestamps, library version
    ├── profile.json        # statistical fingerprint of each column
    ├── diff.json           # what changed since the previous run
    ├── cleaning.json       # what was converted, flagged, or left unconverted
    ├── data.parquet        # the validated DataFrame
    ├── invalid.parquet     # quarantined rows (if any)
    └── raw/page_1.html     # exact HTML received

Two comparisons, two questions

Both run automatically on every Scrape.run() when a prior run exists. They are deliberately separate so you always know which one fired.

Drift checks implemented today

Politeness

robots.txt respected by default, Retry-After honoured on 429/503, per-host rate limiting. Caching is off by default because drift detection needs fresh data.

What's New

v0.3.0 (Current Release)

• 🔁 Run-to-run diff (result.changes), keyed and keyless
• 📅 Date cleaning with a four-step confidence ladder and result.cleaning
• 🔌 JSON connector (records_path, fields)
• 🤝 robots.txt and Retry-After support
• 🔢 Fixes for all numeric-handling bugs from v0.2.1
• 🧪 147 tests (84% coverage)

Full history in CHANGELOG.md.

Status & Limitations

Current limitations (v0.3.0)

• Storage is not fully transactional. A crash mid-run can leave the run directory and SQLite index inconsistent; no content hashes or orphan recovery
• No baseline management. The baseline is the first successful run until you delete it by hand
• Drift blind spots: same-mean distribution changes, categorical drift, dtype drift, duplicate-rate changes; near-zero-mean columns can false-alarm
• No SSRF protection: loopback, private, link-local, and cloud-metadata addresses are not blocked; redirects are not re-validated
• No response-size cap or parse-time limit
• No offline replay API, no async fetcher, thin CLI

Roadmap

• HTTP fetcher with retries, rate limits, robots.txt, Retry-After
• Playwright fetcher for JS-rendered pages
• JSON connector
• Date cleaning and cleaning report
• Schema validation with quarantine
• Versioned storage, profiles, drift detection
• Run-to-run diff
• v0.3.0 — numeric bug fixes, date cleaning, run-to-run diff, JSON connector, robots.txt support
• v0.3.x (planned) — drift detector rework (categorical, dtype, same-mean distribution), baseline management, SSRF protection, pagination strategies
• v0.4.0 (planned) — storage backend abstraction (local vs. cloud), async fetcher, RAD module
• v0.5.0 (not yet scheduled) — data contracts, self-healing selectors

Documentation

Contributing

Chronicle is in early, fast-moving development. Issues, ideas, and PRs are welcome. Found a bug? Open an issue with a minimal reproduction.

git clone git@github.com:TheDataNormad/chronicle.git
cd chronicle
python -m venv .venv
source .venv/Scripts/activate   # Windows Git Bash
# or: .\.venv\Scripts\Activate.ps1   # Windows PowerShell
pip install -e ".[dev]"
pytest

Before opening a PR, read ARCHITECTURE.md (layers import downward only) and CORE.md. Bug fixes should add a case to the matching end-to-end test.

License

MIT. See LICENSE.

Citation

@software{chronicle2026,
  author = {TheDataNormad},
  title  = {Chronicle: The web scraping library that remembers},
  year   = {2026},
  url    = {https://github.com/TheDataNormad/chronicle}
}

Contact

• GitHub: @TheDataNormad
• LinkedIn: linkedin.com/company/chronicle-ds
• PyPI: pypi.org/project/chronicle-ds

Happy scraping, and remember everything. 🕰️