codebookpy 0.1.0

LLM-powered codebook reader for exploratory data analysis

codebookpy

LLM-powered codebook reader for exploratory data analysis.

codebookpy lets you import a statistical codebook PDF into Python and look up variable descriptions, value codes, and data summaries alongside your DataFrame — all from a single function call.

---

Installation

pip install codebookpy

Install with your preferred LLM backend:

pip install "codebookpy[anthropic]"   # Claude (recommended)
pip install "codebookpy[openai]"      # OpenAI
pip install "codebookpy[all]"         # everything including OCR support

OCR support (for scanned/image-based PDFs)

If your codebook is a scanned PDF, install the OCR extras and Tesseract:

pip install "codebookpy[ocr]"

Then install the Tesseract engine:

• Windows: download from UB-Mannheim
• macOS: brew install tesseract
• Linux: sudo apt install tesseract-ocr

Update TESSERACT_CMD at the top of core.py to match your install path if needed (Windows only).

---

Quick start

import pandas as pd
from codebookpy import Codebook, cb_lookup

# Load your data
df = pd.read_csv("my_data.csv")

# Parse the codebook — pass df so the LLM anchors to your actual column names
cb = Codebook(
    "my_codebook.pdf",
    anthropic_api_key="sk-ant-...",   # or openai_api_key="sk-..."
    df=df,
)

# Look up variables — codebook info only
cb_lookup(cb, ["AGE", "SEX", "RACE"])

# Look up variables — with DataFrame summary
cb_lookup(cb, ["AGE", "SEX", "RACE"], df=df, summarize_df=True)

---

API reference

Codebook(pdf_path, *, anthropic_api_key=None, openai_api_key=None, df=None, known_vars=None, verbose=True)

Parses a codebook PDF using an LLM backend. Pass either anthropic_api_key or openai_api_key (not both).

Parameter	Type	Description
pdf_path	str | Path	Path to the codebook PDF
anthropic_api_key	str	Anthropic API key
openai_api_key	str	OpenAI API key
df	DataFrame	If provided, all column names are used to anchor the LLM during parsing
known_vars	list[str]	Explicit variable list (overrides df if both provided)
verbose	bool	Print parsing progress (default True)

cb_lookup(codebook, var_list, df=None, summarize_df=False)

Look up variables and print formatted summaries to the console.

Parameter	Type	Description
codebook	Codebook	A parsed Codebook object
var_list	list[str]	Variable names to look up
df	DataFrame	Optional DataFrame for data summaries
summarize_df	bool	If True and df provided, prints dtype, missingness, and numeric/frequency/date summaries

---

How it works

1. Text extraction — pdfplumber extracts text from the PDF. If the PDF is malformed, pikepdf repairs it first. If it's a scanned image-based PDF, OCR kicks in automatically via pymupdf + pytesseract.
2. Anchor-guided parsing — Your DataFrame column names are passed to the LLM so it searches for each variable by name rather than guessing. Variables are sent in batches of 10 with the most relevant codebook sections.
3. Rate limiting — Requests are throttled automatically to stay within API rate limits.
4. Lookup — cb_lookup() queries the in-memory parsed result and renders a formatted panel per variable using rich.

---

License

MIT