openstax-llm

Pedagogical semantic chunking, RAG dataset preparation, and LLM fine-tuning pipelines from OpenStax textbooks.

This is the core library and CLI of the openstax-llm project. It transforms OpenStax college textbooks into structured, citation-aware, formula-safe datasets for vector search (RAG) and model fine-tuning.

If you want to drive this from an AI coding agent, see the sibling packages:

Install

uv add openstax-llm

CLI

# Search the OpenStax catalog
openstax-llm search physics

# Inspect chunk statistics for a textbook
openstax-llm info astronomy-2e

# Compile and chunk a textbook into a JSONL dataset
openstax-llm prepare astronomy-2e -o datasets/astronomy-2e.jsonl

Library

from openstax_llm import DocumentChunker, TextBookDataset

chunker = DocumentChunker(target_words=400, max_words=600, overlap_words=50)
dataset = TextBookDataset.from_textbook("calculus-volume-1", chunker=chunker)
dataset.to_jsonl("calculus.jsonl")

print(dataset.summary())

DocumentChunker also works on arbitrary markdown, which is useful for tests and for non-OpenStax sources:

from openstax_llm import DocumentChunker

chunks = DocumentChunker().chunk_markdown(
    "## 1.2 Functions\n\nAn inline formula $f(x)=x^2$ stays intact.\n",
    book_slug="my-notes",
    section="1.2",
    section_title="Functions",
)

Guarantees

1. Formula integrity — a chunk boundary is never placed inside an unclosed $...$ or $$...$$ block.
2. Pedagogical cohesion — worked examples, definitions, problem sets, and summaries are kept whole whenever they fit within max_words.
3. Provenance — every chunk carries book_slug, book_title, chapter, section, section_title, and chunk_type.

Chunk schema

Development

This package lives in the openstax-llm uv workspace. Run every check from the repository root:

uv sync
uv run pytest -v
uv run ruff check . && uv run ruff format --check .
uv run mypy

License

MIT