textbook-rag

Ask questions about technical books you own. Get a prose answer with citations to the page it came from — and a check that the model did not make it up.

$ tb ask "how do I avoid callback hell"

Q: how do I avoid callback hell

The book identifies the problem as code whose nesting grows with each
asynchronous step until it becomes unreadable [1]. Three remedies are
offered: early return from error branches so the happy path stays flat [1],
naming and hoisting callbacks out of the nested position [2], and using the
sequential-execution pattern to run a known set of tasks in order [3].

------------------------------------------------------------------------------
Sources:
 *[1] nodejs-design-patterns · Callbacks and Events > The callback hell · p104
 *[2] nodejs-design-patterns · ... > Callback discipline · pp106-107
 *[3] nodejs-design-patterns · Asynchronous Control Flow > Sequential execution · p115
  [4] nodejs-design-patterns · ... > The problem with callbacks · p103

tokens: 3412 in / 198 out · top score 0.612

Why this is not another RAG wrapper

Two checks run on every answer, because a language model's confidence carries no information about its correctness.

1. Every [n] must refer to a passage the model was actually given. An invented [7] against six passages is a fabricated source.
2. Every line of code inside a fence must appear in those passages.

The second check exists because the first was not enough. Asked for a concurrency limiter, the model emitted a TaskQueue class and cited [3] — a real, valid passage number. No retrieved passage contained that class. Check 1 passed it. The number was valid and the content was invented.

For a programming book that is the dangerous failure, because a reader cannot tell fabricated code from quoted code by looking at it.

Treat every code block in an answer as unverified until the verifier clears it. Prose answers are reliable; generated code is not. The tool tells you which is which rather than pretending the distinction does not exist.

It also refuses. Ask about something the corpus does not cover and you get a decline and exit code 2, not a confident answer built on the nearest unrelated chunk.

$ tb ask "how do I train a convolutional neural network"
I can't answer that from this corpus. (best match scored 0.196, below the
0.25 floor; nearest section was "Running CPU-bound tasks")
$ echo $?
2

Install

pip install textbook-rag

That is the query path. To add books of your own you also need the parser stack, which pulls PyTorch and is about 2.5 GB:

pip install 'textbook-rag[ingest]'

Setup

You configure credentials by not configuring them. The tool never reads your access key; boto3 resolves credentials through its own chain, so aws configure, AWS_PROFILE, SSO and instance roles all work untouched.

aws configure          # or: aws sso login --profile you
tb doctor

doctor checks everything in dependency order and names the fix for whatever is broken:

$ tb doctor
✓ config           /Users/you/.config/textbook-rag/config.toml
✓ credentials      account 123456789012, region us-east-1
✓ bedrock embed    amazon.titan-embed-text-v2:0, 1024 dims
✓ bedrock generate us.amazon.nova-pro-v1:0
✗ books            none ingested (looked in ~/.local/share/textbook-rag)
                   → tb ingest path/to/book.pdf
! ingest extra     not installed — querying works, ingesting does not
                   → pip install 'textbook-rag[ingest]'

You need Bedrock model access enabled for the embedding, generation and rerank models in your region — that is a separate switch from IAM, in the Bedrock console under Model access. doctor tells you if it is missing.

Then ingest a book you own:

tb ingest ~/books/nodejs-design-patterns.pdf

No corpus ships with this package, and none can: book text is copyrighted. You point it at PDFs you have.

Commands

tb ask "..."	answer with citations and verification
tb search "..."	retrieve only — one embedding call, no generation, no cost
tb ingest book.pdf	parse, chunk and embed a book
tb books list	what is in the corpus
tb books info <slug>	pages, chunks, which embedding model built it
tb books remove <slug>	delete a book's index
tb config get/set/path	region and model ids
tb doctor	diagnose setup
tb eval --gold gold.json	score retrieval against a gold set

Every command takes --json. Exit codes: 0 answered, 1 failed, 2 refused.

How retrieval works

question ──► decompose ──► embed each part ──► rerank each against ITSELF ──► interleave
                                                                                  │
                                           refusal gate ◄── best cosine ──────────┤
                                                                                  ▼
                                                        parent sections ──► model ──► VERIFY

Four strategies, selectable with --strategy. Measured on 41 questions against one book:

	recall@3	recall@6	multi-hop	zero-leak recall
bi-encoder	0.722	0.889	8/11	—
reranked	0.778	0.944	9/11	—
per-subquery (default)	0.917	0.972	10/11	1.000

"Zero-leak recall" is the number that matters. Questions are split by whether they share vocabulary with the section that answers them. "What is the reactor pattern" against a section called Reactor pattern is answerable by keyword matching — BM25 passes it, and it proves nothing about semantic search. The zero-overlap questions are the real test: "the writer produces data faster than the reader can consume it" → Backpressure, sharing no content word with its answer. There is no lexical path to that.

A single aggregate recall number hides this split. A purely lexical system would score 10/10 on the leaky half, 0/10 on the rest, and still report ~0.55 — reading as mediocre rather than fundamentally broken.

Given any RAG evaluation with one recall number, the first question to ask is what fraction of its questions share vocabulary with their answers.

Ingestion, and why it runs two parsers

Every ML layout parser reflows code as prose. Marker, Docling and PyMuPDF4LLM all collapse a multi-line listing onto one line, and two of them corrupt it doing so — cb(null, filename, false) came out as cb(null filename false). That is a category behaviour, not a bug in one tool, so no choice of parser avoids it.

So two extractors run over the same pages:

Marker    headings, hierarchy, prose      9/9 sections, 0 false positives
PyMuPDF   anything in a monospace face    verbatim, never reflowed
          └── merged by locating Marker's collapsed rendering and swapping it back

Font is the discriminator because it is ground truth from the PDF, not a guess from indentation — indentation also matches block quotes, ASCII tables and poetry.

Chunking then follows three rules, in priority order: cut on headings; never split a code fence; give every chunk its section path, which is both what makes a chunk full of pronouns findable and what becomes the citation.

Configuration

Settings resolve in one place: CLI flag → environment → config file → default.

tb config set region eu-west-1
tb config set generate_model anthropic.claude-sonnet-4-5-20250929-v1:0

setting	default
region	us-east-1
generate_model	us.amazon.nova-pro-v1:0
embed_model	amazon.titan-embed-text-v2:0
rerank_model	cohere.rerank-v3-5:0
refusal_threshold	0.25
top_n	6
wide_k	25
rerank_depth	10

Those last four are measured, not chosen by taste. top_n = 6 because coverage plateaus there and k=10 costs 54% more tokens for no gain — and measurably degrades code fidelity, as more code passages make the model blend and reconstruct across them instead of quoting one. wide_k = 25 because recall@10 equals recall@6 (widening finds nothing new) while recall@25 is 1.000.

Changing embed_model is a migration, not a flag. An index is a card catalogue written in one model's language; vectors from a different model cannot be compared with existing ones even when the dimensions match. The model id is recorded in every index, so a mismatch is reported rather than silently returning nonsense:

! books   1 ingested, but these use a different embedding model than the
          current setting: nodejs-design-patterns (embedded with titan-v1)
          → tb ingest --reembed nodejs-design-patterns

Where things live

config	~/.config/textbook-rag/config.toml
corpus	~/.local/share/textbook-rag/books/<slug>/

Both follow each OS's convention. TEXTBOOK_RAG_HOME overrides the data root.

Requirements

Python 3.11+, an AWS account with Bedrock model access.

License

MIT