Lumberjack

Turn documents and flat records into retrieval-ready chunks with structure and provenance intact.

Lumberjack is a Python library and CLI for RAG preprocessing. It reads a document tree instead of cutting plain text, preserves headings and source metadata, respects tables and fenced code, and keeps every chunk within a token budget whenever its content can be split safely.

中文说明 · Documentation · PyPI

Why Lumberjack?

• Context survives splitting. Chunks separate heading metadata from body content, so retrieval can retain the section that introduced an answer.
• Structure stays honest. Markdown/HTML/DOCX become heading trees; CSV/TSV, JSONL, and logs remain ordered records rather than fabricated sections.
• Budgets are practical. The default splitter plans with fast incremental estimates, then records an authoritative final token count.
• Topology is your choice. Pack sibling sections, preserve subtrees, or process each section body independently.

Input formats: now and planned

Lumberjack currently supports the formats below. A “planned” entry is a design commitment, not an installation promise: do not pass it to format or rely on automatic detection until it is marked supported in a release.

Flat data is a first-class design case, not a disguised heading tree. CSV, TSV, JSONL, Parquet, and record-oriented JSON will use ordered record/row units with schema and field-path provenance. Heading-oriented policies such as sibling section packing are not meaningful for those inputs; their adapters must select row/record-aware packing, preserve complete protected rows when configured, and report logical locations such as row number, column name, JSON path, page, or sheet. This behavior is planned; it is not implemented by the current three parsers.

Inspect every pipeline stage

Today, Lumberjack.saw() returns SplitResult(document, chunks), and the public parser, splitter, and finalizer can be called individually when an integration needs an intermediate DocTree or ChunkDraft.

The planned pipeline trace API will make every built-in and optional parsing stage inspectable through one stable result: raw Document, extraction output (for example OCR/layout), normalized DocTree, ChunkDrafts, rendered text, normalized/transformed text, and final Chunks. Visual PDF parsers such as MinerU, Docling, PaddleOCR-VL, and dots.mocr will be optional integrations—not core dependencies—and will retain page, bounding-box, and parser provenance.

Install

pip install lumberjack-py

# Exact tokenizers, DOCX/XLSX, Tree-sitter code parsing, and the Web API
pip install "lumberjack-py[tokenizers,docx,spreadsheets,code-parsing,web]"

Requires Python 3.10 or newer.

Split your first document

from lumberjack import Lumberjack

result = Lumberjack(max_tokens=500).saw("""
# Deployment

Deploy the service through the approved release workflow.

## Rollback

Keep the previous image available until health checks pass.
""")

for chunk in result.chunks:
    print(chunk.own_heading, chunk.body, chunk.token_count)

result.document contains the parsed DocTree, title, metadata, and source provenance. Each Chunk includes its body, final token_count, heading context, and source line range when available. See the five-minute guide for file inputs, output handling, and optional dependencies.

Use the CLI

# Infer a supported format from the file extension.
lumber handbook.md --max-tokens 1200

# Preserve CSV rows as atomic records (also use this for TSV, JSONL, and logs).
lumber people.csv --splitter record --max-tokens 1200

# Emit JSON suitable for an ingestion job.
lumber report.docx --tokenizer tiktoken --splitter subtree > chunks.json

The CLI reference lists every option and its default.

Choose a splitter

Tokenizer choice and counting mode are independent. For example, tiktoken works with incremental-sibling. Read the splitter decision guide before tuning a production pipeline.

Web API

lumberjack-serve --reload

With the web extra installed, the service exposes POST /lumber/api/split/text for UTF-8 Markdown, HTML, TXT, LOG, CSV/TSV, JSON/JSONL, XML, and YAML requests, and POST /lumber/api/split/file for those formats plus DOCX and XLSX (install spreadsheets for XLSX). Select splitter: "record" for LOG, CSV/TSV/XLSX, JSON/JSONL, XML, and YAML. The remaining planned formats above are not accepted yet. FastAPI serves interactive OpenAPI documentation at /docs while the server is running.

Learn more

• Core concepts — Document to Chunk, rendering, budgets, and source metadata.
• Configuration guide — block policies and budget controls.
• Custom components — parsers, tokenizers, splitters, and post-processing stages.
• Python API reference
• Contributing · Security · License

Benchmark reporting is under construction; this project does not make unverified quality or performance claims.