Lumberjack

Lumberjack is a document preprocessing library for retrieval-augmented generation (RAG). It converts documents into token-budgeted chunks, preserves document structure and source metadata, and splits tables by row with configurable header repetition.

Lumberjack can be used through the Python API, command-line interface, or Web API. Optional adapters support integration with LangChain, LlamaIndex, and Haystack.

中文说明 · Documentation · PyPI

Three ways to use Lumberjack

1. Python API

Pass text, bytes, a file path, or a Document to Lumberjack.saw(). The result contains the parsed document and finalized chunks.

from lumberjack import Lumberjack

result = Lumberjack(max_tokens=500).saw("""
# Deployment

Deploy through the approved release workflow.
""")

for chunk in result.chunks:
    print(chunk.body, chunk.token_count)

For multiple documents, saw_many() returns results as an iterator. See the Python API reference.

2. Command line

Pass a file, directory, or glob to lumber. The command-line interface emits JSON and supports batch output, with one result file per input document.

pip install lumberjack-py
lumber handbook.md --max-tokens 1200
lumber 'docs/**/*.md' --recursive --output-dir chunks/

Use --input-format to select a parser explicitly. For row- or record-oriented inputs such as CSV, TSV, JSONL, and logs, select --splitter record. See the CLI reference.

3. Web API

Install the web extra and start the service. Submit text or upload a file; the response contains the resulting chunks in JSON format.

pip install 'lumberjack-py[web]'
lumberjack-serve --host 127.0.0.1 --port 9612

The API provides POST /lumber/api/split/text and POST /lumber/api/split/file. Interactive request documentation is available at http://127.0.0.1:9612/docs while the service is running.

What it does well

• Broad format support. Markdown, HTML, DOCX, PDF, EPUB, PPTX/PPT/ODP, EML/MSG/MBOX, plain text and logs, CSV/TSV, JSON/JSONL, XML, YAML/TOML, XLSX, SQLite/SQL, source code, and Jupyter notebooks are parsed into a shared document model. Model-based tokenizers and enhanced code parsing require optional dependencies.
• Structure and provenance. Heading context is retained with each chunk. Record-oriented inputs preserve row, field, sheet, and source-location metadata rather than being represented as artificial sections.
• Row-aware table splitting. Markdown and HTML tables are split at row boundaries. Table headers are repeated in each generated chunk by default and can be configured by table type.
• Content-aware splitting. Code fences retain their delimiters when split. Lists and other block types have dedicated handling. Protected blocks that cannot be split safely may exceed the configured token budget.
• Configurable chunk organization. Select sibling-section packing, subtree-first grouping, per-section processing, or record-aware packing for flat inputs.
• Token-budget controls. The default approximate tokenizer requires no model download. Optional tiktoken and Transformers tokenizers provide model-based counts. Exact and incremental counting modes can be selected independently.

Supported input formats

RTF, ODT, legacy Excel, image, and columnar formats are not currently supported.

Table splitting

Tables are treated as structured blocks. When a table exceeds its block budget, Lumberjack groups complete rows into separate chunks and repeats the Markdown header and delimiter, or the HTML table header, by default. Set repeat_header=False to omit the header from subsequent chunks. A row that individually exceeds the budget is preserved intact and may cause its chunk to exceed that budget.

Python callers configure tables with typed options:

from lumberjack import Lumberjack
from lumberjack.block import MarkdownTableConfig
from lumberjack.splitter import SectionSplitter
from lumberjack.tokenizer import ApproxByteTokenizer

tokenizer = ApproxByteTokenizer()
splitter = SectionSplitter(
    tokenizer,
    max_tokens=500,
    block_options=[MarkdownTableConfig(repeat_header=True, max_tokens=300)],
)
result = Lumberjack(tokenizer=tokenizer, splitter=splitter).saw("report.md")

The CLI accepts equivalent settings, for example --block table:repeat-header=false,max-tokens=300. HTML tables use HTMLTableConfig in Python or html_table in CLI configuration.

Choosing a splitter

The exact-* variants recount candidate text during planning. The default and incremental-* variants use incremental counts during planning and report a final authoritative count. Tokenizer choice is independent of counting mode.

Install optional features

# PDF, presentation, and MSG parsing, model tokenizers, Web API, and Tree-sitter parsing
pip install 'lumberjack-py[pdf,presentations,msg,tokenizers,web,code-parsing]'

# Optional framework adapters
pip install 'lumberjack-py[langchain,llama-index,haystack]'

Framework adapters are available under lumberjack.integrations. They convert chunks to framework documents or nodes and provide native splitter or parser components. See the integration guides.

Learn more

• Core concepts — document parsing, chunking, rendering, and source metadata.
• Configuration guide — block policies and budget controls.
• Custom components — parsers, tokenizers, splitters, and post-processing.
• Contributing · Security · License