omnidoc-rag 0.1.4

RAG pipeline for omnidoc-sdk — intent-aware chunking, evaluation, streaming, graph linking, and vector DB integrations

omnidoc-rag

Modular, file-type-aware RAG pipeline for the OmniDoc document intelligence ecosystem

---

What is omnidoc-rag?

omnidoc-rag is the companion RAG SDK for omnidoc-sdk. It converts Document objects from the extraction layer into vector-DB-ready SemanticChunk objects with a dedicated chunker per file type — each format gets its own splitting strategy, boundary detection, and intent mapping rather than a one-size-fits-all approach.

What makes it different:

• 41 file extensions — each routed to a purpose-built chunker (PDF, DOCX, PPTX, XLSX, MD, Python, JS, Java, CSV, JSON, XML, YAML, HTML, EPUB, EML, IPYNB, and more)
• 57 Excel chunk types — structural, semantic, analytical, validation, visual, VBA, cross-reference, connectivity, and operational chunks extracted from every layer of a workbook
• Intent classification — 6 canonical types (metric, table, process, value_proposition, heading, narrative) with per-intent token budgets
• Adaptive chunking — boundary strategy differs by type (slide per chunk for PPTX, function per chunk for Python, cell per chunk for notebooks)
• Deterministic confidence scoring — quality signal for retrieval ranking, no LLM required
• Streaming — true lazy generator for large documents
• Retrieval evaluation — query-term coverage, source diversity, verdict scoring
• Graph linking — NEXT / SAME_INTENT / METRIC_OF edge graph
• Cross-document stitching — merge equivalent sections across multiple documents
• 4 vector DB adapters — ChromaDB, Pinecone, Weaviate, PostgreSQL/pgvector

---

Table of Contents

1. Installation
2. Quick Start
3. Modular Chunking by File Type
   • Supported File Types
   • Auto-detection
   • Explicit chunker selection
   • Chunking strategies by format
4. Excel — 57 Chunk Types
5. Intent Types
6. Streaming
7. Confidence Scoring
8. Retrieval Evaluation
9. Graph Linking
10. Cross-Document Stitching
11. Vector DB Adapters
    • ChromaDB
    • Pinecone
    • Weaviate
    • PostgreSQL / pgvector
12. Schema Reference
13. Package Structure
14. Optional Extras
15. Contributing & Development
16. Changelog

---

Installation

Core (no vector DB)

pip install omnidoc-rag

With a specific vector DB

pip install "omnidoc-rag[chroma]"
pip install "omnidoc-rag[pinecone]"
pip install "omnidoc-rag[weaviate]"
pip install "omnidoc-rag[pgvector]"

Everything

pip install "omnidoc-rag[all]"

---

Quick Start

from omnidoc.loader.load import load_document      # omnidoc-sdk
from omnidocrag import chunk_document, evaluate_rag_result

# 1. Extract — any supported file type
doc = load_document("investor_deck.pdf")

# 2. Chunk — file type auto-detected, right chunker selected
chunks = chunk_document(doc)

for c in chunks:
    print(f"[{c.intent:<18}] conf={c.confidence:.2f}  p{c.page}  {c.text[:80]}")

# 3. Evaluate a retrieval result
result = evaluate_rag_result(
    query="What was the revenue growth rate?",
    answer="Revenue grew 24% year-over-year to $4.2B.",
    chunks=chunks,
)
print(result["overall"], result["verdict"])   # 0.87  excellent

---

Modular Chunking by File Type

Supported File Types

Every extension is routed to a dedicated chunker class. 41 extensions are registered:

Category	Extensions	Chunker	Strategy
Office — Documents	.pdf	PDFChunker	Heading-aware sections + per-row table chunks
	.docx .doc	DOCXChunker	Paragraph-boundary + batched table chunks
	.pptx .ppt	PPTXChunker	One chunk per slide + speaker notes
	.xlsx .xls .xlsm	XLSXChunker	57 chunk types (see below)
Text	.txt	TXTChunker	Paragraph-boundary, no heading detection
Markdown	.md	MDChunker	Heading hierarchy + fenced code blocks + MD tables
Code	.py	PythonChunker	def / class / async def boundaries
	.js .ts	JSChunker	function / arrow function / class / interface
	.java	JavaChunker	class / interface / enum + package block
Structured	.csv	CSVChunker	Header-aware row batches (30 rows/chunk)
	.json	JSONChunker	Top-level key / array-item batches
	.xml	XMLChunker	Top-level element chunks
	.yaml .yml	YAMLChunker	Top-level key blocks
Web	.html .htm	HTMLChunker	h1–h6 headings + <code> blocks + blockquotes
Images	.png .jpg .jpeg .tiff .webp .gif .bmp	ImageChunker	OCR text paragraphs (or metadata stub)
Archives	.zip .tar .gz .7z .rar .bz2 .xz	ArchiveChunker	Manifest chunk + contained-file delegation
E-Books	.epub	EPUBChunker	Chapter/spine-item-aware
	.odt	ODTChunker	Heading-aware paragraphs + table batches
	.rtf	RTFChunker	Paragraph-based, RTF artifact stripping
Email	.eml	EMLChunker	Header + body + quoted-reply + attachments
	.msg	MSGChunker	Outlook header + body + attachments + meeting detection
Notebooks	.ipynb	IPYNBChunker	Cell-aware — code/markdown/output per cell

Auto-detection

chunk_document reads doc.metadata["file"] and selects the right chunker automatically:

from omnidocrag import chunk_document

chunks = chunk_document(doc)                     # auto-detects from file path
chunks = chunk_document(doc, ext=".pdf")         # force a specific chunker

Explicit chunker selection

from omnidocrag.chunkers import get_chunker, chunk_by_type, supported_extensions

# Get an instantiated chunker by file path / extension
chunker = get_chunker("report.pdf")              # → PDFChunker()
chunker = get_chunker("notebook.ipynb")          # → IPYNBChunker()
chunker = get_chunker("data.xlsx")              # → XLSXChunker()

# Chunk directly
chunks = chunker.chunk(doc)
chunks = chunker.chunk(doc, overlap_chars=150, min_chars=30)

# Convenience dispatcher
chunks = chunk_by_type(doc)                      # same as chunk_document
chunks = chunk_by_type(doc, ext=".json")         # override extension

# Inspect all registered extensions
print(supported_extensions())
# ['.7z', '.bmp', '.bz2', '.csv', '.doc', '.docx', '.eml', '.epub', ...]

Chunking strategies by format

Documents — PDF / DOCX

# Headings flush the buffer and become standalone heading chunks
# Each table row → standalone metric chunk with header prepended
# Character overlap (default 100) carries context across boundaries
chunks = chunk_document(doc)          # PDFChunker or DOCXChunker

Presentations — PPTX

# Each slide = one chunk group (title as heading)
# Bullet content chunked within the slide's token budget
# Speaker notes → separate narrative chunk tagged content_type="speaker_notes"
for c in chunks:
    print(c.metadata.get("slide_number"), c.intent, c.text[:60])

Source Code — Python / JS / Java

# Python: def/class/async def blocks including decorators and docstrings
# JS/TS: function declarations, arrow functions, class/interface blocks
# Java: class/interface/enum top-level types + package + import preamble
# All code chunks carry: language, block_type metadata
for c in chunks:
    print(c.metadata["language"], c.metadata["block_type"], c.heading)

Notebooks — IPYNB

# Each cell → one chunk
# code cells → intent="process"
# markdown cells → intent classified from content
# outputs (stdout, display_data, errors) → separate chunks
for c in chunks:
    ct = c.metadata.get("content_type")
    if ct == "notebook_cell":
        print(f"Cell {c.metadata['cell_index']} [{c.metadata['cell_type']}]: {c.text[:60]}")
    elif ct == "cell_output":
        print(f"  Output: {c.text[:40]}")

Structured Data — CSV / JSON / XML / YAML

# CSV: header row prepended to every batch, metric intent if financial columns
# JSON: top-level keys become headings; arrays batched in groups of 20
# XML: root element summary + one chunk per top-level child element
# YAML: one chunk per top-level key block

Email — EML / MSG

# Chunk 1: header fields (From, To, Subject, Date, …)
# Chunk 2+: body paragraphs
# Quoted replies ("> ") → separate narrative sub-chunks
# Attachment list → process chunk
for c in chunks:
    print(c.metadata.get("email_part"), c.text[:60])
    # "header" | "body" | "quoted_reply" | "attachment_list" | "signature"

---

Excel — 57 Chunk Types

XLSXChunker uses openpyxl to extract every layer of a workbook into typed SemanticChunk objects. Each chunk carries a chunk_type field in metadata.

Structural (chunks 1–9)

#	Chunk Type	Key metadata fields
1	WorkbookChunk	name, sheet_names, author, created_at, has_macros, file_size_kb
2	SheetChunk	sheet_name, sheet_index, sheet_type, is_visible, used_range, tab_color, is_protected
3	TableChunk	table_name, address, style, has_total_row, col_count, row_count
4	SchemaChunk	columns[name, data_type, null_rate, unique_rate, sample_values], inferred_pk
5	RowChunk	row_index, values, parent_table, batch_start, batch_end, is_total_row
6	GroupChunk	direction, level, start_index, end_index, is_collapsed
7	ParentContextChunk	workbook, sheet, table, row_range (breadcrumb)
8	MergedCellChunk	merge_range, merged_value, row_span, col_span
9	FrozenPaneChunk	freeze_row, freeze_col, sheet_name

Semantic (chunks 10–15)

#	Chunk Type	Key metadata fields
10	SummaryChunk	row_count, col_count, numeric_cols, date_cols, null_rate, dupe_rate
11	SemanticNarrativeChunk	narrative, subject, time_scope, grain, generated_by
12	ColumnSemanticChunk	col_name, role (id/measure/dimension/date/flag), unit, is_pk, is_fk
13	HeaderAliasChunk	original_header, aliases, abbreviations, normalized_name
14	EntityChunk	entity_type (date/org/currency/product), entities, frequencies
15	CellAnnotationChunk	cell_address, annotation_type, text, author

Analytical (chunks 16–23)

#	Chunk Type	Key metadata fields
16	FormulaDefinitionChunk	formula_string, formula_type (scalar/array/dynamic/lambda), precedents
17	FormulaResultChunk	computed_value, value_type, has_error, error_type
18	KPIChunk	kpi_name, value, target, variance, variance_pct, source_cell
19	AggregationChunk	col_name, aggregations {SUM/AVG/COUNT/MIN/MAX/MEDIAN}, source_range
20	TemporalChunk	date_col, frequency, start_date, end_date, gap_count, is_sorted
21	TrendChunk	col_name, direction, delta_abs, delta_pct, regression_slope
22	OutlierChunk	col_name, outlier_type (zscore/iqr), cell_addr, value, z_score, severity
23	LookupMapChunk	lookup_type (VLOOKUP/INDEX-MATCH/XLOOKUP), formula, lookup_range

Validation / QA (chunks 24–28)

#	Chunk Type	Key metadata fields
24	ValidationChunk	range_addr, validation_type, allowed_values, formula, error_msg
25	ErrorChunk	cell_addr, error_type (#REF/#DIV0/…), formula, likely_cause
26	ConditionalFormatChunk	range_addr, rule_type, condition_formula, business_meaning
27	ProtectionChunk	scope, is_password_protected, locked_ranges
28	DataQualityChunk	col_name, blank_count, dupe_count, type_mismatch_count, flagged_cells

Visual / Embedded (chunks 29–41)

#	Chunk Type	Key metadata fields
29	ChartChunk	chart_type, title, x_axis, y_axis, series, sheet_anchor
30	ChartSeriesChunk	series_name, source_range, color, trendline_type
31	ChartAnnotationChunk	annotation_type, text, position
32	PivotTableChunk	pivot_name, row_fields, col_fields, value_fields, filter_fields
33	PivotFieldChunk	field_name, field_type, agg_function, sort_order
34	PivotCacheChunk	cache_range, last_refreshed, record_count
35	SlicerChunk	slicer_name, field_name, active_filters
36	TimelineChunk	timeline_name, date_field, granularity
37	SparklineChunk	sparkline_type, host_cell, source_range
38	ShapeChunk	shape_type, text_content, cell_anchor
39	ImageChunk	image_type, anchor_cell, width_px, height_px
40	FormControlChunk	control_type, linked_cell, value, macro_assigned
41	ActiveXControlChunk	control_type, name, linked_cell, event_macro

VBA / Macro (chunks 42–46)

#	Chunk Type	Key metadata fields
42	MacroChunk	macro_name, module_name, trigger_type, line_count, scope
43	VBAModuleChunk	module_name, module_type, procedure_names, line_count
44	VBAEventChunk	event_name, module, trigger_condition, affected_range
45	CustomFunctionChunk	function_name, parameters, return_type, is_udf
46	RibbonCustomizationChunk	tab_name, button_label, macro_assigned

Cross-Reference / Linkage (chunks 47–50)

#	Chunk Type	Key metadata fields
47	RelationshipChunk	source_cell, target_cell, target_sheet, rel_type, formula
48	NamedRangeChunk	range_name, refers_to, scope, usage_count
49	ExternalLinkChunk	source_cell, target_file, is_broken, update_mode
50	DependencyGraphChunk	nodes, edges, max_depth, has_circular_ref

Connectivity / Power Features (chunks 51–54)

#	Chunk Type	Key metadata fields
51	PowerQueryChunk	query_name, source_type, source_path, transformation_steps
52	DataModelChunk	tables, relationships, dax_measures
53	PowerPivotMeasureChunk	measure_name, dax_expression, source_table
54	ConnectionChunk	connection_name, connection_type, connection_string_sanitized

Operational / Metadata (chunks 55–57)

#	Chunk Type	Key metadata fields
55	PrintSettingsChunk	print_area, orientation, fit_to_pages, header_text, footer_text
56	ChangeLogChunk	cell_addr, old_value, new_value, author, changed_at
57	ChunkIndexChunk	all_chunk_ids, type_counts, source_address_map, created_at

Usage

from omnidocrag.chunkers import get_chunker

chunker = get_chunker("financials.xlsx")
chunks = chunker.chunk(doc)

# Filter by chunk type
kpi_chunks    = [c for c in chunks if c.metadata.get("chunk_type") == "KPIChunk"]
formula_chunks = [c for c in chunks if c.metadata.get("chunk_type") == "FormulaDefinitionChunk"]
pivot_chunks  = [c for c in chunks if c.metadata.get("chunk_type") == "PivotTableChunk"]

# Access structured metadata
for c in kpi_chunks:
    print(c.metadata["kpi_name"], c.metadata["value"], c.metadata.get("variance_pct"))

Note: XLSXChunker requires openpyxl and direct filesystem access to the .xlsx file via doc.metadata["file"]. If the file path is unavailable, it falls back to basic row-batch extraction using doc.tables.

---

Intent Types

Every chunk is labelled with one of six canonical intents. The intent drives chunk sizing and confidence scoring.

Intent	Token budget	Typical content
heading	60	Section/slide titles, worksheet names
metric	150	KPIs, financial figures, percentages, table rows
table	200	Structured data rows, CSV batches
value_proposition	250	Benefits, ROI claims, competitive statements
narrative	350	Prose, analysis, background paragraphs, email body
process	400	Code, workflows, steps, formulas, configuration

Classification is deterministic — no LLM call required:

from omnidocrag.core.intent import classify_intent
# or via top-level API:
from omnidocrag import classify_intent

classify_intent("Revenue grew 24% YoY to $4.2B")        # "metric"
classify_intent("Step 1: Configure the API key")         # "process"
classify_intent("This solution reduces costs by 30%")    # "value_proposition"
classify_intent("EXECUTIVE SUMMARY")                     # "heading"
classify_intent("The company was founded in 2012.")      # "narrative"
classify_intent("Col A | Col B\nVal 1 | Val 2")         # "table"

---

Streaming

stream_chunks is a true Python generator — emits one chunk at a time without building a full list in memory:

from omnidocrag.pipeline.stream import stream_chunks
# or:
from omnidocrag import stream_chunks

for chunk in stream_chunks(doc, overlap_chars=100):
    # process or upsert each chunk immediately
    vector_db.upsert([chunk])

Ideal for large documents or live vector DB indexing pipelines.

---

Confidence Scoring

from omnidocrag.core.confidence import score_chunk
# or:
from omnidocrag import score_chunk

score_chunk("Revenue grew 24% YoY to $4.2B in fiscal 2024.")  # ≥ 0.8
score_chunk("ROI: 38%", intent="metric")                       # ≥ 0.7 (not penalised for length)
score_chunk("See below")                                        # < 0.7
score_chunk("")                                                  # 0.1 (floor)

Scoring factors: length, multi-line structure, dense-fact density (financials/percentages/currency). Short-text penalty is not applied to metric, table, or value_proposition intents.

---

Retrieval Evaluation

Score a RAG result without an LLM — fully deterministic and local:

from omnidocrag.pipeline.evaluation import evaluate_rag_result
# or:
from omnidocrag import evaluate_rag_result

result = evaluate_rag_result(
    query="What was the EBITDA margin in fiscal 2024?",
    answer="EBITDA margin reached 28% driven by cost efficiencies.",
    chunks=chunks,
)

Return value

{
    "overall":          0.84,        # composite score 0.0–1.0
    "coverage":         0.75,        # fraction of query terms found in chunks
    "confidence":       0.91,        # average chunk confidence
    "source_diversity": 3,           # unique pages used
    "chunks_used":      4,
    "verdict":          "good",      # "excellent" | "good" | "weak" | "unsafe"
    "missing_terms":    ["ebitda"],
}

Verdict	Score
excellent	≥ 0.85
good	≥ 0.70
weak	≥ 0.50
unsafe	< 0.50 or no chunks

---

Graph Linking

from omnidocrag.pipeline.graph import link_chunks
# or:
from omnidocrag import link_chunks

graph = link_chunks(chunks, source="investor_deck.pdf")
graph["nodes"]   # [{id, intent, confidence, page, heading, text_preview}, ...]
graph["edges"]   # [{from, to, relation, weight}, ...]

Relation	Description
NEXT	Sequential order — every adjacent pair
SAME_INTENT	Non-adjacent chunks sharing the same intent
METRIC_OF	Metric chunk linked back to its nearest heading

---

Cross-Document Stitching

Merge semantically equivalent sections from multiple documents into a unified de-duplicated set:

from omnidocrag.pipeline.stitcher import stitch_documents
# or:
from omnidocrag import stitch_documents

docs = [
    {"metadata": {"file": "q3_report.pdf"}, "chunks": [c.to_dict() for c in chunks_q3]},
    {"metadata": {"file": "q4_report.pdf"}, "chunks": [c.to_dict() for c in chunks_q4]},
]

merged = stitch_documents(docs, similarity_threshold=0.80)

for chunk in merged:
    if len(chunk["sources"]) > 1:
        print(f"Merged from: {chunk['sources']}  {chunk['text'][:80]}")

Two chunks merge when their heading + intent match at SequenceMatcher similarity ≥ similarity_threshold. The merged chunk's sources lists all contributing files.

---

Vector DB Adapters

All four adapters share the same upsert(chunks) / query(text) interface and accept either SemanticChunk objects or plain dicts.

---

ChromaDB

from omnidocrag.vectordb.chroma import ChromaAdapter

adapter = ChromaAdapter(collection_name="omnidoc")   # in-memory

# Persistent
import chromadb
adapter = ChromaAdapter(
    collection_name="omnidoc",
    client=chromadb.PersistentClient(path="/data/chroma"),
)

count = adapter.upsert(chunks)
results = adapter.query("Q3 revenue growth", n_results=5, where={"intent": "metric"})

Requires pip install "omnidoc-rag[chroma]".

---

Pinecone

from omnidocrag.vectordb.pinecone import PineconeAdapter

adapter = PineconeAdapter(
    index_name="omnidoc-prod",
    embedding_fn=lambda text: model.encode(text).tolist(),
    api_key="pc-xxxxxxxxxxxxx",
    namespace="reports",
)

count = adapter.upsert(chunks)
results = adapter.query("EBITDA margin", top_k=5)

Requires pip install "omnidoc-rag[pinecone]".

---

Weaviate

from omnidocrag.vectordb.weaviate import WeaviateAdapter

# Local Weaviate (localhost:8080)
adapter = WeaviateAdapter(
    collection_name="OmnidocChunks",
    embedding_fn=lambda text: model.encode(text).tolist(),
)

# Weaviate Cloud
adapter = WeaviateAdapter(
    collection_name="OmnidocChunks",
    embedding_fn=lambda text: model.encode(text).tolist(),
    wcd_url="https://my-cluster.weaviate.network",
    wcd_api_key="wcd-api-key",
)

count = adapter.upsert(chunks)
from weaviate.classes.query import Filter
results = adapter.query(
    "revenue growth", limit=5,
    filters=Filter.by_property("intent").equal("metric"),
)
adapter.close()

Requires pip install "omnidoc-rag[weaviate]".

---

PostgreSQL / pgvector

from omnidocrag.vectordb.pgvector import PgVectorAdapter

adapter = PgVectorAdapter(
    embedding_fn=lambda text: model.encode(text).tolist(),
    dsn="postgresql://user:password@localhost:5432/ragdb",
    table="omnidoc_chunks",
    dimensions=384,
    create_index=True,
)

count = adapter.upsert(chunks)
results = adapter.query("revenue growth", top_k=5)
results = adapter.query(
    "revenue growth", top_k=5,
    where="intent = %s", where_params=("metric",),
)
adapter.delete(["id1", "id2"])

with PgVectorAdapter(embedding_fn=..., dsn=..., dimensions=384) as adapter:
    adapter.upsert(chunks)

Requires pip install "omnidoc-rag[pgvector]" and CREATE EXTENSION IF NOT EXISTS vector; on your PostgreSQL instance.

---

Adapter comparison

	ChromaDB	Pinecone	Weaviate	pgvector
Embedding fn required	No	Yes	No (or custom)	Yes
Self-hosted	Yes	No	Yes / WCD	Yes
Persistent by default	No (in-memory)	Yes	Yes	Yes
Filter on query	Yes (where dict)	No	Yes (Filter API)	Yes (raw SQL)
Similarity metric	Cosine (distance)	Cosine	Cosine / certainty	Cosine (<=>)
Install extra	chroma	pinecone	weaviate	pgvector

---

Schema Reference

from omnidocrag.schema import SemanticChunk

chunk.id           # str  — SHA1 deterministic ID
chunk.text         # str  — chunk content
chunk.intent       # str  — metric|table|process|value_proposition|heading|narrative
chunk.confidence   # float — 0.1 … 1.0
chunk.page         # int  — source page number
chunk.heading      # str | None — nearest ancestor heading
chunk.keywords     # List[str] — top-15 BM25 non-stopword terms
chunk.metadata     # Dict[str, Any] — source, chunk_index, char_length,
                   #                  embedding_hint, chunk_type, + type-specific fields

chunk.to_dict()    # → plain dict, JSON-safe, vector DB ready

All Excel chunks carry additional chunk_type and type-specific metadata fields (see Excel — 57 Chunk Types).

---

Package Structure

omnidoc-rag/
├── omnidocrag/
│   ├── __init__.py              Public API — chunk_document, stream_chunks,
│   │                            evaluate_rag_result, get_chunker, …
│   ├── schema.py                SemanticChunk dataclass
│   │
│   ├── core/                    NLP primitives
│   │   ├── intent.py            classify_intent() — deterministic regex classifier
│   │   ├── confidence.py        score_chunk() — quality scoring
│   │   ├── adaptive.py          tokens_for_intent() — per-intent token budgets
│   │   └── hybrid_metadata.py   BM25 keywords + SHA1 embedding hint
│   │
│   ├── pipeline/                RAG pipeline components
│   │   ├── chunker.py           chunk_document() — registry dispatcher
│   │   ├── stream.py            stream_chunks() — lazy generator
│   │   ├── graph.py             link_chunks() — NEXT/SAME_INTENT/METRIC_OF
│   │   ├── stitcher.py          stitch_documents() — cross-doc merging
│   │   └── evaluation.py        evaluate_rag_result() — retrieval scoring
│   │
│   ├── chunkers/                Modular file-type chunkers
│   │   ├── __init__.py          Registry — get_chunker(), chunk_by_type(),
│   │   │                        supported_extensions()
│   │   ├── base.py              BaseChunker ABC + shared helpers
│   │   │
│   │   ├── office/
│   │   │   ├── pdf.py           PDFChunker — heading-aware + table rows
│   │   │   ├── docx.py          DOCXChunker — paragraph + table batches
│   │   │   ├── pptx.py          PPTXChunker — slide-per-chunk + speaker notes
│   │   │   └── xlsx/            XLSXChunker — 57 chunk types
│   │   │       ├── __init__.py  XLSXChunker (orchestrator + fallback)
│   │   │       ├── _helpers.py  Shared helpers (make_chunk, infer_dtype, …)
│   │   │       ├── structural.py   Chunks 1–9
│   │   │       ├── semantic.py     Chunks 10–15
│   │   │       ├── analytical.py   Chunks 16–23
│   │   │       ├── validation.py   Chunks 24–28
│   │   │       ├── visual.py       Chunks 29–41
│   │   │       ├── vba.py          Chunks 42–46
│   │   │       ├── crossref.py     Chunks 47–50
│   │   │       ├── connectivity.py Chunks 51–54
│   │   │       └── operational.py  Chunks 55–57
│   │   │
│   │   ├── text/txt.py          TXTChunker
│   │   ├── markdown/md.py       MDChunker
│   │   ├── code/
│   │   │   ├── python.py        PythonChunker
│   │   │   ├── javascript.py    JSChunker (.js + .ts)
│   │   │   └── java.py          JavaChunker
│   │   ├── structured/
│   │   │   ├── csv.py           CSVChunker
│   │   │   ├── json_chunker.py  JSONChunker
│   │   │   ├── xml_chunker.py   XMLChunker
│   │   │   └── yaml_chunker.py  YAMLChunker
│   │   ├── web/html.py          HTMLChunker
│   │   ├── image/image.py       ImageChunker
│   │   ├── archive/archive.py   ArchiveChunker
│   │   ├── ebook/
│   │   │   ├── epub.py          EPUBChunker
│   │   │   ├── odt.py           ODTChunker
│   │   │   └── rtf.py           RTFChunker
│   │   ├── email/
│   │   │   ├── eml.py           EMLChunker
│   │   │   └── msg.py           MSGChunker
│   │   └── notebook/ipynb.py    IPYNBChunker
│   │
│   └── vectordb/                Vector DB adapters
│       ├── chroma.py            ChromaAdapter
│       ├── pinecone.py          PineconeAdapter
│       ├── weaviate.py          WeaviateAdapter (v4 Collections API)
│       └── pgvector.py          PgVectorAdapter (psycopg2 + pgvector)
│
├── tests/
│   ├── test_rag.py              111 tests — intent, confidence, chunker,
│   │                            evaluation, graph, stitcher, stream
│   └── test_vectordb.py         All 4 vector DB adapters (mock-based, no live DB)
├── pyproject.toml
└── README.md

---

Optional Extras

Extra	Install	Unlocks
chroma	pip install "omnidoc-rag[chroma]"	ChromaDB adapter
pinecone	pip install "omnidoc-rag[pinecone]"	Pinecone adapter
weaviate	pip install "omnidoc-rag[weaviate]"	Weaviate v4 adapter
pgvector	pip install "omnidoc-rag[pgvector]"	PostgreSQL + pgvector adapter
all	pip install "omnidoc-rag[all]"	All four adapters

---

Contributing & Development

Setup

git clone https://github.com/your-org/omnidoc-rag.git
cd omnidoc-rag
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate

pip install -e ".[all]"
pip install build twine pytest pytest-cov ruff black mypy

omnidoc-sdk is a required dependency:

pip install omnidoc-sdk
# or from local source:
pip install -e ../omnidoc-sdk

Running Tests

# All 111 tests
pytest tests/ -v

# Specific class
pytest tests/test_rag.py::TestChunkDocument -v
pytest tests/test_vectordb.py::TestChromaAdapter -v

# With coverage
pytest tests/ --cov=omnidocrag --cov-report=term-missing

All tests use fake Document objects and mocked DB clients — no omnidoc-sdk install or live database required.

Lint and type-check

ruff check omnidocrag/
black --check omnidocrag/
mypy omnidocrag/ --ignore-missing-imports

# Auto-fix
black omnidocrag/
ruff check omnidocrag/ --fix

Building & Publishing

rm -rf dist/ build/ omnidocrag.egg-info/
python -m build
twine check dist/*
twine upload --repository testpypi dist/*   # test first
twine upload dist/*                          # production

Versioning

Follows Semantic Versioning. Version is defined in pyproject.toml.

Release Checklist

[ ] ruff check omnidocrag/          — zero errors
[ ] black --check omnidocrag/        — no formatting changes
[ ] pytest tests/ -v                 — all 111 tests pass
[ ] Version bumped in pyproject.toml
[ ] Changelog updated
[ ] rm -rf dist/ && python -m build
[ ] twine check dist/*               — both artifacts PASSED
[ ] TestPyPI round-trip verified
[ ] twine upload dist/*
[ ] git tag vX.Y.Z && git push origin vX.Y.Z

Troubleshooting

ImportError: chromadb/pinecone/weaviate-client/psycopg2 not installed → Install the corresponding extra: pip install "omnidoc-rag[chroma|pinecone|weaviate|pgvector]"

XLSXChunker produces only basic row chunks (fallback mode) → The file path in doc.metadata["file"] must point to an accessible .xlsx file on disk. openpyxl reads it directly for full 57-chunk extraction.

evaluate_rag_result returns overall=0.0 / verdict="unsafe" → The chunks list is empty. Verify chunk_document(doc) ran on a document with non-empty sections.

WeaviateConnectionError — start a local Weaviate with Docker:

docker run -d -p 8080:8080 -p 50051:50051 cr.weaviate.io/semitechnologies/weaviate:latest

pgvector could not open extension "vector"

CREATE EXTENSION IF NOT EXISTS vector;   -- run as superuser

---

Changelog

[0.1.3] — 2026-05-15

Added — Modular file-type chunkers

• 41 file extensions now route to dedicated chunker classes via a central registry
• chunkers/ package with one file per format, organised into sub-packages:
  • office/ — PDFChunker, DOCXChunker, PPTXChunker, XLSXChunker
  • text/ — TXTChunker
  • markdown/ — MDChunker (heading hierarchy, fenced code blocks, MD tables, front matter)
  • code/ — PythonChunker (def/class), JSChunker (function/class/interface), JavaChunker (class/method)
  • structured/ — CSVChunker, JSONChunker, XMLChunker, YAMLChunker
  • web/ — HTMLChunker (semantic tag-aware)
  • image/ — ImageChunker (OCR text or metadata stub)
  • archive/ — ArchiveChunker (manifest + contained-file delegation)
  • ebook/ — EPUBChunker, ODTChunker, RTFChunker
  • email/ — EMLChunker, MSGChunker (Outlook meeting detection)
  • notebook/ — IPYNBChunker (cell-aware, output chunks)
• chunkers/__init__.py — get_chunker(path), chunk_by_type(doc), supported_extensions()
• BaseChunker ABC with shared _make_chunk, _flush, _source, _is_heading helpers

Added — Excel 57 chunk types

• XLSXChunker rewritten as a 10-file package (xlsx/) using openpyxl direct file access
• Structural (9): WorkbookChunk, SheetChunk, TableChunk, SchemaChunk, RowChunk, GroupChunk, ParentContextChunk, MergedCellChunk, FrozenPaneChunk
• Semantic (6): SummaryChunk, SemanticNarrativeChunk, ColumnSemanticChunk, HeaderAliasChunk, EntityChunk, CellAnnotationChunk
• Analytical (8): FormulaDefinitionChunk, FormulaResultChunk, KPIChunk, AggregationChunk, TemporalChunk, TrendChunk, OutlierChunk, LookupMapChunk
• Validation (5): ValidationChunk, ErrorChunk, ConditionalFormatChunk, ProtectionChunk, DataQualityChunk
• Visual (13): ChartChunk, ChartSeriesChunk, ChartAnnotationChunk, PivotTableChunk, PivotFieldChunk, PivotCacheChunk, SlicerChunk, TimelineChunk, SparklineChunk, ShapeChunk, ImageChunk, FormControlChunk, ActiveXControlChunk
• VBA/Macro (5): MacroChunk, VBAModuleChunk, VBAEventChunk, CustomFunctionChunk, RibbonCustomizationChunk
• Cross-ref (4): RelationshipChunk, NamedRangeChunk, ExternalLinkChunk, DependencyGraphChunk
• Connectivity (4): PowerQueryChunk, DataModelChunk, PowerPivotMeasureChunk, ConnectionChunk
• Operational (3): PrintSettingsChunk, ChangeLogChunk, ChunkIndexChunk (master manifest)
• Graceful fallback to basic row-batch extraction when file path is inaccessible

Added — Package restructuring

• core/ sub-package: intent.py, confidence.py, adaptive.py, hybrid_metadata.py
• pipeline/ sub-package: chunker.py, stream.py, graph.py, stitcher.py, evaluation.py
• schema.py remains at package root (shared by all layers)
• All imports updated to canonical new paths; no shim files retained

Changed

• chunk_document() now auto-routes to the right chunker by file extension instead of using the single generic algorithm for all types
• Import paths updated: omnidocrag.core.intent, omnidocrag.pipeline.chunker, etc.
• Test suite updated to canonical import paths; all 111 tests continue to pass

---

[0.1.0] — 2026-04-10

Initial release of omnidoc-rag as a standalone SDK.

Added

• SemanticChunk dataclass with to_dict() serialisation
• classify_intent() — deterministic 6-label regex classifier
• tokens_for_intent() — per-intent token budgets (heading=60 → process=400)
• score_chunk() — confidence scoring with intent-aware length-penalty exemption
• hybrid_metadata() — BM25 keyword extraction + SHA1 embedding hint
• chunk_document() — heading-aware chunker with overlap carry-over and table row expansion
• stream_chunks() — true lazy generator
• evaluate_rag_result() — coverage / confidence / diversity / verdict scoring
• link_chunks() — NEXT, SAME_INTENT, METRIC_OF edge graph
• stitch_documents() — cross-document heading+intent merging
• ChromaAdapter, PineconeAdapter, WeaviateAdapter (v4), PgVectorAdapter
• 111-test suite; mock-based vector DB tests requiring no live connections

---

omnidoc-rag  ·  v0.1.3  ·  Apache 2.0  ·  Extraction layer → omnidoc-sdk