koji-ingest 0.1.0

Ingestion pipeline for the koji-db database: parsing, chunking, and multi-vector embeddings

Shikomi

Ingestion pipeline for Kōji — document parsing, chunking, multi-vector embeddings, and VLM enrichment.

Shikomi transforms raw documents (PDF, DOCX, PPTX, audio, images, and more) into ColBERT-style multi-vector embeddings ready for storage and semantic search in Kōji.

Installation

# Core ingestion (parsing, chunking, embeddings)
pip install shikomi

# With document parsing support (Docling, PDF)
pip install "shikomi[document]"

# With LibreOffice-based rendering (DOCX/PPTX page images)
pip install "shikomi[rendering]"

# With MLX VLM enrichment engine (Apple Silicon)
pip install "shikomi[enrichment]"

# With Kōji storage integration (PyArrow)
pip install "shikomi[koji]"

Features

Multi-Vector Embeddings

Shikomi generates ColBERT-style multi-vector embeddings where each document is represented by a set of per-token vectors, enabling MaxSim late-interaction scoring:

from shikomi import MultiVectorEmbedding
import numpy as np

data = np.random.randn(10, 128).astype(np.float32)
emb = MultiVectorEmbedding(num_tokens=10, dim=128, data=data)

blob = emb.to_blob()          # serialize for Lance storage
recovered = MultiVectorEmbedding.from_blob(blob)

Document Parsing and Chunking

Parse and chunk documents in a variety of formats via Docling:

from shikomi import parse, chunk, IngestConfig

config = IngestConfig()
parsed = parse("report.pdf", config=config)
chunks = chunk(parsed, config=config)

Supported formats include PDF, DOCX, PPTX, Markdown, HTML, images, and audio files.

DenseTextEngine (Lightweight Embedding)

For text-only workloads without vision, use DenseTextEngine backed by mlx-embeddings:

from shikomi import DenseTextEngine

engine = DenseTextEngine()
embedding = engine.embed("The quick brown fox")

Gemma 4 E4B VLM Enrichment (Apple Silicon)

Augment parsed documents with VLM-generated descriptions, code analysis, formula interpretations, and document summaries using the Gemma 4 E4B model via mlx-vlm:

from shikomi.enrichment import GemmaEnrichmentEngine

engine = GemmaEnrichmentEngine()
enriched = await engine.enrich(parsed_content)
# enriched.figures now contain model-generated captions
# enriched.summary contains a document-level summary

GPU / Metal Memory Management

Release cached GPU or Metal memory between batch operations to prevent OOM errors:

from shikomi.gpu import release_gpu_memory

for batch in large_corpus:
    process(batch)
    release_gpu_memory()   # returns allocations to OS after each batch

LibreOffice Page Renderer

Convert DOCX and PPTX slides to page images for visual ingestion:

from shikomi.parser import parse

# Renders pages via LibreOffice headless, returns per-page images in ParsedContent
result = parse("slides.pptx", config=config)

Audio Chunking

Ingest long-form audio with automatic chunking:

from shikomi import IngestConfig, AudioConfig

config = IngestConfig(audio=AudioConfig(chunk_duration_secs=30))
result = parse("interview.mp3", config=config)

Full Ingestion Pipeline

from shikomi import Ingester, IngestConfig

ingester = Ingester(config=IngestConfig())
result = await ingester.ingest("document.pdf")

print(result.chunks)       # List[TextChunk]
print(result.embeddings)   # List[MultiVectorEmbedding]

License

Apache-2.0