koji-ingest 0.1.0rc0

Ingestion pipeline for the koji-db database: parsing, chunking, and multi-vector embeddings

Shikomi

Ingestion pipeline for Koji - parsing, chunking, and multi-vector embeddings.

Installation

pip install shikomi

Usage

from shikomi import MultiVectorEmbedding, TextChunk, IngestConfig
import numpy as np

# Create an embedding
data = np.random.randn(10, 128).astype(np.float32)
emb = MultiVectorEmbedding(num_tokens=10, dim=128, data=data)

# Serialize to BLOB format for storage
blob = emb.to_blob()

# Deserialize from BLOB
recovered = MultiVectorEmbedding.from_blob(blob)

License

MIT