vera-doc 0.2.3

Embedded storage and search engine for portable .vera archives

vera-doc

vera-doc is VERA's embedded storage and search engine. It stores ready-made text chunks in a portable SQLite .vera file and provides transactional CRUD, embeddings, metadata filters, keyword search, vector search, hybrid search, corpus search, and rebuildable library indexes.

It intentionally contains no PDF parsing, OCR, source extraction, chunking, MCP, CLI, or desktop dependencies. Applications extract and chunk content before calling vera-doc. The separate vera-ingest package provides the standard PDF pipeline.

Documentation: vera-doc guides and API reference

Install

python -m pip install vera-doc

Python 3.10 or newer is required. The default hashing embedder needs no model download or API key.

Quick start

from vera import ChunkRecord, VeraDocument

records = [
    ChunkRecord(
        id="pipe-requirement",
        text="The minimum pipe diameter is 12 inches.",
        metadata={
            "source_filename": "manual.pdf",
            "page_start": 42,
            "heading_path": "Chapter 4 > Pipe Design",
        },
    )
]

with VeraDocument.create("manual.vera") as document:
    document.add(records)

with VeraDocument.open("manual.vera") as document:
    results = document.search(
        text="minimum pipe size",
        mode="hybrid",
        top_k=5,
    )

for result in results:
    print(result.score, result.record.text)

VeraDocument.open() is read-only by default. Use mode="write" when adding, updating, or deleting records.

What is stored in a .vera file?

A VERA 0.2 file is one SQLite database containing:

manual.vera
├── vera_metadata       Format, embedding configuration, archive metadata
├── chunks              Final searchable text and JSON metadata
├── embeddings          One float32 vector per chunk
├── chunks_fts          SQLite FTS5 keyword index
├── attachments         Optional opaque binary payloads
└── chunk_attachments   Typed links from chunks to attachments

The core schema is conceptually:

CREATE TABLE chunks (
    chunk_id      TEXT PRIMARY KEY,
    text          TEXT NOT NULL,
    metadata_json TEXT NOT NULL,
    created_at    TEXT NOT NULL,
    updated_at    TEXT NOT NULL
);

CREATE TABLE embeddings (
    chunk_id        TEXT PRIMARY KEY REFERENCES chunks(chunk_id),
    model_name      TEXT NOT NULL,
    model_dimension INTEGER NOT NULL,
    vector          BLOB NOT NULL,
    vector_format   TEXT NOT NULL,
    created_at      TEXT NOT NULL
);

CREATE TABLE attachments (
    attachment_id TEXT PRIMARY KEY,
    mime_type     TEXT NOT NULL,
    filename      TEXT,
    data          BLOB NOT NULL,
    hash          TEXT NOT NULL,
    metadata_json TEXT NOT NULL,
    created_at    TEXT NOT NULL
);

Pages, headings, citations, bounding boxes, and source identity are optional chunk metadata. Original files and extracted images may be stored as opaque attachments. vera-doc stores these values but does not interpret or extract them.

Public objects

ChunkRecord

The only indexed record type:

ChunkRecord(
    id: str,
    text: str,
    metadata: Mapping[str, JSONValue] = {},
    vector: Sequence[float] | None = None,
    attachments: tuple[AttachmentRef, ...] = (),
)

• id is a non-empty caller-controlled identifier.
• text is final chunk text. vera-doc never splits or cleans it.
• metadata may contain any JSON-compatible object.
• vector may contain a precomputed embedding. When omitted, the configured embedding function embeds text.
• attachments links the chunk to stored attachments.

Records are immutable. IDs, text, metadata, vectors, and attachment references are validated when the object is created or written.

AttachmentRecord

An optional opaque binary payload:

AttachmentRecord(
    id: str,
    data: bytes,
    media_type: str,
    filename: str | None = None,
    checksum: str | None = None,
    metadata: Mapping[str, JSONValue] = {},
)

The SHA-256 checksum is computed automatically. If a checksum is supplied, it must match the bytes. Attachments are not embedded or searchable.

AttachmentRef

Links a chunk to an attachment:

AttachmentRef(
    attachment_id="source-pdf",
    role="source",
)

The role is caller-defined. Common roles include source, figure, and viewer_data.

QueryResult

Returned by VeraDocument.search():

QueryResult(
    record: ChunkRecord,
    score: float,
    semantic_score: float | None,
    keyword_score: float | None,
)

Call result.as_dict() for a JSON-compatible result without the raw vector.

EmbeddingFunction

A structural protocol for custom embedders:

class EmbeddingFunction:
    model_name: str
    dimension: int

    def embed(self, texts: list[str]) -> numpy.ndarray:
        ...

The same model and dimension must be used for stored records and text queries.

VeraDocument methods

Create and open

VeraDocument.create(
    path,
    *,
    embedding_function=None,
    model="hashing",
    metadata=None,
    overwrite=False,
)

VeraDocument.open(
    path,
    *,
    mode="read",
    embedding_function=None,
)

create() publishes a valid database atomically. It raises FileExistsError unless overwrite=True. Both methods return context managers.

Add records

document.add(records)

Inserts an iterable of ChunkRecord objects. Existing IDs raise DuplicateRecordError. The chunk row, embedding, FTS row, and attachment links are written in one transaction.

Insert or replace records

document.upsert(records)

Inserts new IDs and replaces existing records. Replacement updates text, metadata, embedding, keyword index, and attachment links together.

Retrieve records

document.get(
    ids=None,
    *,
    where=None,
    limit=None,
)

Returns ChunkRecord objects, including their vectors and attachment links. where performs exact equality matching on top-level metadata keys:

records = document.get(where={"discipline": "civil"})

Delete records

deleted_count = document.delete(
    ids=None,
    *,
    where=None,
)

Deleting a chunk also deletes its embedding, keyword-index row, and attachment links. It does not delete the attachments themselves.

Search

document.search(
    *,
    text=None,
    vector=None,
    mode="hybrid",
    where=None,
    top_k=10,
)

Supported modes:

• keyword uses SQLite FTS5 and BM25 ranking.
• semantic uses cosine similarity against stored vectors.
• hybrid independently normalizes semantic and keyword scores, then combines them with equal weight.

Semantic search accepts query text or a compatible precomputed vector. Keyword and hybrid search require text.

Attachments

document.put_attachments(attachments, upsert=False)
attachment = document.get_attachment("source-pdf")
document.delete_attachment("source-pdf")

Referenced attachments cannot be deleted until their chunk links are removed. Missing attachments raise RecordNotFoundError.

Archive metadata

metadata = document.metadata
document.set_metadata({"project": "stormwater"})

Archive metadata is a JSON-compatible object separate from per-chunk metadata.

Transactions

with document.transaction():
    document.put_attachments(attachments)
    document.add(records)

The entire block commits together. An exception rolls it back. Nested transactions are intentionally rejected.

Inspection and validation

info = document.inspect()
report = document.validate()

Inspection reports the format, model, dimension, normalization policy, counts, and archive metadata. Validation checks SQLite integrity, required tables and metadata, embedding and FTS parity, vector lengths, declared L2 normalization, JSON payloads, foreign keys, and attachment hashes. Older archives without a normalization policy report unknown and remain valid.

Close

document.close()

Context managers call close() automatically.

Exceptions

• DuplicateRecordError — add() received an existing ID.
• RecordNotFoundError — a chunk references an unknown attachment or a requested attachment does not exist.
• ReadOnlyError — a mutation was attempted after a read-only open.
• Standard FileNotFoundError, FileExistsError, TypeError, and ValueError are used for ordinary path and validation failures.

Optional attachments example

from vera import (
    AttachmentRecord,
    AttachmentRef,
    ChunkRecord,
    VeraDocument,
)

source = AttachmentRecord(
    id="source-pdf",
    data=pdf_bytes,
    media_type="application/pdf",
    filename="manual.pdf",
    metadata={"role": "source"},
)

chunk = ChunkRecord(
    id="chunk-1",
    text="The final, already-extracted chunk.",
    metadata={"page_start": 42},
    attachments=(AttachmentRef("source-pdf", role="source"),),
)

with VeraDocument.create("manual.vera") as document:
    with document.transaction():
        document.put_attachments([source])
        document.add([chunk])

Custom embeddings

import numpy as np

from vera import ChunkRecord, VeraDocument


class MyEmbedder:
    model_name = "example/my-embedder"
    dimension = 2

    def embed(self, texts: list[str]) -> np.ndarray:
        return np.asarray([[1.0, 0.0] for _ in texts], dtype=np.float32)


embedder = MyEmbedder()

with VeraDocument.create(
    "custom.vera",
    embedding_function=embedder,
) as document:
    document.add([ChunkRecord(id="one", text="Example text")])

with VeraDocument.open(
    "custom.vera",
    embedding_function=embedder,
) as document:
    results = document.search(text="example", mode="semantic")

Callers may instead provide ChunkRecord.vector and search with a query vector.

Libraries of .vera files

VeraCorpus searches a directory of .vera files as one corpus:

from vera import VeraCorpus

with VeraCorpus.open("./library", recursive=True) as corpus:
    results = corpus.search("detention requirements", top_k=5)

For larger libraries, create a persistent derived index:

from vera import (
    build_library_index,
    library_index_status,
    update_library_index,
)

build_library_index("./library", recursive=True)
print(library_index_status("./library"))
update_library_index("./library")

The .vera-index/ directory is rebuildable. Individual .vera files remain the source of truth.

Package source structure

src/vera/
├── __init__.py          Public exports
├── models.py            Chunk, attachment, and query value objects
├── document.py          Storage, CRUD, search, and viewer helpers
├── corpus.py            Multi-file corpus search
├── collection.py        Persistent library index
└── core/
    ├── schema.py        SQLite schema and format version
    ├── validation.py    Integrity and contract validation
    └── embeddings.py    Embedders and vector serialization

Source ingestion lives under packages/vera-ingest, and MCP integration lives under packages/vera-mcp.

Format and API references

• VERA 0.2 specification
• Full Python API guide
• Architecture