VectorPrism

Positional Subspace Multiplexing (PSM) & Intent-Gated 2-Stage Retrieval Engine for High-Scale RAG.

One contiguous 1024d tensor. Six independently trained relevance subspaces. Stage-1 HNSW + Stage-2 intent-gated rescoring. Baseline vector-DB storage cost — not 6× multi-vector inflation.

Interactive demo · Benchmarks · Pilot guide · Bitemporal · Technical report

---

Critical: VectorPrism is the embedding + retrieval path — not a chunker

Developer feedback we keep seeing: teams feed docs through an AI stack that mentions VectorPrism (or copies our chunk JSONL shape), then embed and search with Onyx / default dense embeddings. That path cannot deliver VectorPrism recovery results.

Step	Who owns it	What to use
Split documents into passages	You (Onyx, LangChain, custom splitter, etc.)	Plain chunk_text
Encode each chunk into the index	VectorPrism	Frozen base encoder + MultiTaskProjectionAdapter checkpoint → 1024d (6 channel slices + header)
Query encode + Stage-1/2 search	VectorPrism	Same checkpoint + same base encoder

Supported

your chunker → chunk_text
           → VectorPrism encode (encoder + adapter ckpt) → store 1024d
           → VectorPrism search (same ckpt) → multi-channel top-k

Not supported (will look like “VectorPrism didn’t help”)

your chunker → Onyx / OpenAI / Voyage / other dense vector → cosine ANN

Foreign embeddings are a different vector space. They do not populate the dense / relational / disentangled / hyperbolic / identity / causal slices, so Stage-2 fusion never runs on real channel signal. Benchmarks (dense Miss@10 → multi recovery) only apply when both ingest and search use VectorPrism’s 6-channel tensors.

CLI ingest/search print this banner and warn on encoder↔checkpoint mismatch (vectorprism.encode_guards).

---

Keywords: pgvector multi-vector cost reduction, Intent-gated RAG retrieval engine, Causal retrieval for enterprise RAG, Positional subspace multiplexing vector search, Reduce hallucinations in root-cause RAG, VectorPrism, HNSW

Benchmarks (adversarial pack)

Scope honesty: the table below is from our calibrated hard_adversarial finance pack (dense is designed to miss). It is not a claim that every public corpus shows the same Miss@10. For partner corpora use scripts/corpus_recovery_audit.py — see demos/external_audit/.

Dense fails on purpose in that pack. Multi-channel recovers those misses.

Metric	Result
Dense R@10	7.1%
Dense Miss@10	13/14 (93%)
Multi z-score recovered@10	13/13 (100%)
RRF recovered@10 (conservative)	10–11/13 (77–85%)
Auto-graph recovered@10	11/13 (85%)
1000-doc scale recovered@10	13/13 (~1.8 ms)

Full tables, caveats, and reproduce commands → BENCHMARKS.md
Interactive query comparison (dense vs multi) → demo site
Raw JSON/MD artifacts → demos/finance_demo/results/

---

The Core Problem (Why VectorPrism?)

Enterprise RAG is stuck between two bad defaults:

1. Flat cosine over a single embedding — semantically “close” neighbors that are causally wrong, taxonomically wrong, or temporally expired. Teams call them funny neighbors; production calls them hallucination fuel.
2. Multi-vector indexing (one ANN index per representation) — better signal, but 500%–1,000% storage and query fan-out on pgvector / Qdrant bills.

VectorPrism multiplexes six specialized representation subspaces plus a 16-float Control Header into a single 1024-dimensional contiguous buffer per chunk:

Constraint	VectorPrism answer
Storage	1× vector footprint (one vector(1024) / named full tensor)
Stage 1	HNSW only on the 368d dense core slice
Stage 2	In-RAM zero-copy slice scoring with intent weights
Early exit	Header filters (epistemic_truth, anchor_dist, model_version) before heavy math
Latency target	< 15ms end-to-end search SLA (see benchmarks)

Philosophically grounded channel design. Engineering-grounded memory contract. Production path for pgvector and Qdrant.

---

High-Value Enterprise Use Cases

1. Root-Cause Causal Analysis & Incident Logs

Keywords: causal retrieval, incident log RAG, DevOps root-cause analysis, “why did the service fail”

When on-call asks “Why did Server X crash at 3 AM?”, cosine-only RAG returns symptom-adjacent text. VectorPrism’s Time ODE & Directional Causality slice ([896:1024)) is trained with an asymmetric bilinear score (q^{\top} M c) (PSMRetrievalEngine.causal_score). Intent routing up-weights the causal channel on “why / cause / reason” queries so Stage-2 rescoring prefers cause→effect order, not merely lexical neighbors.

2. Enterprise Knowledge Graphs & Taxonomy Search

Keywords: hyperbolic embeddings RAG, taxonomy search, medical ontology retrieval, legal hierarchy search

Parent–child trees distort badly in Euclidean space. The Hyperbolic Taxonomy (Porphyry) slice ([640:768)) lives in a Poincaré ball (norm < 1) and is scored with Poincaré distance in Stage 2. Hierarchy intents (“category”, “parent”, “type of”, “tree”) shift IntentClassifier weights toward hyperbolic structure for medical, legal, and product taxonomies.

3. Bitemporal & Compliance Audit Trail Retrieval

Keywords: bitemporal retrieval, compliance RAG, healthcare audit trail, finance document expiry filter

Opt-in only (0.1.3+). Default search is unchanged: if you never pass as_of / as_of_transaction (CLI: never pass --as-of / --as-of-transaction), Stage-1 behaves exactly as before. Temporal gates are exact int64 filters, not a 7th embedding channel. Design: docs/BITEMPORAL.md.

Mode	What happens
Default (no flags)	All ingested chunks compete in Stage-1/2 as usual
Opt-in as_of=T	Only chunks with valid time covering T ([valid_from, valid_to))
Opt-in as_of_transaction=T	Plus: only chunks the system had recorded by T

Before Stage-2 matrix math, the 16d Control Header Manifest ([0:16)) exposes O(1) metadata:

• Epistemic truth score (soft by default; hard filter opt-in after ECE calibration)
• Identity anchor distance (OOD / injection-risk gate in Stage 1)
• Valid time (stored always; filtered only when as_of is set) — valid_timestamp / valid_to_timestamp
• Transaction time (stored when provided; filtered only when as_of_transaction is set)
• Model version for safe re-ingest after retrains (applied as a Stage-1 filter)

# Default — no temporal gate (same product behavior as pre-0.1.3)
hits = engine.search(q, "wire transfer limit", top_k=5)

# Opt-in — only policies/facts true at T
hits = engine.search(q, "wire transfer limit", top_k=5, as_of="2024-06-01T00:00:00Z")

# Default
vectorprism search --checkpoint ckpt.pt --query "wire limit"

# Opt-in bitemporal
vectorprism search --checkpoint ckpt.pt --query "wire limit" --as-of 2024-06-01T00:00:00Z

When opted in, Stage 1 also rejects out-of-window chunks before rescoring. Epochs are always unix seconds as int/BIGINT — never float embedding values.

4. Cost-Optimized Scale for pgvector & Qdrant

Keywords: multi-vector RAG cost reduction, pgvector HNSW, Qdrant named vectors, high-scale vector search

Instead of six ANN indexes, VectorPrism stores one 1024d tensor. Stage 1 indexes only the generated 368d dense_core_slice. Stage 2 pulls the full tensor for the top-~100 candidates and rescored slices in RAM. AI SaaS platforms keep multi-signal retrieval without multi-vector sticker shock.

---

1024-Dimensional Tensor Memory Map (Code Contract)

Ground truth: PSMTensorContract / VectorPrismTensorContract in tensor_contract.py.

1024-d VectorPrism Tensor (float32)
┌──────────────────────────────────────────────────────────────────────────┐
│ [  0 ..  15]  16d   Control Header Manifest                              │
│ [ 16 .. 383] 368d   Dense Semantic Core          (Hume / Wittgenstein)   │
│ [384 .. 511] 128d   Relational Group Algebra     (Aristotle / Al-Khwarizmi)│
│ [512 .. 639] 128d   Disentangled Latent Space    (Jabir)                 │
│ [640 .. 767] 128d   Hyperbolic Taxonomy          (Porphyry)              │
│ [768 .. 895] 128d   Identity Consistency         (Ibn Sina)              │
│ [896 ..1023] 128d   Time ODE & Causality         (Mulla Sadra / Spinoza) │
└──────────────────────────────────────────────────────────────────────────┘
         ▲ Stage-1 HNSW indexes ONLY dense_core [16:384) → 368 dims

Inclusive range	Code slice (start:end)	Dims	Channel	Role
[0000..0015]	HEADER [0:16)	16	Control Header Manifest	Bitmask, truth, anchor dist, timestamp, model version
[0016..0383]	DENSE_CORE [16:384)	368	Dense Semantic Core	L2-normalized cosine space; Stage-1 ANN
[0384..0511]	RELATIONAL [384:512)	128	Relational Group Algebra	Train: TransE (S+R\approx O); serve today: L2 proximity (-|q_{\mathrm{rel}}-c_{\mathrm{rel}}|) (no query-time relation id yet)
[0512..0639]	DISENTANGLED [512:640)	128	Disentangled Latent (Jabir)	VIB latent (z)
[0640..0767]	HYPERBOLIC [640:768)	128	Hyperbolic Taxonomy (Porphyry)	Poincaré ball
[0768..0895]	IDENTITY [768:896)	128	Identity Consistency (Ibn Sina)	Distance-to-frozen (v_0); Stage-1 gate only
[0896..1023]	CAUSAL_TIME [896:1024)	128	Time ODE & Causality (Spinoza)	Scored as (q^{\top} M c)

Header sub-layout (exact packing via PSMTensorContract.pack_header / unpack_header):

Slot	Field	Encoding
[0]	Channel bitmask	uint32 ↔ float32 bit reinterpret
[1]	Epistemic truth	float32 in [0, 1]
[2]	Identity anchor distance	float32
[3:5]	Valid-time timestamp	int64 ↔ 2×float32 bit reinterpret (unix seconds). Never a float value.
[5]	Model version	uint32 ↔ float32 bit reinterpret
[6:8]	Transaction time (optional)	Same int64 packing when set; zeros = unset
[8:16]	Reserved	zero-filled

---

Quickstart & Code Examples

Installation

# From PyPI
pip install "vectorprism[all]"

# Or from git (latest main / full adversarial packs)
git clone https://github.com/insightitsGit/VectorPrism.git
cd VectorPrism
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -U pip
pip install -e ".[all]"

vectorprism version
vectorprism pilot-check
pytest test_psm.py test_phases.py -q

The PyPI wheel ships schema.sql and data/*.example.jsonl (enough for pilot-check / run-all-smoke). Full adversarial finance packs stay in git, not on PyPI.

Publish / release: PUBLISH.md · External pilot: PILOT.md · Production: PRODUCTION.md

Core deps: torch, numpy, scipy, scikit-learn. Optional extras: encoder, postgres, qdrant, dev, all.

Production path (Docker + Postgres/pgvector) — recommended on Windows

docker compose up -d db
docker compose run --rm test
docker compose run --rm finance-pg
docker compose run --rm production-smoke

• DB: localhost:5433 · DSN postgresql://vectorprism:vectorprism@localhost:5433/vectorprism
• Results: demos/finance_demo/results/ (PRODUCTION_RESULTS.md, eval, live search JSON)
• Full checklist: PRODUCTION.md · Docker notes: DOCKER.md

Example 1 — Multi-Task Ingestion Adapter

Encode raw text with a frozen 768d encoder → MultiTaskProjectionAdapter → contiguous 1024d tensor (matches ingestion_adapter.py + ingest_pipeline.py).

import time
import torch
import numpy as np

from base_encoder import SentenceTransformerEncoder
from ingestion_adapter import MultiTaskProjectionAdapter, VectorPrismProjectionAdapter
from tensor_contract import PSMTensorContract as C, VectorPrismTensorContract
from losses import anchor_distance_score

# Frozen base encoder (768d) + trainable 6-head adapter
encoder = SentenceTransformerEncoder("sentence-transformers/all-mpnet-base-v2")
adapter = MultiTaskProjectionAdapter(base_dim=768)  # alias: VectorPrismProjectionAdapter
adapter.eval()

texts = ["Cache eviction storm preceded the 3 AM outage on Server X."]
base = encoder.encode(texts)  # (1, 768)

header = C.pack_header(
    bitmask=C.default_channel_bitmask({"dense": True, "identity": True, "causal": True}),
    epistemic_truth=1.0,
    anchor_distance=0.0,
    timestamp=int(time.time()),
    model_version=1,
)
header_t = torch.from_numpy(header).unsqueeze(0)  # (1, 16)

with torch.no_grad():
    tensor_1024d, raw = adapter(base, header_t)
    # Fill identity distance into header slot [2]
    dist = anchor_distance_score(raw["identity"], adapter.identity_anchor_v0)
    out = tensor_1024d.cpu().numpy().astype(np.float32)
    out[0, C.HDR_ANCHOR.start] = float(dist[0].item())

assert out.shape == (1, 1024)
assert out[0, C.DENSE_CORE.start:C.DENSE_CORE.end].shape == (368,)
meta = C.unpack_header(out[0])
print(meta)  # bitmask, epistemic_truth, anchor_distance, timestamp, model_version

Production shorthand (upsert path):

from checkpointing import load_checkpoint
from db_client import PgVectorClient  # or QdrantVectorClient
from ingest_pipeline import VectorPrismIngestPipeline, IngestDocument
from base_encoder import SentenceTransformerEncoder

ckpt = load_checkpoint("checkpoints/vectorprism.pt")
encoder = SentenceTransformerEncoder("sentence-transformers/all-mpnet-base-v2")
db = PgVectorClient("postgresql://user:pass@localhost:5432/vectorprism")

pipe = VectorPrismIngestPipeline(
    encoder=encoder,
    adapter=ckpt["adapter"],
    db=db,
    model_version=ckpt["model_version"],
    enabled_channels=ckpt.get("enabled_channels"),
)
pipe.upsert_documents([
    IngestDocument(document_id="inc-42", chunk_text="Cache eviction preceded the outage."),
])

Example 2 — Stage 1 / Stage 2 Hybrid Search

Intent classification → HNSW on dense core → zero-copy slice rescoring (PSMRetrievalEngine.search).

from checkpointing import load_checkpoint
from base_encoder import SentenceTransformerEncoder
from db_client import PgVectorClient
from ingest_pipeline import VectorPrismIngestPipeline
from retrieval_engine import PSMRetrievalEngine, IntentClassifier, VectorPrismRetrievalEngine
from tensor_contract import PSMTensorContract as C

ckpt = load_checkpoint("checkpoints/vectorprism.pt")
encoder = SentenceTransformerEncoder("sentence-transformers/all-mpnet-base-v2")
db = PgVectorClient("postgresql://user:pass@localhost:5432/vectorprism")

pipe = VectorPrismIngestPipeline(encoder, ckpt["adapter"], db, model_version=ckpt["model_version"])
engine = PSMRetrievalEngine(  # alias: VectorPrismRetrievalEngine
    db_client=db,
    causal_matrix=ckpt["causal_matrix"],  # learned M for qᵀ M c
    hard_truth_filter=False,              # keep soft until ECE-calibrated
)

query_text = "Why did Server X crash at 3 AM?"
query_1024d = pipe.encode_query(query_text)

# Optional: inspect intent weights (dense, relational, disentangled, hyperbolic, causal)
w_intent, filters = engine.classifier.classify(query_text)
print("w_intent=", w_intent, "filters=", filters)

hits = engine.search(query_1024d, query_text, top_k=5)
for h in hits:
    print(h["document_id"], h["final_score"], h.get("chunk_text", "")[:120])

# Stage-2 scoring uses exact slices, e.g. causal:
#   q_c = query_1024d[C.CAUSAL_TIME.start:C.CAUSAL_TIME.end]
#   s_causal = engine.causal_score(q_c, candidate_causal_matrix)

CLI equivalents:

python train.py --channel dense --data data/dense_pairs.example.jsonl \
  --encoder sentence-transformers/all-mpnet-base-v2 --out checkpoints/vectorprism.pt

python vectorprism.py ingest --checkpoint checkpoints/vectorprism.pt \
  --documents data/documents.example.jsonl --backend pgvector --dsn "$VECTORPRISM_PG_DSN"

python vectorprism.py search --checkpoint checkpoints/vectorprism.pt \
  --query "Why did Server X crash at 3 AM?" --backend pgvector --dsn "$VECTORPRISM_PG_DSN"

---

Database Setup & Schema

PostgreSQL + pgvector

Exact DDL from schema.sql:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE IF NOT EXISTS psm_document_embeddings (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id VARCHAR(255) NOT NULL UNIQUE,
    chunk_text TEXT NOT NULL,

    -- Full 1024-Dimensional Composite Tensor Payload
    tensor_1024d vector(1024) NOT NULL,

    -- Generated Column for Stage 1 Dense Core Slice [16..383] (368d)
    -- pgvector subvector() is 1-indexed: (17, 368) == zero-indexed [16:384)
    dense_core_slice vector(368) GENERATED ALWAYS AS (
        subvector(tensor_1024d, 17, 368)
    ) STORED,

    epistemic_truth FLOAT NOT NULL DEFAULT 1.0,
    anchor_dist FLOAT NOT NULL DEFAULT 0.0,
    valid_timestamp BIGINT NOT NULL,           -- valid_from (unix seconds)
    valid_to_timestamp BIGINT,                 -- exclusive end; NULL = open
    transaction_timestamp BIGINT,              -- when system recorded version
    model_version INTEGER NOT NULL DEFAULT 0,

    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX IF NOT EXISTS idx_psm_dense_core_hnsw
ON psm_document_embeddings
USING hnsw (dense_core_slice vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

CREATE INDEX IF NOT EXISTS idx_psm_epistemic_truth ON psm_document_embeddings (epistemic_truth);
CREATE INDEX IF NOT EXISTS idx_psm_anchor_dist ON psm_document_embeddings (anchor_dist);
CREATE INDEX IF NOT EXISTS idx_psm_model_version ON psm_document_embeddings (model_version);
CREATE INDEX IF NOT EXISTS idx_psm_document_id ON psm_document_embeddings (document_id);

Apply:

psql "$VECTORPRISM_PG_DSN" -f schema.sql

Qdrant named-vector collection

Matches QdrantVectorClient in db_client.py:

from qdrant_client import QdrantClient
from qdrant_client.http import models as qmodels

client = QdrantClient(url="http://localhost:6333")
collection = "psm_document_embeddings"

if not client.collection_exists(collection):
    client.create_collection(
        collection_name=collection,
        vectors_config={
            # Stage 1 — HNSW on dense core only
            "dense_core_slice": qmodels.VectorParams(
                size=368,
                distance=qmodels.Distance.COSINE,
                hnsw_config=qmodels.HnswConfigDiff(m=16, ef_construct=128),
            ),
            # Stage 2 — full 1024d tensor (flat / m=0, not a second ANN tax)
            "full_tensor": qmodels.VectorParams(
                size=1024,
                distance=qmodels.Distance.COSINE,
                hnsw_config=qmodels.HnswConfigDiff(m=0),
            ),
        },
    )

Payload fields used for Stage-1 filters: epistemic_truth, anchor_dist, model_version, valid_timestamp / valid_to_timestamp (as_of), transaction_timestamp (as_of_transaction). Payload also stores chunk_text and document_id.

---

Architecture Benchmarks & SLA

Targets enforced by the design and benchmark_harness.py / live_benchmark.py budgets:

Stage	Operation	Budget
Header	Bitmask / header unpack [0:16) via PSMTensorContract.unpack_header	< 0.05 ms
Stage 1	HNSW coarse search on dense_core_slice (368d) + header filters → top 100	< 10 ms
Stage 2	RAM zero-copy slice rescoring (z-score fuse × w_intent)	< 2 ms
E2E	Encode path excluded in pure Stage-2 harness; search SLA	< 15 ms

# Stage-2 focused latency (synthetic corpus, real PSMRetrievalEngine.search)
python benchmark_harness.py

# End-to-end against a live backend (ingest + encode_query + search)
python vectorprism.py live-benchmark \
  --checkpoint checkpoints/vectorprism.pt \
  --documents data/documents.example.jsonl \
  --backend memory --n-trials 20 --p95-budget-ms 15

Architecture path (unchanged pillars):

Text ─► Frozen 768d Encoder ─► MultiTaskProjectionAdapter ─► 1024d tensor
                              │
                              ▼
              pgvector / Qdrant (dense HNSW + full tensor)
                              │
         IntentClassifier ─► w_intent + filters
                              │
         Stage 1: HNSW(dense_core_slice) + truth/anchor filters
                              │
         Stage 2: dense / rel / dis / hyp / causal scores → top-k
                  (Identity is Stage-1 gate only — not double-counted)

---

Train Channels the Right Way

Channels are earned, not assumed. One channel at a time:

python train.py --channel dense --data your_pairs.jsonl \
  --encoder sentence-transformers/all-mpnet-base-v2 --out checkpoints/vectorprism.pt

python vectorprism.py eval --checkpoint checkpoints/vectorprism.pt \
  --documents your_docs.jsonl --eval your_eval.jsonl \
  --encoder sentence-transformers/all-mpnet-base-v2

# Only after dense DoD: add causal / relational / hyperbolic / ...
python train.py --channel causal --data your_causal.jsonl \
  --init checkpoints/vectorprism.pt --out checkpoints/vectorprism.pt

See IMPLEMENTATION_SPEC.md for phased Definitions of Done and the gap matrix.

---

Deploying VectorPrism at Scale?

Building a regulated RAG stack, a multi-tenant AI SaaS retrieval plane, or a private compliance-aware knowledge system?

Insight ITS works with enterprise architects on:

• Custom multi-task adapter fine-tuning for your ontology / incident / audit corpora
• Private compliance connectors (bitemporal filters, calibrated epistemic truth, HITL review)
• Managed control planes for versioned re-ingest across pgvector & Qdrant fleets

Talk to us

• Email: insightits.info@gmail.com
• Soft CTA: email subject RECOVER → insightits.info@gmail.com
• GitHub Discussions: insightitsGit/VectorPrism
• Community: GitHub Discussions (no public Discord invite is published)

---

License

Apache License 2.0 — see LICENSE (or repository license metadata).

---

Citation

If VectorPrism informs your research or production retrieval stack:

@software{vectorprism2026,
  title  = {VectorPrism: Positional Subspace Multiplexing for Intent-Gated Retrieval},
  author = {Amin Parva},
  year   = {2026},
  url    = {https://github.com/insightitsGit/VectorPrism}
}

---

VectorPrism — six signals, one tensor, baseline storage cost, intent-gated speed.

---

Links

• Author: Amin Parva (insightits.info@gmail.com)
• Company: https://www.insightits.com
• GitHub: https://github.com/insightitsGit/VectorPrism
• PyPI: https://pypi.org/project/vectorprism/
• Product page: https://www.insightits.com/products/vectorprism.html