Nexus — Enterprise Intelligence Framework

Nexus is an open-source enterprise intelligence framework for building secure, governed AI applications, retrieval systems, agents, and intelligent workflows.

It sits upstream and around large language models: turning fragmented enterprise data into clean, grounded chunks, semantic embeddings, contextual knowledge graphs, and grounded, policy-checked answers — without locking you into a particular model provider, vector database, or runtime.

Data Connectivity → Processing & Enrichment → Knowledge & Retrieval → Intelligent RAG
                                     → AI Orchestration → Governance → Observability

---

Contents

• What Nexus gives you
• Supported Formats & Modalities
• Installation
• Quick start
• Multimodal Ingestion (Office, Audio, Video, Code, DBs)
• Semantic embeddings & re-ranking
• Live sources: CDC, webhooks & Slack
• Build a RAG workflow
• Architecture
• Configuration
• Running tests
• Upgrading from 3.0.0
• Documentation
• Contributing
• License

---

What Nexus gives you

Nexus provides seven composable capabilities. Each is an independently installable package that talks to the others through typed configs, JSONL contracts, CLI, and HTTP — never by importing another layer's code. That is what makes any layer swappable for your own systems.

Capability	Package	What it does
Data Connectivity	nexus.pipeline	REST connectors with pagination and SSRF defense, batch file drops, streaming events; CDC normalization (Debezium, Maxwell, MongoDB change streams, PostgreSQL pgoutput logical replication, generic webhooks) with HMAC-signed delivery
Processing & Enrichment	nexus.processing	Native pure-Python parsers for Word (.docx), Excel (.xlsx), PowerPoint (.pptx), PDF, Audio (WAV/MP3/AIFF), Video (MP4/MOV), Images (PNG/JPEG/BMP), Code AST, OpenAPI, SQLite, MySQL, MongoDB, Email, Chat, Slack exports, CSV, Markdown, and Text; optional OCR / Whisper / video demuxing; format-preserving tokenization (FF1)
Knowledge & Retrieval	nexus.retrieval	Semantic embeddings (FastEmbed bge-small-en-v1.5, 384D, local ONNX — or OpenAI), cross-encoder re-ranking, lexical (BM25-style), hybrid RRF, and knowledge-graph retrieval with pluggable stores
Intelligent RAG	nexus.guardrails	Grounded answers with citations, PII masking, prompt-injection defense, fail-closed policy checks
AI Orchestration	nexus.experience	REST API, SDK, CLI, assistant sessions, channel adapters, API-key auth
Governance	nexus.security	RBAC, multi-tenant isolation, authenticated encryption, immutable audit log
Observability	nexus.observability	Metrics, structured logs, distributed trace spans, AI interaction events, alerting

Design properties worth knowing about:

• Runs locally. Embeddings and re-ranking run on CPU through ONNX (FastEmbed); models download once and are cached (NEXUS_MODEL_CACHE_DIR). No API calls unless you choose the OpenAI provider.
• Resource-aware. OCR and Whisper models load lazily, are shared process-wide, use CUDA or Apple MPS when present, and can be unloaded when idle (release_idle_models).
• Thread-safe and serverless-friendly. In-memory stores are guarded by threading.Lock; in_memory_only=True (the default) skips disk I/O entirely.
• Typed configuration end to end. Every layer's config is a Pydantic model, so a control plane can introspect the schema and render forms automatically.
• Multi-tenant by construction. Encryption and tokenization derive a tenant-bound salt (HKDF-SHA256), so two tenants processing identical data produce cryptographically distinct ciphertext.

Processing and embedding are separate steps. process_* methods return clean, grounded chunks (text + citations + metadata) and never vectors. Embed at storage time with embed_texts() / embed_query() so your index always uses one model. See Semantic embeddings & re-ranking.

---

Supported Formats & Modalities

Nexus includes pure-Python binary and text decoders. Every modality produces text chunks with format-aware grounded citations, so all of them can be embedded into one semantic index.

Category	Supported Formats	Native Features & Grounding
Office Documents	.docx (Word), .xlsx (Excel), .pptx (PowerPoint)	OpenXML archive decompression; heading hierarchy (Heading1..6) & markdown tables for Word; multi-sheet cell matrices & row narratives for Excel; slide text, speaker notes & layouts for PowerPoint; embedded image OCR.
Documents & Data	.pdf, .csv, .md, .txt, .json, .jsonl	ISO 32000-1 PDF stream decompression & page citations; image XObject extraction; embedded image OCR; CSV row-level narratives ([Row ID: 1] col: val); smart boundary-aware paragraph chunking for Markdown & text.
Audio & Speech	.wav, .aiff, .mp3	Python 3.13-safe RIFF/AIFF/ID3 decoders; RMS loudness; Zero-Crossing Rate; Voice Activity Detection (VAD); 7-band spectral decomposition (20Hz–20kHz); temporal windows ([00:00 - 00:10]); Whisper transcription.
Video	.mp4, .mov, .m4v, .webm	ISO BMFF container parser (moov/trak/mdia/minf); temporal scene framing; keyframe OCR and audio-track transcription (optional [video], [ocr], [audio-ml]).
Images	.png, .jpeg, .jpg, .bmp, .gif, .tiff, .webp	Chunked binary parser; dimensions and metadata; OCR of visible text (optional [ocr], downscaled to 2560 px).
Code & APIs	.py, .ts, .js, .go, .rs, .java, .cpp, .cs, OpenAPI 3.0/3.1, Swagger 2.0	Standard library Python AST (signatures, parameters, decorators, cyclomatic complexity); polyglot regex scanners; OpenAPI operation and schema model extraction.
Databases	SQLite (.db, .sqlite), MySQL, MongoDB	Binary SQLite header and B-tree page extraction; CDC change-event normalizer (Debezium/Maxwell/MongoDB change streams); recursive BSON parser and dot-notation document flattener.
Messaging	.eml (RFC 822/MIME), Chat (slack, teams), Slack export .zip / channel .json, live Slack events	Multipart MIME extraction, DKIM/SPF auth headers, thread conversation resolution, speaker turns, timestamp grounding; Slack threads grouped with replies and mentions resolved to names.

---

Installation

Requires Python 3.11 or 3.12.

pip install veloxs-nexus

Optional extras:

pip install "veloxs-nexus[postgres]"   # pgvector + SQLAlchemy persistence
pip install "veloxs-nexus[yaml]"       # YAML configuration files
pip install "veloxs-nexus[ocr]"        # image / slide / keyframe OCR (EasyOCR)
pip install "veloxs-nexus[audio-ml]"   # speech transcription (faster-whisper)
pip install "veloxs-nexus[video]"      # video demuxing (PyAV)
pip install "veloxs-nexus[all-ml]"     # all three

Semantic embeddings (FastEmbed) are part of the core install. On CPU-only servers, install the CPU build of PyTorch before the ML extras to avoid multi-gigabyte CUDA wheels:

pip install torch --index-url https://download.pytorch.org/whl/cpu

To work on Nexus itself, see CONTRIBUTING.md.

---

Quick start

Process a document through the full pipeline and inspect the execution trace:

import nexus

client = nexus.NexusClient(tenant_id="org-finance", in_memory_only=True)

csv_data = """employee_id,department,salary_usd,contact_email
101,Engineering,145000,john.doe@example.com
102,Security,160000,jane.smith@example.com"""

doc = client.process_document(
    document_id="doc-ledger-01",
    name="salaries.csv",
    text=csv_data,
    file_type="csv",
    enable_guardrails=True,
)

print(f"{doc.name}: {len(doc.chunks)} chunks")
print(doc.chunks[0].text)
# Columns: employee_id, department, salary_usd, contact_email
# [Row ID: 1] employee_id: 101 | department: Engineering | salary_usd: 145000 | contact_email: [EMAIL]
# ...

for step in doc.execution_trace:
    print(f"[{step.step_number}/5] {step.stage_name} ({step.duration_ms}ms)")

Embed the chunks when you store them:

vectors = client.embed_texts([c.text for c in doc.chunks])
print(len(vectors[0]), client.embedding_info())
# 384 {'provider': 'fastembed', 'model': 'BAAI/bge-small-en-v1.5', 'dimensions': 384}

Raw fidelity mode

When you need verbatim text — audit logs, code, account identifiers — bypass redaction:

raw = client.process_document(
    document_id="doc-audit-02",
    name="audit.txt",
    text="Transaction 9842 authorized by admin@example.com",
    file_type="txt",
    enable_guardrails=False,
)
print(raw.chunks[0].text)  # preserved verbatim

---

Multimodal Ingestion (Office, Audio, Video, Code, DBs)

Nexus provides two flexible ways to ingest multimodal content:

1. Universal Ingestion (client.process_document)

Pass raw bytes or a local file path along with the file name. Nexus automatically identifies the format, decompresses binary containers, frames structures, scrubs PII, and returns grounded chunks:

import nexus

client = nexus.NexusClient(in_memory_only=True)

# 1. Spreadsheets (.xlsx) — sheets, rows, and cell matrices
doc_excel = client.process_document(
    name="financial_model.xlsx",
    text=excel_bytes,  # or filepath "path/to/financial_model.xlsx"
)

# 2. Word Documents (.docx) — headings, sections, and markdown tables
doc_word = client.process_document(
    name="master_agreement.docx",
    text=docx_bytes,
)

# 3. Audio Streams (.wav, .mp3, .aiff) — metadata, VAD windows, Whisper transcript when installed
doc_audio = client.process_document(
    name="earnings_call.wav",
    text=wav_bytes,
)

# 4. Video Files (.mp4, .mov) — scene windows, transcript and keyframe OCR when installed
doc_video = client.process_document(
    name="product_walkthrough.mp4",
    text=mp4_bytes,
)

# 5. Images (.png, .jpg) — metadata and OCR text when installed
doc_image = client.process_document(
    name="system_topology.png",
    text=png_bytes,
)

2. Dedicated Modality Methods

When you need granular control over windowing, sample rates, or format-specific parameters, call the dedicated methods directly:

# Word (.docx) with heading hierarchy
doc = client.process_word(name="contract.docx", docx_bytes=raw_bytes)

# Spreadsheets (.xlsx) with sheet & row-level narrative framing
doc = client.process_spreadsheet(name="budget.xlsx", spreadsheet_bytes=raw_bytes)

# Presentations (.pptx) with slide text and speaker notes
doc = client.process_presentation(name="strategy.pptx", presentation_bytes=raw_bytes)

# Audio (.wav, .mp3, .aiff) with configurable temporal windowing
doc = client.process_audio(name="speech.wav", audio_bytes=raw_bytes, window_seconds=10.0)

# Video (.mp4, .mov) with temporal scene framing
doc = client.process_video(name="demo.mp4", video_bytes=raw_bytes, window_seconds=10.0)

# Images (.png, .jpeg, .bmp) with OCR text
doc = client.process_image(name="chart.png", image_bytes=raw_bytes)

# SQLite binary databases (.sqlite, .db)
doc = client.process_sqlite(name="app.db", db_bytes=sqlite_bytes)

# Source code AST (Python, TypeScript, Go, Rust, Java, C++)
doc = client.process_code(name="pipeline.py", code_input=source_code)

# OpenAPI 3.0 / 3.1 & Swagger 2.0 specs
doc = client.process_openapi(name="openapi.json", spec_data=spec_content)

3. Unified Cross-Modal Search

Every modality produces text chunks embedded by the same semantic model, so you can index and query across text, spreadsheets, audio transcripts, and diagrams at once:

# Index multi-format documents into a single collection
client.index_document(doc_excel, collection="enterprise_assets")
client.index_document(doc_word, collection="enterprise_assets")
client.index_document(doc_audio, collection="enterprise_assets")

# Query with natural language across all modalities
results = client.search("quarterly revenue and SLA commitments", collection="enterprise_assets")
for r in results:
    print(f"[{r.score:.3f}] {r.text[:120]}")

---

Semantic embeddings & re-ranking

client = nexus.NexusClient(in_memory_only=True)

passages = ["All database connections require TLS 1.3.", "Employees receive 20 days of PTO."]
doc_vectors = client.embed_texts(passages)                 # passage encoding (batched)
query_vector = client.embed_query("What encryption is required?")  # query encoding
scores = client.rerank("What encryption is required?", passages)   # cross-encoder, 0..1

Setting	Default	Purpose
NEXUS_EMBEDDING_PROVIDER	fastembed	fastembed (local ONNX) or openai
NEXUS_EMBEDDING_MODEL	BAAI/bge-small-en-v1.5	Any FastEmbed text model
NEXUS_EMBEDDING_DIMENSIONS	model size (OpenAI: 384)	Output dimensions
NEXUS_MODEL_CACHE_DIR	FastEmbed default	Where models are cached — mount a volume in containers
NEXUS_ML_DEVICE	auto (cuda > mps > cpu)	Device for OCR / Whisper

Long-running services can free OCR / Whisper memory between jobs:

from nexus.processing.ml_providers import release_idle_models

release_idle_models(max_idle_seconds=600)  # models reload transparently on next use

---

Live sources: CDC, webhooks & Slack

from nexus.processing import cdc, slack_export

# Change data capture — one normalizer for Debezium, Maxwell, MongoDB change streams and generic events
events = cdc.normalize_change_events(payload)       # dict, list, {"events": [...]} or NDJSON
for event in events:
    text = cdc.change_event_text(event)             # one chunk per record
    ...                                             # upsert / delete by event.key

# PostgreSQL logical replication (built-in pgoutput plugin, no server extension)
from nexus.processing.pgoutput import PostgresLogicalStream

stream = PostgresLogicalStream("postgresql://reader@db/bank?sslmode=verify-full", "my_slot", "my_publication")
assert not stream.prerequisites()          # wal_level, REPLICATION role, publication, primary keys
stream.ensure_slot()
for txn in stream.transactions():           # committed transactions, in order
    if txn:
        apply(txn.events)                   # your durable write
        stream.confirm(txn.end_lsn)         # only now may the server free that WAL

# Signed webhook delivery (HMAC-SHA256 with a 5-minute replay window)
ok = cdc.verify_webhook_signature(secret, raw_body, signature_header, timestamp_header)

# Slack Events API
ok = slack_export.verify_slack_signature(signing_secret, raw_body, x_slack_timestamp, x_slack_signature)
action, message, deleted_ts = slack_export.slack_event_message(event)   # upsert | delete | ignore
chunks = slack_export.messages_to_chunks("eng", messages, users)        # same chunking as export files

---

Build a RAG workflow

Index documents and ask grounded questions. Answers are checked against retrieved context and refused when they cannot be grounded:

import nexus

client = nexus.NexusClient(in_memory_only=True)

doc = client.process_document(
    document_id="arch-01",
    name="architecture.md",
    text=(
        "# Infrastructure\n"
        "All database connections require TLS 1.3 encryption "
        "and mutual certificate authentication."
    ),
    file_type="md",
)
client.index_document(doc)

response = client.ask("What encryption is required for database connections?")
print(response.decision)  # allowed
print(response.answer)  # grounded in the indexed chunk

Using layers individually

Every capability works standalone:

# Semantic embedding
from nexus.retrieval.semantic import create_text_embedder

vector = create_text_embedder().embed_query("Enterprise cloud infrastructure")

# PII masking
from nexus.guardrails.pii import mask_pii
from nexus.guardrails.config import PiiConfig

clean = mask_pii("Contact user@example.com", PiiConfig())

# Tenant-bound encryption
from nexus.security.encryption import encrypt_text, decrypt_text
from nexus.security.config import EncryptionConfig

cfg = EncryptionConfig(secret_key="replace-me", tenant_id="org-acme")
plain = decrypt_text(encrypt_text("Confidential Record", cfg), cfg)

# Format-aware chunking
from nexus.processing.engine import ProcessingEngine

chunks = ProcessingEngine().chunk_document("id,val\n1,Alpha\n2,Beta", file_type="csv")

# Batch ingestion
from nexus.pipeline.batch import run_batch

Command line

Every command takes the path to a platform config:

nexus validate-config configs/nexus.json    # load and validate the config
nexus layers configs/nexus.json             # list configured layers
nexus validate-platform configs/nexus.json  # check every layer is ready
nexus ask configs/nexus.json "What is the MFA policy?" --channel assistant

---

Architecture

                        ┌──────────────────────────────┐
   your application ──▶ │  nexus.experience            │  REST · SDK · CLI · channels
                        └──────────────┬───────────────┘
                                       │
                        ┌──────────────▼───────────────┐
                        │  nexus.guardrails            │  grounded RAG · PII · policy
                        └──────────────┬───────────────┘
                                       │
                        ┌──────────────▼───────────────┐
                        │  nexus.retrieval             │  vector · lexical · hybrid · graph
                        └──────────────┬───────────────┘
                                       │
                        ┌──────────────▼───────────────┐
                        │  nexus.processing            │  chunking · enrichment · tokenization
                        └──────────────┬───────────────┘
                                       │
                        ┌──────────────▼───────────────┐
                        │  nexus.pipeline              │  REST · batch · streaming · CDC
                        └──────────────────────────────┘

   cross-cutting:  nexus.security (RBAC · tenancy · encryption · audit)
                   nexus.observability (metrics · logs · traces · alerts)

Layers integrate only through configs, JSONL contracts, CLI, and HTTP. Replacing nexus.retrieval with your own vector database, or nexus.guardrails with your own policy engine, requires no changes to the layers around it.

Full detail: docs/ARCHITECTURE_OVERVIEW.md · docs/USING_NEXUS.md

PostgreSQL + pgvector

For durable persistence, nexus.database builds a reference schema for your embedding size:

from nexus import pgvector_ddl

print(pgvector_ddl(384))   # vector(384) + HNSW index + generated tsvector with a GIN index

Above 2000 dimensions the column becomes halfvec automatically (HNSW supports halfvec up to 4000 dimensions). mysql_ddl(dim) and mongo_atlas_vector_index(dim) cover MySQL 9 and MongoDB Atlas Vector Search.

---

Configuration

Every layer reads a JSON (or, with the [yaml] extra, YAML) config validated by a Pydantic model. The root config at configs/nexus.json wires the layers together.

nexus validate-config configs/nexus.json

Because configs are typed models, you can introspect any layer's schema programmatically:

from nexus.retrieval.config import RetrievalConfig

print(RetrievalConfig.model_json_schema())

Secrets are never read implicitly from the environment by library code. Pass them explicitly, or use the documented env:VAR_NAME indirection. See SECURITY.md.

---

Running tests

The suite is deterministic and needs no cloud services; the first run downloads the small FastEmbed models (~100 MB) into the model cache.

git clone https://github.com/Veloxs-ai/nexus.git
cd nexus
python3 -m venv .venv && source .venv/bin/activate
python -m pip install -e ".[dev]"
python -m pytest -q

Each layer has its own suite:

for layer in enterprise-data-pipeline data-processing-enrichment \
             embedding-retrieval-intelligence orchestration-guardrails \
             experience-api-engagement security-governance \
             observability-monitoring; do
  (cd "$layer" && python -m pip install -e ".[dev]" -q && python -m pytest -q)
done

Lint and format with Ruff:

ruff check .
ruff format --check .

---

Upgrading from 3.0.0

3.0.1 separates processing from embedding and removes the legacy 3072D hashing projection:

Removed	Use instead
NexusClient.embed(), ProcessedChunk.embedding, chunk_embeddings=	client.embed_texts([...]) at storage time, client.embed_query(q) at search time
embed_image / embed_audio / embed_video_scene	Embed the chunk text produced by process_image / process_audio / process_video
HashingEmbedder, local_hashing provider	fastembed (default) or openai
normalize_mysql_cdc_event, normalize_mongo_change_event	cdc.normalize_change_event
PGVECTOR_DDL_SCHEMA, MYSQL_DDL_SCHEMA, MONGO_ATLAS_VECTOR_SEARCH_INDEX	pgvector_ddl(dim), mysql_ddl(dim), mongo_atlas_vector_index(dim)
semantic extra	Nothing — FastEmbed is a core dependency

Stored 3072D vectors are not compatible with the new model: re-embed stored chunk text once with embed_texts(). Full list in the CHANGELOG.

---

Documentation

Guide	What it covers
Integrator Guide	The comprehensive reference — every layer in detail, the security model, extension points, and what is production-grade today. Start here after the quick start.
Architecture Overview	Design principles, the loose-coupling rule, and per-layer capabilities
Integration Guide	Installation, library vs. CLI integration patterns, environment variables
Processing Reference	Ingestion formats, the five processing phases, output structure, database setup
Processing & Embedding Spec	Chunking rules and how chunks are embedded, specified precisely

---

Contributing

Contributions are welcome. Start with CONTRIBUTING.md for the development setup, project conventions, and pull-request process.

• 🐛 Report a bug
• ✨ Request a feature
• 🔐 Report a vulnerability privately — please do not open a public issue
• 💬 Getting help

Everyone participating is expected to follow our Code of Conduct.

---

License

Copyright © 2026 Veloxs AI Inc.

Licensed under the Apache License, Version 2.0. See NOTICE for attribution and third-party dependency information.

Trademarks. "Nexus", "Veloxs", and "Veloxs AI", together with associated logos and branding, are trademarks of Veloxs AI Inc. As set out in Section 6 of the Apache License, this license grants no rights to use these marks. You may state truthfully that your software is built on Nexus; you may not imply endorsement by or affiliation with Veloxs AI Inc. See NOTICE for details.