SEMQ

Deterministic symbolic states for float32 vectors.

When you re-embed a corpus (new GPU, new batch size, a library bump, a model update), you need to know whether anything actually changed, and if so, where it changed. SEMQ turns unit-norm float32 vectors produced by your embedding or feature pipeline into compact symbolic rows kept under your ids, with an identity for the whole state derived from the rows themselves (a SHA-256). You can measure your pipeline's own rebuild noise (via Floor) and diff new builds against it row by row, so rebuild noise passes and real changes fail with identical results in every binding and on every architecture.

A hash of the raw floats tells you whether the bytes changed, not whether the state did. A rebuild on another device or with another batch size never produces the same bytes, so a float hash fires on every rebuild and says nothing about where. SEMQ encodes each vector as a few quantized symbols, so rebuild noise below the bin width leaves the rows and their state_id unchanged; for the rows that do move, diff against a floor measured on your own null rebuilds tells noise from change, row by row.

Build with our Python, Rust, Go, and TypeScript/WASM bindings, or call the C core directly.

SEMQ does not generate embeddings, train a model, normalize vectors, or search. The input is unit-norm, finite float32.

What the SDK provides

The bytes and verdicts are pinned by the shared conformance vectors.

Where it fits

embedding model → float32 unit-norm vectors → SEMQ state (.semq) → diff, floor, verdict

Install

The following source installation keeps the Python binding and core at the same revision. It requires Python 3.11+, a C/C++ toolchain, CMake 3.20+, and Ninja.

git clone https://github.com/The-SEMQ-Group/semq.git
cd semq
python3 -m venv .venv
. .venv/bin/activate
python -m pip install cmake ninja
python -m pip install .

These commands use a POSIX shell. See Installation for Windows setup and the Rust, Go, and TypeScript builds.

Encode, save and compare

Three embeddings of dimension 4 under your own ids, no model or dataset needed. Encode them, save the state, load it back and diff it against a rebuild in which doc-3 changed:

import numpy as np
from semq import Codec, Encoding

vectors = np.array([
    [0.6, 0.8, 0.0, 0.0],
    [0.0, 0.6, 0.8, 0.0],
    [0.0, 0.0, 0.6, 0.8],
], dtype=np.float32)
ids = ["doc-1", "doc-2", "doc-3"]
manifest = {"encoder": "example-encoder", "encoder_revision": "1"}

codec = Codec.quant(dim=4, bins=4)
codec.encode(vectors, ids=ids, manifest=manifest).save("corpus.semq")

reference = Encoding.load("corpus.semq")
print(reference)

rebuilt = vectors.copy()
rebuilt[2] = [0.8, 0.0, 0.0, 0.6]  # doc-3 changed
print(reference.diff(codec.encode(rebuilt, ids=ids, manifest=manifest)))

Expected output:

Encoding(3 rows, quant(dim=4, bins=4), state_id 6e6f8a89bae4...)
1 of 3 rows changed: doc-3 (hamming 3). 0 added, 0 removed.

diff.as_dict() is the machine-readable report. To tell a change like this from the noise of a rebuild that changed nothing, measure a floor from null rebuilds and gate against it: the quickstart does that in four languages, and Gate a rebuild from the command line.

Documentation

Read the SDK documentation site or browse its source landing page.

License and community

The SDK is licensed under the PolyForm Noncommercial License 1.0.0. Production use by companies, and any other use the license does not permit, requires a separate license from The SEMQ Group Inc.; write to licensing@thesemqgroup.ai. The licensing FAQ answers common questions about permitted use. See NOTICE for the required notice and the Code of Conduct for community expectations. The npm package also includes the third-party components listed in THIRD_PARTY_LICENSES.