Skip to main content

embedxray

Diagnose what your sentence embeddings encode and how much retrieval changes when you compress them. embedxray evaluates a Sentence Transformers model on your own corpus, queries, and contrast pairs. It runs inference without updating model weights and restores the model's training mode after evaluation.

Diagnostic What it measures
Token attribution Which tokens align most strongly with the pooled embedding, and how much of their positive contribution comes from boilerplate.
Contrast collisions How often text pairs you expect to distinguish exceed a cosine similarity threshold.
Embedding geometry Sampled pairwise cosine, spectral entropy, and the fraction of low-variance dimensions.
Hubness Whether a small set of documents appears disproportionately often in the nearest neighbors.
Compression sensitivity Top-k neighbor overlap and vector fidelity across dimension truncation and FP16, INT8, INT4, and binary simulations.

Installation

Requires Python 3.10 or later, PyTorch, and Sentence Transformers. The test suite has passed on CPU with Sentence Transformers 3.0.0 and 6.3.0.

Install the package from PyPI once the first release is live.

python -m pip install embedxray

Install the current version directly from GitHub.

python -m pip install 'git+https://github.com/asysdev/embedxray.git'

Quickstart

Use representative documents and queries from your retrieval workload. The first run of this example downloads the named model if it is missing from the local cache. You can also pass a local model directory to SentenceTransformer.

from sentence_transformers import SentenceTransformer

from embedxray import EmbeddingXRayEvaluator

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

evaluator = EmbeddingXRayEvaluator(
    corpus_texts=[
        "Reset your password using the link on the sign-in page.",
        "Add a recovery phone number in your account settings.",
        "Update your payment method from the billing dashboard.",
        "Export past invoices from the billing dashboard.",
        "Contact support if your account is locked.",
    ],
    query_texts=["How do I reset my password?", "Where can I download an invoice?"],
    contrast_pairs=[
        ("Two-factor authentication is enabled.", "Two-factor authentication is disabled."),
    ],
    k=2,
    matryoshka_dims=[384, 128, 64],
    cliff_threshold=0.95,
)

report = evaluator.evaluate_report(model, output_path="xray_out")
print(f"X-Ray health score: {report.health_score:.1f}/100")
print(report.fracture["fracture_cliff_config"])

Call evaluator(model) to get a flat metric dictionary for logging or a Sentence Transformers trainer. The default primary metric is xray_health_score. The most recent structured report is available as evaluator.last_report.

Evaluation options

Option Behavior
query_texts With no queries supplied, the corpus is queried against itself and each row excludes itself. An explicit empty list skips retrieval.
contrast_pairs Supply pairs that should be distinguishable. A collision occurs above collision_threshold, which defaults to 0.90.
attribution_texts Defaults to the first 64 corpus entries. Supply representative text for attribution, or an empty list to skip it.
matryoshka_dims, quantization_modes Select prefix dimensions and simulated precisions. Each selected dimension includes an FP32 result.
k, chunk_size Set the neighbor count and similarity block size. See report.metadata["actual_k"] for the effective count after clamping to the eligible corpus size.

Reading the results

  • Health score is a 0–100 summary of geometry, dimension health, hubness, compression stability, and attribution or contrast collisions. Compare runs on the same corpus, queries, and settings. Use labeled retrieval metrics such as Recall@k or nDCG to measure relevance.
  • Overlap@k is the average fraction of the full-dimensional FP32 neighbor set retained by a configuration. Self-similarity drift reports mean cosine similarity between original and simulated vectors at the same dimension. Values closer to 1 indicate better directional preservation.
  • Fracture cliff is the lowest estimated vector payload among configurations that reach the overlap threshold. The simulation estimates vector payload from dimension and precision. Index overhead, calibration metadata, padding, and retrieval latency are outside that estimate.
  • Unavailable measurements appear as null in the structured report and are omitted from flat attribution and overlap metrics. Attribution samples include an unavailable_reason when token features cannot be aligned. When overlap cannot be measured, the cliff uses the full FP32 storage estimate.

When output_path is supplied, the evaluator writes these files.

File Contents
xray_report.md Human-readable diagnostic summary and compression grid.
xray_report.json Structured scores, per-sample token evidence, and details of offending contrast pairs. This file can contain original input text.
xray_evaluation_results.csv An appended flat metric row with epoch and step columns.

Use name to label report filenames and prefix metric keys. Use write_csv=False or write_report=False to disable the corresponding exports.

Using precomputed embeddings

Run geometry or compression diagnostics directly on PyTorch tensors shaped (samples, dimensions). For example, use evaluate_fracture(corpus_embeddings, query_embeddings, k=10) to inspect the compression grid. The corresponding geometry entry points are compute_anisotropy, compute_dimension_health, and compute_hubness, all exported from embedxray.

Retrieval uses exact batched similarity comparisons. Runtime increases with the number of query–corpus pairs and compression configurations.

Development

git clone https://github.com/asysdev/embedxray.git
cd embedxray
python -m pip install -e '.[dev]'
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python -m pytest

The tests use synthetic embeddings and locally constructed models, so they run without downloading model weights.

License

Apache License 2.0

Metadata

Release files for embedxray 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for embedxray 0.1.1
File Size Uploaded Provenance Actions
embedxray-0.1.1.tar.gz 32.9 kB With provenance Details for embedxray-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for embedxray 0.1.1
File Interpreter ABI Platform Actions
embedxray-0.1.1-py3-none-any.whl Python 3 none any Details for embedxray-0.1.1-py3-none-any.whl

Total release size: 60.2 kB

Release files / embedxray-0.1.1.tar.gz

Download URL embedxray-0.1.1.tar.gz
Size 32.9 kB
Tags Source
SHA-256 checksum
How to use checksums
139556e0f82f9a5ecf8a281c61a8bb70f384fd56a602108979b0f50f3bbe5eb0
BLAKE2b-256 checksum
How to use checksums
d90e5c42021185a171febdd4d60e5076dab07524ca65579440568eaff5c5d273
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Release files / embedxray-0.1.1-py3-none-any.whl

Download URL embedxray-0.1.1-py3-none-any.whl
Size 27.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
d8215ba3253f9520dde044ceee3e58bfd2ff28be6c05377931da0c58047d0ddb
BLAKE2b-256 checksum
How to use checksums
653f5bfb824392b13f951dbc2502159493ef1fa9daab3d29d333fb5f9f9fc395
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Security for embedxray 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for embedxray-0.1.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page