json-semantic-diff

Semantic similarity scoring for JSON structures — not just whether they differ, but how similar they are.

Overview

json-semantic-diff compares two JSON documents and returns a normalised similarity score in [0.0, 1.0], along with an audit trail of which keys matched, which were renamed, and which are missing. It handles naming convention differences (camelCase, snake_case, PascalCase, kebab-case) transparently.

Traditional JSON diff tools are binary: match or no match. json-semantic-diff tells you how similar two documents are, which is what you actually need when testing LLM outputs, validating API migrations, or measuring generator stability.

Core algorithm: STED (Semantic Tree Edit Distance) — tree edit distance extended with semantic key matching via the Hungarian algorithm and per-level normalisation.

Key Features

• Similarity scoring — Normalised float in [0.0, 1.0] instead of binary match/no-match
• Naming convention tolerance — user_name, userName, UserName, user-name all score 1.0 against each other
• Explainable scores — Per-path breakdown of what drove the score down: value changes (30 -> 31), key renames, keys only on one side
• Rich audit trail — Key mappings, matched pairs, unmatched paths, computation time
• Generator consistency — Single-number stability metric for LLM outputs
• CLI included — json-semantic-diff left.json right.json, with --explain, --json, and CI-friendly --threshold exit codes
• Three backends — Zero-dependency Levenshtein, local ONNX embeddings, or OpenAI cloud
• Evaluation platform adapters — pytest, LangSmith, Braintrust, W&B Weave
• Configurable — Structural/content weights, array comparison modes, type coercion, null handling
• Zero global state — Each comparator instance owns its own cache; thread-safe by design

Installation

pip install json-semantic-diff

With optional backends:

pip install json-semantic-diff[fastembed]   # Local ONNX embeddings (384-dim)
pip install json-semantic-diff[openai]      # OpenAI cloud embeddings (1536-dim)

With evaluation platform adapters:

pip install json-semantic-diff[langsmith]
pip install json-semantic-diff[braintrust]
pip install json-semantic-diff[weave]

Quick Start

Compare two JSON documents

from json_semantic_diff import compare

# Note: `mailAddress` vs `email_address` is a fuzzy (not exact) key match,
# so the score is close to but below 1.0.
result = compare(
    {"user_name": "Alice", "email_address": "alice@corp.com"},
    {"userName": "Alice", "mailAddress": "alice@corp.com"},
)

print(result.similarity_score)   # ~0.99
print(result.key_mappings)       # {"user_name": "userName", "email_address": "mailAddress"}
print(result.unmatched_left)     # ()
print(result.unmatched_right)    # ()
print(result.computation_time_ms)  # sub-millisecond on small docs

# Naming-convention-only differences score a perfect 1.0:
compare(
    {"user_name": "Alice"},
    {"userName": "Alice"},
).similarity_score  # 1.0

Quick similarity score

from json_semantic_diff import similarity_score

# Pure naming-convention difference -> identical after normalisation.
score = similarity_score(
    {"first_name": "Bob", "last_name": "Smith"},
    {"firstName": "Bob", "lastName": "Smith"},
)
print(score)  # 1.0

Boolean equivalence check

from json_semantic_diff import is_equivalent

# Passes — naming convention difference only
is_equivalent({"user_name": "Alice"}, {"userName": "Alice"})  # True

# Fails — different structure entirely
is_equivalent({"name": "Alice"}, {"product": "Widget"})  # False

# Custom threshold
is_equivalent({"user_name": "Alice"}, {"userName": "Alice"}, threshold=0.99)

Explain why a score isn't 1.0

from json_semantic_diff import STEDConfig, compare, format_diff

config = STEDConfig(collect_explanation=True)
result = compare(
    {"user_name": "Alice", "age": 30, "debug": True},
    {"userName": "Alice", "age": 31},
    config=config,
)

# Programmatic access: per-path contributions, highest impact first
for c in result.explanation:
    print(c.path, c.kind, c.detail)
# /debug unmatched_left debug
# /age   value_mismatch 30 -> 31

# Or render the whole result as a report
print(format_diff(result))
# Similarity: 0.58  (computed in 0.2 ms)
#
# Matched (2):
#   age <-> age
#   user_name <-> userName
#
# Unmatched in left (1):
#   /debug
#
# Differences (2):
#   /debug: only in left
#   /age: 30 -> 31

Command line

The json-semantic-diff console script (also python -m json_semantic_diff) compares two files, or stdin via -:

json-semantic-diff left.json right.json          # prints the score
json-semantic-diff --explain left.json right.json  # human-readable report of what differs
json-semantic-diff --json left.json right.json   # full audit trail as JSON
cat left.json | json-semantic-diff - right.json

# CI gate: exit 0 when similarity >= threshold, 1 otherwise.
# Combine with --explain to see why a gate failed.
json-semantic-diff --threshold 0.95 --explain expected.json actual.json

Flags: --explain, --json, --threshold FLOAT, --verbose, --structural-weight, --content-weight, --array-mode {ordered,unordered,auto} (default ordered, matching the library default), --version.

Measure generator consistency

from json_semantic_diff import consistency_score

# Stable generator → 1.0
docs = [
    {"name": "Alice", "age": 30},
    {"name": "Alice", "age": 30},
    {"name": "Alice", "age": 30},
]
print(consistency_score(docs))  # 1.0

# Erratic generator → low score
erratic = [
    {"name": "Alice", "age": 30},
    {"fullName": "Alice", "years": 30},
    {"person": "Alice"},
]
print(consistency_score(erratic))  # ~0.32

API Reference

Public Functions

Function	Signature	Returns
compare(left, right, config=None)	Two JSON values + optional config	ComparisonResult
similarity_score(left, right, config=None)	Two JSON values + optional config	float in [0.0, 1.0]
is_equivalent(left, right, threshold=0.85, config=None)	Two JSON values + threshold + config	bool
consistency_score(docs, config=None)	List of JSON values + optional config	float in [0.0, 1.0]
compare_batch(lefts, right, config=None)	Many left values vs one right	list[ComparisonResult]
compare_batch_pairs(pairs, config=None)	List of (left, right) tuples	list[ComparisonResult]
format_diff(result, indent=2)	A ComparisonResult	Human-readable report str

ComparisonResult

Returned by compare(). Frozen dataclass with seven fields:

Field	Type	Description
similarity_score	float	Normalised similarity in [0.0, 1.0]. 1.0 = identical.
matched_pairs	tuple[tuple[str, str], ...]	JSON Pointer pairs for matched KEY nodes
key_mappings	dict[str, str]	Raw left key name → raw right key name
unmatched_left	tuple[str, ...]	JSON Pointer paths with no match in right document
unmatched_right	tuple[str, ...]	JSON Pointer paths with no match in left document
computation_time_ms	float	Wall-clock duration in milliseconds
explanation	tuple[NodeContribution, ...]	Per-path contributions, highest impact first. Populated when STEDConfig(collect_explanation=True); empty tuple otherwise.

Each NodeContribution has path (JSON Pointer), contribution (raw distance — higher means it moved the score more), kind (value_mismatch, key_rename, unmatched_left, unmatched_right, or matched for a container whose subtree differs), and detail (e.g. 30 -> 31 for a value change, email_address -> mailAddress for a rename).

STEDConfig

Immutable configuration for the algorithm. All parameters have sensible defaults.

from json_semantic_diff import STEDConfig, ArrayComparisonMode

config = STEDConfig(
    w_s=0.5,                                          # Structural weight [0, 1]
    w_c=0.5,                                          # Content weight [0, 1] (must sum to 1.0)
    lambda_unmatched=0.5,                              # Penalty for unmatched children (>= 0)
    array_comparison_mode=ArrayComparisonMode.ORDERED,  # ORDERED | UNORDERED | AUTO
    type_coercion=False,                               # "42" == 42?
    null_equals_missing=False,                         # {x: null} == {}?
    ignore_paths=("/timestamp", "/users/*/id"),        # Drop volatile keys before comparing
    numeric_tolerance=0.0,                             # |a - b| <= tol counts as equal
    aliases=(("uid", "user_id"),),                     # Domain synonyms match at 1.0
    max_depth=None,                                    # Cap traversal depth (None = full)
    collect_explanation=False,                         # Populate result.explanation
)

result = compare(doc1, doc2, config=config)

Parameter	Type	Default	Description
w_s	float	0.5	Structural weight. Higher → structure matters more.
w_c	float	0.5	Content weight. Higher → values matter more. Must sum to 1.0 with w_s.
lambda_unmatched	float	0.5	Penalty per unmatched child. 0.0 = ignore extras. 1.0 = full penalty.
array_comparison_mode	ArrayComparisonMode	ORDERED	How arrays are compared.
type_coercion	bool	False	When True, "42" and 42 compare as equal.
null_equals_missing	bool	False	When True, {"x": null} and {} compare as equal.
ignore_paths	tuple[str, ...]	()	JSON Pointer patterns stripped from both inputs before comparison (timestamps, generated ids). * matches one path component: /users/*/id.
numeric_tolerance	float	0.0	Absolute tolerance for numeric comparison. abs(a - b) <= tol counts as equal (bools excluded).
aliases	tuple[tuple[str, str], ...]	()	Key pairs forced to similarity 1.0, symmetric. Applied to raw and normalised labels.
max_depth	int | None	None	Cap traversal depth. Subtrees past the cap contribute full cost unless identical. Faster on deep documents, less resolution past the cap.
collect_explanation	bool	False	When True, result.explanation carries the per-path breakdown of what drove the score. Off by default — no overhead on the hot path.

Array Comparison Modes

Mode	Strategy	Use case
ORDERED	Positional DP alignment	Sequences, logs, ordered lists
UNORDERED	Hungarian matching	Tags, feature flags, sets
AUTO	Infer from content (scalars → unordered, objects → ordered)	When array semantics vary

Backends

Three embedding backends for key matching, each optimising for different constraints:

Backend	Dependencies	Latency	Key matching quality	Cost
StaticBackend (default)	None	~0.1ms	Naming conventions	Free
FastEmbedBackend	fastembed	~1-2s cold start	Semantic understanding	Free
OpenAIBackend	openai, tenacity	~100-500ms	Best quality	API costs

StaticBackend (default)

Uses Levenshtein edit distance on normalised keys. No ML model, no API calls.

from json_semantic_diff.backends import StaticBackend

backend = StaticBackend()
backend.similarity("user_name", "userName")   # 1.0
backend.similarity("user_name", "address")    # < 0.5

FastEmbedBackend

Local ONNX embeddings via sentence-transformers/all-MiniLM-L6-v2 (384-dim).

from json_semantic_diff.backends.fastembed import FastEmbedBackend
from json_semantic_diff.comparator import STEDComparator

backend = FastEmbedBackend()  # ~1-2s cold start for model loading
cmp = STEDComparator(backend=backend)
result = cmp.compare(doc1, doc2)

OpenAIBackend

Cloud embeddings via text-embedding-3-small (1536-dim). API key from OPENAI_API_KEY environment variable.

from json_semantic_diff.backends.openai import OpenAIBackend
from json_semantic_diff.comparator import STEDComparator

backend = OpenAIBackend()  # Reads OPENAI_API_KEY from env
cmp = STEDComparator(backend=backend)
result = cmp.compare(doc1, doc2)

Auto-retries rate-limited requests with jittered exponential backoff (up to 6 attempts via tenacity).

Using STEDComparator directly

For batch comparisons where you want cache reuse across calls:

from json_semantic_diff.comparator import STEDComparator

cmp = STEDComparator()  # Or with backend= and config=
for doc in documents:
    result = cmp.compare(doc, reference)
    # Embedding cache reused across all calls

Integrations

pytest

Auto-discovered fixture — install the package and it's available in every test:

def test_api_response(assert_json_equivalent):
    actual = get_user_from_api()
    expected = {"user_name": "Alice", "age": 30}

    # Passes even if API returns {"userName": "Alice", "age": 30}
    assert_json_equivalent(actual, expected)

Custom threshold and config:

from json_semantic_diff import STEDConfig

def test_strict(assert_json_equivalent):
    config = STEDConfig(type_coercion=True)
    assert_json_equivalent(actual, expected, threshold=0.95, config=config)

Failure messages include full context:

AssertionError: JSON documents not equivalent:
  similarity=0.42 < threshold=0.85
  actual:   {"product": "Widget", "price": 19.99}
  expected: {"user_name": "Alice", "age": 30}
  key_mappings: {"product": "user_name"}
  unmatched_left: ["/price"]
  unmatched_right: ["/age"]

LangSmith

from json_semantic_diff.comparator import STEDComparator
from json_semantic_diff.integrations import LangSmithEvaluator

comparator = STEDComparator()
evaluator = LangSmithEvaluator(comparator, output_key="response")
# Pass directly to langsmith.evaluate()

Braintrust

from json_semantic_diff.comparator import STEDComparator
from json_semantic_diff.integrations import BraintrustScorer

comparator = STEDComparator()
scorer = BraintrustScorer(comparator)
# Returns float in [0.0, 1.0] or None when no expected value

W&B Weave

from json_semantic_diff.comparator import STEDComparator
from json_semantic_diff.integrations import WeaveScorer

comparator = STEDComparator()
scorer = WeaveScorer(comparator)
# Returns {"semantic_similarity": float}

How It Works

The STED Algorithm

1. JSON → Typed Tree. Each JSON value becomes a tree node with a type (OBJECT, KEY, ARRAY, ELEMENT, SCALAR). KEY nodes store both raw and normalised labels.

2. Key Normalisation. A five-pass regex pipeline converts all naming conventions to a canonical form: userName → "user name", user_name → "user name", UserName → "user name", user-name → "user name".

3. Hungarian Key Matching. For each pair of OBJECT nodes, a cost matrix is built using key similarity scores. The Hungarian algorithm finds the optimal key-to-key assignment.

4. Blended Cost. Each node comparison blends structural and content distance: cost = w_s * structural_distance + w_c * content_distance.

5. Per-Level Normalisation (Zhang-Shasha). At each structural tree level: similarity = 1.0 - min(1.0, [d_matched + λ·|n_left - n_right|] / max(n_left, n_right, 1)), where n_left / n_right are the sum of children subtree sizes on each side (not just the child count). This keeps scores in [0.0, 1.0] regardless of document depth and prevents deep value-subtree edits from binary-collapsing the score when the parent has few direct children.

Consistency Score Formula

pairwise_scores = [compare(docs[i], docs[j]).similarity_score for all unique (i, j) pairs]
consistency = max(0, mean(pairwise_scores) - std(pairwise_scores))

Penalises both low average similarity (different outputs) and high variance (erratic outputs).

Embedding Cache

All unique KEY labels are collected before the algorithm runs and embedded in a single batch call. The algorithm then runs entirely from cache — zero additional backend calls per comparison.

Each STEDComparator instance owns its own cache (no global state). The public API functions create a fresh comparator per call for isolation. Use STEDComparator directly for batch comparisons where cache reuse matters.

Development

git clone https://github.com/mokhld/json-semantic-diff.git
cd json-semantic-diff
poetry install --with dev

Running tests

poetry run pytest                    # All 661 tests
poetry run pytest -x                 # Stop on first failure
poetry run pytest tests/unit/        # Unit tests only
poetry run pytest tests/algorithm/   # Algorithm tests only

Type checking

poetry run mypy src/          # Strict mode

Linting

poetry run ruff check src/ tests/
poetry run ruff format src/ tests/

Project structure

src/json_semantic_diff/
├── __init__.py              # Public API exports
├── api.py                   # 4 public functions
├── comparator.py            # STEDComparator orchestrator
├── scorer.py                # ConsistencyScorer
├── result.py                # ComparisonResult dataclass
├── protocols.py             # EmbeddingBackend Protocol
├── cache.py                 # EmbeddingCache (LRU)
├── algorithm/
│   ├── config.py            # STEDConfig + ArrayComparisonMode
│   ├── sted.py              # STEDAlgorithm (recursive core)
│   ├── costs.py             # Insert/delete/update cost functions
│   ├── matcher.py           # Hungarian algorithm wrapper
│   └── normalizer.py        # Per-level similarity normalisation
├── backends/
│   ├── static.py            # Levenshtein (zero dependencies)
│   ├── fastembed.py         # ONNX (local embeddings)
│   └── openai.py            # OpenAI (cloud embeddings)
├── tree/
│   ├── nodes.py             # TreeNode + NodeType
│   ├── builder.py           # JSON → Tree conversion
│   └── normalizer.py        # Key normalisation pipeline
└── integrations/
    ├── _pytest_plugin.py    # pytest fixture (auto-discovered)
    ├── _langsmith.py        # LangSmith evaluator adapter
    ├── _braintrust.py       # Braintrust scorer adapter
    └── _weave.py            # W&B Weave scorer adapter

License

MIT