Semantic JSON Transport

Find locations cheaply. Compose query-conditioned evidence. Transport the original source.

Semantic JSON Transport is an experimental RAG/evidence-construction project for long documents. Its core premise is that the indexed unit does not have to be the final context unit.

Documents are stored as fine-grained, source-grounded SemanticUnits. A query first locates promising positions, then evidence is composed from the original source while preserving provenance.

Install

pip install semantic-json-transport

Current package version: 0.2.0a3.

The PyPI runtime remains the stable v0.2 SemanticRepository / Region Compatibility API. The newer scalar-potential Local Composer and neural Global Evidence-Set Value/Search stack described below is a research-preview architecture under benchmark validation, not yet claimed as the default packaged inference runtime.

Runtime quick start

from semantic_json import SemanticRepository

text = """
B기업은 주요 거래처와 공급계약을 체결하고 있다.
해당 계약은 내년 말 만료될 예정이다.
해당 거래처 매출 의존도는 높은 수준이다.
중장기 상환능력을 낙관하기 어렵다.
"""

repo = SemanticRepository()
repo.add_text(
    text,
    document_id="company_b",
    source_uri="documents/company_b.txt",
)

result = repo.search("B기업의 중장기 상환능력", top_k=5)

print(result.to_json())
print(result.to_text())

Lite fallback:

from semantic_json import LiteEmbedder, SemanticRepository

repo = SemanticRepository(
    embedder=LiteEmbedder(),
    region_model="lite",
)

Current research architecture

The research line has converged on a three-stage semantic architecture plus deployment indexing:

Offline indexing / hierarchy / reusable embeddings
                    ↓
              Navigation
       where should we look?
                    ↓
       Local EvidenceRegion Composer
          Φθ(Q, R, ∂R)
                    ↓
          source-contiguous basins
             R1*, R2*, ... Rm*
                    ↓
        Global Evidence-Set Value
              Vω(Q, E)
                    +
           Structured Search
                    ↓
             Evidence Set E*
                    ↓
              downstream LLM

Local composition

A Local Region is a source-contiguous interval R=[i,j]. The research Local Composer learns a boundary-aware scalar potential:

Φθ(Q, R, ∂R)

Neighboring boundary edits are compared through potential differences rather than independent edit heads. This replaced an earlier action-field formulation that produced cyclic trajectories.

Global composition

Local potential values are intentionally not treated as globally calibrated relevance scores. Distant Local maxima are handled by a separate Evidence-Set Value:

Vω(Q, E)

The current research implementation uses a unary relevance foundation plus a learned set-interaction residual and combines the learned value with structured beam search.

Scaling principle

Large-document deployment is designed as coarse-to-fine reduction:

N primitive units
      ↓
approximate / hierarchical navigation
      ↓
k candidate anchors
      ↓
Local potential trajectories
      ↓
m unique Region basins
      ↓
Global value-guided search

Reusable representation work belongs at ingestion time. Expensive neural evaluation should operate only on the reduced query-time frontier.

Benchmark status

The research code is evaluated on QASPER with two deliberately separated metric families:

1. Internal diagnostics — source-unit Evidence F1, Boundary IoU, Oracle candidate-space F1, and selection efficiency.
2. QASPER-compatible text-evidence evaluation — paragraph Evidence F1 for the text evidence represented by the current source-unit pipeline.

We do not combine MSE, correlation, pair accuracy, or other diagnostics into a synthetic checkpoint score. Global checkpoints are selected by the actual deployed validation search Evidence F1.

Fixed-RAG comparison

The phase-1 benchmark is designed around the practical baseline the project aims to improve:

Fixed token chunks
      ↓
vector retrieval
      ↓
Top-k context

Fixed chunk length, overlap, and retrieval depth are tuned on validation. The v0.16 held-out experiment showed higher project text-evidence paragraph F1 for query-conditioned Evidence Construction than for the validation-tuned fixed-chunk + exact-cosine vector-retrieval baseline under the same maximum context-token budget.

The v0.17 closing benchmark strengthens this test with:

• a validation-tuned Fixed RAG + neural reranker baseline;
• actual-token quality Pareto analysis;
• paired bootstrap 95% confidence intervals;
• no arbitrary composite performance score.

Until the complete official QASPER task protocol, including non-text evidence conditions, is reproduced, published QASPER leaderboard scores are treated as orientation points rather than apples-to-apples claims.

See docs/research_architecture_v0.17.md for the current research formulation and benchmark rules.

SemanticUnit and source provenance

SemanticUnit is a fine-grained, contiguous source span. It is a retrieval/composition primitive, not a rewritten summary.

Every returned EvidenceRegion retains source coordinates. The source remains authoritative:

region = result[0]
repo.locate(region)
repo.get_source(region)
repo.verify_source(region)

Core invariant:

document.text[region.start_char:region.end_char] == region.text

Structured transport

SearchResult is canonically serializable as versioned JSON while plain text remains available for inspection.

result.to_dict()
result.to_json()
result.to_text()

The structured output retains document identity, Region identity, anchors, source coordinates, original text, and boundary traces.

Training utilities

The package includes dependency-light utilities for Region Compatibility training-data generation:

from semantic_json import LLMRegionTeacher, RegionDatasetBuilder, RegionQuery

teacher = LLMRegionTeacher(my_judge_callable, name="internal-llm")
builder = RegionDatasetBuilder(teacher, random_seed=13)

Fine-tuning tools remain optional:

pip install "semantic-json-transport[training]"

LLM teachers are offline labeling tools and are not mandatory production dependencies.

Research principles

The project currently follows these rules:

• keep original-source provenance intact;
• separate Navigation, Local Composition, and Global Selection;
• use diagnostics to understand models, but select checkpoints using deployed-task performance;
• compare against validation-tuned conventional RAG rather than weak straw-man baselines;
• report evidence quality and compute/context cost as separate axes instead of inventing a weighted aggregate score;
• treat negative benchmark results as evidence against the hypothesis rather than adding complexity solely to rescue it.

Status

semantic-json-transport is alpha software. APIs and research formulations may change. The v0.2 runtime remains usable for source-grounded retrieval experiments; the v0.17 research architecture should be treated as experimental until it is converted into a stable package API and reproduced across additional datasets.

Apache-2.0 license.