toolkit-data-contracts

Contracts for LLM-shaped data. A small, dependency-free Python CLI that checks chat transcripts, tool-call arguments, SFT and preference (DPO) rows and RAG chunk records against a JSON Schema contract, and flags drift at every nested path. Contracts interoperate with the Open Data Contract Standard (ODCS). It is meant as a cheap CI gate in front of training, evaluation or retrieval data.

toolkit-contracts check --input train.jsonl --preset openai-chat --tools tools.json

It reads JSONL (and, with the parquet extra, Parquet or CSV) in one streaming pass. If you need warehouse connectors or a full suite of statistical drift tests, look at larger projects such as datacontract-cli or Evidently.

Status

Install

Requires Python 3.10+. The core has no runtime dependencies. Optional extras: odcs (PyYAML, for ODCS YAML files), tiktoken (tiktoken token counts), lang (langdetect, for language drift), parquet (pyarrow, for Parquet and CSV input).

git clone https://github.com/AKIVA-AI/toolkit-data-contracts.git
cd toolkit-data-contracts
pip install .            # or ".[odcs,tiktoken,lang,parquet]"; pip install -e ".[dev]" for development
toolkit-contracts --help

Five-minute example

examples/data/ultrachat_200k_test_sft_40.jsonl holds 40 chat rows from the public UltraChat 200k dataset (MIT license; see examples/data/README.md). From the repository root:

# 1. Does the file have the shape trainers expect for SFT?
toolkit-contracts check --input examples/data/ultrachat_200k_test_sft_40.jsonl --preset sft --format table

Status: PASS

Validation: OK (no issues)

Drift: OK (no issues)

# 2. Freeze its shape as a contract (roles become an enum) and profile it as the baseline
toolkit-contracts infer --input examples/data/ultrachat_200k_test_sft_40.jsonl \
  --out chat.contract.json --enum-max 3
toolkit-contracts profile --input examples/data/ultrachat_200k_test_sft_40.jsonl \
  --contract chat.contract.json --out baseline.profile.json

# 3. Make a bad batch: assistant answers cut to 20 characters, a system turn added
python - <<'EOF'
import json
with open("examples/data/ultrachat_200k_test_sft_40.jsonl", encoding="utf-8") as src, \
        open("bad.jsonl", "w", encoding="utf-8") as out:
    for line in src:
        row = json.loads(line)
        for m in row["messages"]:
            if m["role"] == "assistant":
                m["content"] = m["content"][:20]
        row["messages"].insert(0, {"role": "system", "content": "You are terse."})
        out.write(json.dumps(row) + "\n")
EOF

# 4. Gate it (exit code 4)
toolkit-contracts check --input bad.jsonl --contract chat.contract.json \
  --baseline baseline.profile.json --format table

Status: FAIL

Validation Issues:
  Kind                      Field                 Count  Message
  ------------------------- -------------------- ------  ------------------------------
  enum_mismatch             messages[].role          40  enum

Drift Issues:
  Kind                      Field                 Count  Message
  ------------------------- -------------------- ------  ------------------------------
  drift_text_length         messages[].content        1  psi=5.707 exceeds 0.250 (mean 902.6 -> 154.5)
  drift_token_count         messages[].content        1  psi=5.642 exceeds 0.250 (mean 144.3 -> 25.4)
  drift_categorical         messages[].role           1  psi=0.941 exceeds 0.250

# 5. Share the contract as an Open Data Contract Standard document (needs the odcs extra)
toolkit-contracts odcs export --contract chat.contract.json --out chat.odcs.yaml --name ultrachat_sft

tests/test_readme_walkthrough.py runs these steps in CI.

Usage

# 1. Infer a contract from known-good records (reads the whole file; --limit N to cap)
toolkit-contracts infer --input samples.jsonl --out contract.json

# 2. Profile known-good records as the drift baseline
toolkit-contracts profile --input baseline.jsonl --contract contract.json --out baseline.profile.json

# 3. Validate a new batch, and drift-check it against the baseline
toolkit-contracts check --input new_batch.jsonl --contract contract.json \
  --baseline baseline.profile.json

check prints a JSON report, or writes it with --out FILE. Other formats: --format table and --format markdown (for a PR comment) are for people; --format json-legacy prints the pre-1.0 {ok, validation_issues, drift_issues} object and will be removed in the next minor version.

Report format

The JSON report is the shared toolkit report envelope v1: an in-toto Statement v1 in canonical JSON (sorted keys, no extra whitespace, trailing newline), so its SHA-256 is stable and it can be signed. See docs/report-envelope.md and the JSON Schema schemas/report-envelope.v1.json.

• subject: the checked batch file, with its SHA-256.

• predicate.kind: data.check.

• predicate.inputs: the contract (or preset:NAME), baseline profile and --tools files, with their SHA-256.

• predicate.verdict / exit_code: pass/0, fail/4, or error/2 (the input was readable but could not be judged, for example a malformed line). If the input file itself cannot be read, no report is written.

• predicate.summary (what a CI gate reads):

  Key	Meaning
  ok	true only when there are no validation or drift issues
  records	records read from the batch
  validation_issues	distinct (kind, field, message) validation issues
  invalid_occurrences	total occurrences across those issues
  drift_checked	whether --baseline was given
  drift_issues	number of drift issues
  invalid_records	records with at least one validation issue
  tool_calls_checked, tool_calls_unchecked	with the openai-chat, anthropic-messages and tool-calls presets

  diff reports use kind: data.diff with summary keys mode, changes, breaking, non_breaking and ok, and a details.changes list.

• predicate.details: validation_issues and drift_issues lists (kind, field, message, count), the drift thresholds used, and the preset name.

Set SOURCE_DATE_EPOCH to fix created_at and make a report byte-for-byte reproducible. Signing is optional and not built in; for example, with toolkit-ml-provenance installed:

toolkit-mlsbom sign-file report.json    # and later: toolkit-mlsbom verify-file report.json

Useful check options:

Global options: --verbose, --log-format json, --version.

Text and embedding drift

profile records text statistics for every string path automatically. The tokenizer, the paths with language detection and the embedding paths are stored in the profile's config, and check profiles the new batch the same way, so they are chosen once:

toolkit-contracts profile --input base.jsonl --preset rag-chunks --out base.profile.json   --tokenizer tiktoken:cl100k_base --language text --embedding embedding
toolkit-contracts check --input new.jsonl --preset rag-chunks --baseline base.profile.json

Comparing profiles made with different tokenizers is an error (exit 2).

LLM presets

Built-in contracts for common LLM data shapes. Use --preset NAME instead of --contract with check and profile; toolkit-contracts presets list lists them and presets show NAME prints the JSON Schema so you can copy and tighten it.

--tools FILE takes OpenAI tools ([{"type": "function", "function": {"name", "parameters"}}]), legacy OpenAI functions or Anthropic tools ([{"name", "input_schema"}]), as a list or as {"tools": [...]}. Tools inside a row take precedence. Without any tool definitions, tool calls are counted as tool_calls_unchecked in the report summary instead of failing. Tool arguments use strict JSON Schema integer rules (2.0 is an integer, 2.5 is not). Issue paths look like tool[get_weather].location. The duplicate-id check keeps every chunk id in memory.

Contracts

infer --enum-max 2 on a chat file writes a contract like this (the real file has its keys sorted):

{
  "version": 2,
  "schema": {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
      "id": {"type": "string"},
      "messages": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "role": {"type": "string", "enum": ["assistant", "user"]},
            "content": {"type": ["null", "string"]}
          },
          "required": ["content", "role"],
          "additionalProperties": true
        }
      }
    },
    "required": ["id", "messages"],
    "additionalProperties": true
  }
}

Edit it by hand to tighten it (pattern, minimum/maximum, maxLength, minItems, additionalProperties: false, and so on), or pass any JSON Schema file that stays within the supported subset as --contract. Issue kinds: missing_required, type_mismatch, unexpected_field, enum_mismatch, const_mismatch, range_violation, length_violation, pattern_mismatch, unique_violation, contains_violation, unexpected_item, invalid_property_name, combinator_mismatch, false_schema. When no branch of an anyOf/oneOf matches, the issues of the closest branch are reported. pattern uses Python regular expressions, which differ from ECMA-262 in rare cases (for example \p{...} is rejected).

Input must be strict JSON: NaN, Infinity and numbers that overflow are rejected with exit code 2. Profiles written by older versions still work; regenerate the baseline to get nested paths and categorical counts.

ODCS (Open Data Contract Standard)

ODCS is the Bitol (Linux Foundation) standard for data contracts, also used by tools such as datacontract-cli. Convert in either direction:

pip install ".[odcs]"    # PyYAML; .json output and input work without it
toolkit-contracts odcs export --contract contract.json --out contract.odcs.yaml --name chat_sft
toolkit-contracts odcs import --input contract.odcs.yaml --out contract.json [--object NAME]

How the mapping works:

ODCS has no presence flag for top-level columns, so a top-level key exports as required: true only when it is both required and non-null. Constructs ODCS cannot express (anyOf, additionalProperties: false, $ref, a required-but-nullable top-level key, and so on) are listed as a warning, and the exact JSON Schema is embedded in the schema object as the custom property jsonSchema. odcs import uses that exact schema only while the ODCS fields still describe it; if someone edited the fields, it uses the fields and warns. Quality rules other than a strict valid-values list are reported, not enforced.

Library use

from toolkit_data_contracts_drift.contract import (
    drift_check, infer_contract, profile_records, validate_records,
)

contract = infer_contract(training_rows)
issues = validate_records(contract=contract, records=new_rows)
drift = drift_check(
    baseline=profile_records(contract=contract, records=training_rows),
    current=profile_records(contract=contract, records=new_rows),
)

See examples/ml_pipeline_example.py for a runnable walkthrough.

Exit codes

Reviewing contract changes

diff compares two contracts and classifies every change:

--mode backward (default, as in schema registries) treats tightens and both as breaking; forward treats loosens and both as breaking; full treats all three as breaking. The exit code is 4 when anything is breaking.

toolkit-contracts diff --old main-contract.json --new contract.json --format table

Formats: json (report envelope, kind: data.diff), table, markdown (for a PR comment) and github (workflow annotations).

GitHub Action

on: pull_request
jobs:
  contract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0          # the action reads the base version with git show
      - uses: AKIVA-AI/toolkit-data-contracts@main   # pin a release tag once one exists
        with:
          contract: contracts/chat.contract.json
          mode: backward

The action writes a Markdown summary to the job page, annotates breaking changes on the PR, fails the step when a change is breaking, and exposes the JSON report path as the report output. Pass old and new instead of contract to compare two files directly.

CI example

- name: Data contract gate
  run: |
    pip install git+https://github.com/AKIVA-AI/toolkit-data-contracts.git
    toolkit-contracts check --input data/batch.jsonl \
      --contract contracts/contract.json --baseline profiles/baseline.profile.json

Docker

docker build -t toolkit-data-contracts .
docker run --rm -v "$PWD/data:/app/data" toolkit-data-contracts \
  toolkit-contracts check --input /app/data/batch.jsonl --contract /app/data/contract.json

docker-compose.yml has example infer and check services under the tools profile.

Development

See CONTRIBUTING.md. In short: pytest, ruff check src/ tests/, pyright.

Contributing and security

Contributions are welcome: see CONTRIBUTING.md and the Code of Conduct. Please report security problems privately, as described in SECURITY.md.

Releasing

Releases are cut by pushing a vX.Y.Z tag. CI runs the tests, builds the sdist and wheel, checks them, attaches them to a GitHub Release and publishes them to PyPI with Trusted Publishing. RELEASING.md describes the process and how to verify a release.

License

Apache License 2.0. See LICENSE and NOTICE. Releases before the relicensing remain available under the MIT License.