What is ContextOps?

ContextOps is a deterministic, embedding-free linter for the context you send to an LLM. Feed it the exact payload you're about to hand to the model (an OpenAI message list, a RAG bundle, or a plain string) and it returns, before inference:

• a Context Health Score (CHS) from 0–100, computed from four structural penalties
• a per-dimension breakdown explaining why the score is what it is
• findings that point at the specific offending chunks
• actionable recommendations with estimated token savings

No LLM calls. No embeddings. No API keys. No network. The same input always produces the same output, which is what makes it safe to wire into CI.

Think of it as ESLint for LLM context.

The one-line mental model

Score = 100 − (Redundancy + Density + Structure + Concentration)

Each penalty is capped, so no single problem can tank the score on its own:

Dimension	Max Penalty	What It Detects
Redundancy	30 pts	Lexically duplicated or near-duplicated context items
Density	30 pts	Token waste from formatting, whitespace, and repetitive boilerplate
Structure	20 pts	Imbalance between context types (retrieval flooding, system-prompt bloat, memory explosion, tool sprawl)
Concentration	20 pts	Over-reliance on a single document or source

What you actually get

contextops inspect prints a real, readable report, not a wall of numbers:

+==================================================+
|          CONTEXTOPS | Context Analysis            |
+==================================================+

  CI Gate Status:  WARN

  Context Score:  62 / 100  (NEEDS WORK)

  Score Breakdown:

    Redundancy     -18.5 / 30  ############
    Density        -4.2  / 30  ##
    Structure      -3.3  / 20  ###
    Concentration  -12.0 / 20  ########

    Total Penalty  -38.0 / 100

  Token Breakdown:
    Total tokens:        2,150
    Wasted tokens:       842
    Potential reduction: 39.2%

  Findings:
    [!]  REDUNDANT_CONTEXT (Likely): refund_policy.md chunks are ~87% similar
    [S]  Retrieval dominance: retrieval is 78% of context

  Recommendations:
    1. Redundant context: refund_policy.md appears twice
       Impact: +15.2 points | Save: 342 tokens
       Fix: Remove the duplicate item → save 342 tokens

(Add --roast for brutally honest commentary, or --json-output for CI parsing.)

See it in action

Why ContextOps exists

Traditional software engineering has deterministic quality gates: compilers catch syntax errors, linters catch code smells, formatters enforce consistency. LLM applications rarely have an equivalent layer for the context they stuff into a model.

Instead, prompts quietly grow over time with:

• Duplicated retrieval chunks
• Bloated system prompts
• Runaway conversation history
• Excessive tool output
• Hidden token waste

These increase latency and cost, make model behavior less predictable, and often go unnoticed until production. ContextOps makes context quality observable, measurable, and testable before inference.

Without ContextOps	With ContextOps
Silent context degradation	Observable context health
Wasted tokens = wasted $	Token optimization guidance
Unpredictable model behavior	Deterministic quality gates
No CI integration for context	contextops check in GitHub Actions
Guesswork debugging	Root-cause diagnostics
No historical tracking	Local telemetry + trends

Who it's for

The one-line test: do you write the code that assembles the messages array sent to an LLM API? If yes, this is for you.

Audience	How you'd use it
LLM app developers (RAG, agents, chatbots)	pip install contextops → analyze your pipeline's payload before the API call
LangChain / LlamaIndex / AutoGen users	Attach the callback, just 2 lines, see below
Prompt engineers	A/B test template variations with contextops diff
MLOps & Platform Teams	contextops check as a CI/CD quality gate
Researchers	Deterministic scoring + ContextBench

Who it's not for (currently)

ContextOps analyzes the payload you assemble. If a closed tool assembles the context internally, there's nothing to intercept:

User type	Why not
ChatGPT / Claude.ai web users	No API access to the payload
Claude Code / Codex users	Closed binary, no context export
Cursor / Windsurf end-users	No native payload export

Analogy: ESLint is useless to people browsing websites. It's for people writing JavaScript. ContextOps is for people writing LLM calls.

When to use it

When	What to run	Why
During development	contextops inspect context.json	Catch bloat and duplication before you ship
Before merging / in CI	contextops check context.json --min-score 75	Exit-code gate that blocks context regressions like a linter
Debugging production	Export a bad trace from LangSmith/Helicone → inspect it	Root-cause weird model behavior structurally
Comparing changes	contextops diff before.json after.json	Prove a prompt/retrieval refactor actually improved the context
Ongoing	contextops telemetry trends	Track context health over time

How it works

ContextOps sits between your prompt/retrieval logic and the LLM API call. It statically analyzes the payload and scores its structural health.

1. Normalize: accept OpenAI message lists, structured dicts (system / messages / chunks / memory / tools), or a plain string.
2. Tokenize: count tokens with tiktoken using the target model's encoding.
3. Analyze: run four independent analyzers: redundancy, density, structure, and concentration.
4. Score: combine the four capped penalties into a single 0–100 score.
5. Recommend & gate: emit actionable fixes, and (in check mode) pass or fail with an exit code.

Retrieval / RAG ─┐
Conversation ────┼─► Context Payload ─► ContextOps ─► Score + Findings
Tools output ────┘                          │
                                            ├─► Pass ─► LLM Inference
                                            └─► Fail ─► CI block / alert

What ContextOps does not do

It's a structural analyzer, by design. It intentionally does not evaluate:

• Prompt-engineering quality or intent
• Reasoning ability or hallucinations
• Factual correctness
• Retrieval relevance (semantic meaning: same idea, different words)

A high score means your context is structurally clean, not that the model will answer correctly.

Quick start

pip install contextops

1. See it in action (zero setup)

contextops demo

Runs a pre-packaged, deliberately messy RAG context so you can see the full report and the roast.

2. Capture your context

Save the payload right before your API call:

import json
import openai

messages = [
    {"role": "system", "content": "You are a helpful customer support bot."},
    {"role": "user", "content": "How long will my refund take?"},
]

# Add two lines before your API call
with open("context.json", "w") as f:
    json.dump(messages, f, indent=2)

response = openai.chat.completions.create(model="gpt-4o", messages=messages)

You can also export a bad trace from LangSmith / Helicone / Datadog and point ContextOps at the saved JSON.

3. Inspect it

contextops inspect context.json

(Use --roast for brutally honest diagnostics, --explain to see the top score drivers.)

4. Enforce it in CI

contextops check context.json --min-score 75

Exits 0 on pass, 1 on fail. Drop it straight into GitHub Actions (see .github/workflows/contextops.yml for the pattern used by this repo).

CLI reference

Command	What it does
contextops demo	Run the built-in demo context (roast on by default)
contextops inspect <file>	Analyze a context file and print a rich report
contextops check <file> --min-score N	CI gate — exit 0 (pass) / 1 (fail) / 2 (bad input)
contextops diff <a> <b>	Compare two snapshots to detect regressions
contextops stability [file]	Verify the determinism guarantee
contextops telemetry status · log · trends	Local-only score history and trends
contextops badge [--score N]	Generate a shields.io badge for your score

Common flags on inspect / check:

• --json-output — machine-readable JSON
• --model <name> — model for token counting (default gpt-4o)
• --profile <name> — archetype: general, rag, agent, chatbot, toolchain
• --explain — show why each penalty fired
• --roast — score-band commentary
• --config <path> — JSON config with custom thresholds

For the full flag reference, see USER_GUIDE.md.

Python API

from contextops.api.inspect import inspect_context

result = inspect_context(
    payload,          # dict, list, or plain string
    model="gpt-4o",   # model for token counting
    archetype="rag",  # archetype profile (optional)
)

print(f"Score: {result.score} / 100")
print(f"Wasted tokens: {result.token_breakdown.wasted_tokens}")
for rec in result.recommendations:
    print(f"  -> {rec.fix}")

Also available: diff_contexts(a, b) and run_stability_report(payload) from contextops.api, and ContextOpsConfig for custom thresholds.

LangChain integration

from contextops import ContextOps

# Log the score (default — non-blocking)
chain = chain.with_config({"callbacks": [ContextOps.auto()]})

# Block execution if context quality is too low
chain = chain.with_config(
    ContextOps.langchain_config(mode="block", min_score=75, profile="rag")
)

ContextOps.auto() returns a LangChain callback handler with three modes:

Mode	Behavior
log	Print the score report (default)
warn	Emit a warning below min_score
block	Raise ContextOpsScoreError below min_score — the LLM call never happens

A full end-to-end RAG demo lives in examples/quickstart_rag.py.

Archetype profiles

Archetypes adjust structural thresholds for your use case. The global 0–100 score is never affected — only which warnings fire.

Profile	When to use	Retrieval	System	Memory	Tool
general	Default — mixed use cases	70%	50%	50%	60%
rag	Pure document retrieval	95%	40%	20%	30%
agent	Autonomous agents with tool loops	50%	40%	40%	90%
chatbot	Conversational apps with large history	40%	50%	85%	30%
toolchain	Multi-tool pipelines	50%	40%	30%	95%

---

ContextBench: the benchmark

ContextBench is a 1,500-sample benchmark for evaluating the structural integrity of context windows, the first benchmark of its kind.

Why it exists

Every existing LLM benchmark measures what the model does: answer correctness, reasoning, hallucination rates, retrieval accuracy. None of them measure what the model is given. Yet modern LLM systems increasingly fail before inference even begins: 50 redundant documents retrieved when 5 would do, multi-agent pipelines passing bloated internal state, uncompressed tool output consuming thousands of wasted tokens, stale memory quietly corrupting a long-running session.

ContextBench evaluates that layer. The question is not "did the model answer correctly?" but "was the context window constructed efficiently and robustly before inference?"

Evaluator-agnostic by design

This is the critical design decision: ContextBench is not coupled to ContextOps. Each sample carries ground-truth expected_properties, for example:

"ground_truth": {
  "failure_modes": ["system_prompt_bloat"],
  "expected_properties": {
    "contains_redundancy": false,
    "contains_density_bloat": true,
    "contains_structure_imbalance": true,
    "contains_source_concentration": false
  }
}

Any evaluator can be scored against these labels objectively: flagging redundancy where contains_redundancy is false is a false positive, missing a declared failure mode is a false negative. That makes the benchmark fair ground for static analyzers, heuristic scorers, observability platforms, LLM-as-a-judge pipelines, and academic frameworks alike, with no built-in bias toward ContextOps' own scoring engine. ContextOps is validated against ContextBench, but the benchmark stands on its own: anyone can build a competing analyzer and evaluate it on the same dataset.

What's in v1

Five categories, 300 samples each, built on domain-grounded enterprise data (legal contracts, stack traces, SQL dumps, clinical notes, Jira histories...) rather than synthetic filler:

Category	What it tests
Optimal Architectures	Healthy baselines, so evaluators must also avoid false positives, not just find broken contexts
Structural Failures	System-prompt bloat, retrieval flooding, memory explosion, tool-output overflow
Redundancy Failures	Near-duplicate clusters, boilerplate explosion, repeated memory loops
Agent Architecture Failures	Multi-agent context explosion, recursive planning loops, uncompressed state passthrough
Temporal Context Drift	Stale memory injection, retrieval drift, summary compression degradation

The adversarial companion, ContextSecBench, adds 9,500 attack payloads: prompt-injection hiding, truncation smuggling, semantic denial-of-service, and context poisoning.

Reproducible and versioned

Samples are synthesized by a generation engine rather than hand-curated, and every sample records mutation metadata describing exactly how it was built. Released versions are immutable (ContextBench_v1, then v2, and so on), so published scores stay comparable over time.

Why it matters

A benchmark is what turns an idea into a discipline. Without one, "context optimization" is anecdote; with one, every optimizer, compressor, and analyzer becomes comparable on the same ground, regressions become detectable, and research becomes reproducible. As agent systems grow persistent and long-running, context construction quality, not model size, is increasingly what determines system performance. ContextBench is built to be the evaluation standard for that layer.

View the Leaderboard →

Requirements & guarantees

• Python 3.10+, with exactly two runtime dependencies: tiktoken (token counting) and click (CLI).
• Deterministic: same input, same score, every time. Enforced by a chaos test suite on every commit.
• Fast: < 2s for 5k tokens, < 5s for 20k, < 10s for 50k.
• Local-only telemetry: opt in with CONTEXTOPS_TELEMETRY=1; scores are written to ~/.contextops/telemetry.jsonl and never leave your machine.

The formal versioning, schema, and determinism contract is in STABILITY.md.

Documentation

Document	Contents
HIGH_LEVEL_DOC.md	Full technical reference — scoring engine, all flags, API, ContextBench
USER_GUIDE.md	Getting-started guide with practical examples
STABILITY.md	Stability contract, versioning policy, schema guarantees
CONTRIBUTING.md	Setup, project structure, PR checklist

Contributing & license

Contributions welcome. Read CONTRIBUTING.md first, and open an issue before non-trivial changes.

• Website: contextops.vercel.app
• Repository: github.com/Abhijeet777ui/contextops
• License: Sustainable Use License