contextops 0.3.3

Deterministic context linter for LLM applications — analyze, score, and optimize your LLM context payloads.

ContextOps

Static analysis for LLM context.

ContextOps is a deterministic, model-independent context linter for LLM applications.

It analyzes the context sent to an LLM before inference and detects structural problems such as redundancy, token waste, context imbalance, and source concentration. It produces a reproducible Context Health Score (CHS) together with actionable diagnostics — without embeddings, model calls, or external services.

Think of ContextOps as ESLint for LLM context.

---

Why ContextOps?

Modern software engineering has deterministic quality gates.

• Compilers catch syntax errors.
• Linters catch code smells.
• Formatters enforce consistency.
• Static analyzers detect architectural problems.

LLM applications rarely have an equivalent layer for the context they send into models.

Instead, prompts quietly grow over time:

• duplicated retrieval chunks
• bloated system prompts
• runaway conversation history
• excessive tool output
• hidden token waste

These issues increase latency and cost, make model behavior less predictable, and often go unnoticed until production.

ContextOps makes context quality observable, measurable, and testable before inference.

---

What ContextOps Measures

ContextOps evaluates four structural dimensions.

Dimension	Max Penalty	What It Measures
Redundancy	30 pts	Lexical duplication across context items
Density	30 pts	Token waste from formatting and structural bloat
Structure	20 pts	Distribution imbalance between context components
Concentration	20 pts	Over-reliance on a single document or source

The result is a deterministic 0–100 Context Health Score together with detailed findings and suggested fixes.

---

What ContextOps Does Not Do

ContextOps intentionally does not evaluate:

• prompt engineering quality
• reasoning ability or hallucinations
• factual correctness
• retrieval relevance
• LLM outputs

It focuses exclusively on the structural quality of the context before inference.

---

Example

contextops inspect context.json

Context Health Score: 81 / 100

✓ Low structural complexity
✓ Good source diversity

Warnings

• 214 duplicated tokens detected
• Retrieval occupies 78% of context
• Two retrieval chunks are near duplicates

Estimated token savings: 12%

Use --roast mode for more candid diagnostics:

contextops inspect context.json --roast

---

Installation

pip install contextops

Run the interactive demo:

contextops demo

Dependencies: Only tiktoken and click. No GPU, no network, no external API keys required.

---

Quick Start

1. Capture your context

Save your LLM payload before sending it to the model.

import json

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": user_query},
]

# Add two lines before your API call
with open("context.json", "w") as f:
    json.dump(messages, f, indent=2)

response = openai.chat.completions.create(model="gpt-4o", messages=messages)

Or use the structured dict format for richer analysis:

{
  "system": "You are a helpful customer support bot.",
  "messages": [
    {"role": "user", "content": "How long will my refund take?"}
  ],
  "chunks": [
    {"content": "Refunds take 3-5 business days.", "source": "docs/refunds.md"}
  ],
  "memory": ["The user asked about a refund yesterday."],
  "tools": [{"name": "search_api", "output": "Tool response text here"}]
}

2. Inspect it

contextops inspect context.json

3. Enforce quality in CI

contextops check context.json --min-score 75

---

CLI Reference

contextops inspect <file>

Analyse a context file and display a rich report.

contextops inspect context.json
contextops inspect context.json --roast
contextops inspect context.json --profile rag
contextops inspect context.json --explain
contextops inspect context.json --json-output

Flag	Default	Description
--json-output	off	Output raw JSON instead of terminal format
--model <name>	gpt-4o	Model for token encoding (e.g. gpt-4, claude-3)
--profile <name>	general	Archetype: rag, agent, chatbot, toolchain
--config <path>	none	Path to a JSON config file with custom thresholds
--retrieval-max-ratio <f>	0.70	Override max allowed ratio for retrieval chunks
--system-max-ratio <f>	0.50	Override max allowed ratio for system prompt
--memory-max-ratio <f>	0.50	Override max allowed ratio for memory entries
--tool-max-ratio <f>	0.60	Override max allowed ratio for tool outputs
--explain	off	Show detailed "Top Score Drivers" for each penalty
--roast	off	Enable brutally honest score-band commentary

---

contextops check <file> --min-score N

CI gate. Exits with code 0 (pass) or 1 (fail).

contextops check context.json --min-score 75

Exit Code	Meaning
0	PASS — score meets or exceeds threshold
1	FAIL — score is below threshold or analysis error
2	ERROR — invalid JSON or unreadable input

GitHub Actions example:

- name: Check context quality
  run: contextops check context.json --min-score 75 --profile rag

---

contextops diff <file_a> <file_b>

Compare two context snapshots to detect regressions.

contextops diff before.json after.json

Shows score delta, which penalties changed, and whether the change is an improvement or regression. Useful for A/B testing retrieval strategies or prompt refactors.

---

contextops stability [file]

Run a deterministic stability report to verify the scoring engine is working correctly.

contextops stability
contextops stability context.json

---

contextops telemetry

Local-only, opt-in telemetry for tracking context quality trends over time.

contextops telemetry status          # Check if telemetry is active
contextops telemetry log --limit 25  # Show recent events
contextops telemetry trends --days 7 # Show 7-day quality trends

---

contextops badge [--score N]

Generate a GitHub shields.io markdown badge.

contextops badge             # Uses your last telemetry score
contextops badge --score 87  # Use a specific score

Output:

[![ContextOps](https://img.shields.io/badge/ContextOps-87-green)](https://github.com/Abhijeet777/contextops)

---

Python API

inspect_context()

from contextops.api.inspect import inspect_context

result = inspect_context(
    payload,          # dict, list, or plain string
    model="gpt-4o",   # model for token counting
    archetype="rag",  # archetype profile (optional)
)

print(result.score)               # int 0–100
print(result.token_breakdown.wasted_tokens)
for rec in result.recommendations:
    print(f"  -> {rec.fix}")

Full result fields:

result.score                  # int 0–100
result.score_breakdown        # redundancy, density, structure, concentration penalties
result.token_breakdown        # total_tokens, by_type, wasted_tokens, estimated_cost_usd
result.redundancy_findings    # list[RedundancyFinding]
result.structure_findings     # list[StructureFinding]
result.recommendations        # list[Recommendation]
result.density_signal         # DensitySignal
result.archetype_resolved     # e.g. "rag"
result.roast                  # str | None (if roast_enabled)
result.metadata               # item_count, model, version

diff_contexts()

from contextops.api.diff import diff_contexts

result = diff_contexts(payload_a, payload_b)

ContextOpsConfig

from contextops.core.config import ContextOpsConfig

config = ContextOpsConfig.default(profile="rag")

config = ContextOpsConfig.from_dict({
    "retrieval_max_ratio": 0.90,
    "system_max_ratio": 0.30,
    "roast_enabled": True
})

---

LangChain Integration

pip install contextops langchain-core

from contextops import ContextOps

# Log the score (default — non-blocking)
chain = chain.with_config({
    "callbacks": [ContextOps.auto()]
})

# Block execution if context quality is too low
chain = chain.with_config({
    "callbacks": [
        ContextOps.auto(
            mode="block",
            min_score=75,
            profile="rag"
        )
    ]
})

result = chain.invoke({"question": "What is the refund policy?"})

Mode	Behaviour
"log"	Prints score report to stdout (default)
"warn"	Emits Python warning if score < min_score
"block"	Raises ContextOpsScoreError if score < min_score — blocks LLM call

---

Archetype Profiles

Archetypes adjust structural thresholds for your specific use case. The global 0–100 score is never affected — only which warnings fire.

Profile	When to Use	Retrieval	System	Memory	Tool
general	Default — mixed use cases	70%	50%	50%	60%
rag	Pure document retrieval	95%	40%	20%	30%
agent	Autonomous agents with tool loops	50%	40%	40%	90%
chatbot	Conversational apps with large history	40%	50%	85%	30%
toolchain	Multi-tool pipelines	50%	40%	30%	95%

Resolution order (highest priority wins):

1. --profile CLI flag
2. archetype= Python API argument
3. "archetype" key inside the JSON payload
4. config.context_profile
5. Default: "general"

---

Input Formats

ContextOps accepts three input formats.

Structured dict (recommended — gives the richest analysis):

{
  "system": "...",
  "messages": [...],
  "chunks": [...],
  "memory": [...],
  "tools": [...]
}

OpenAI message list (paste your existing payload directly):

[
  {"role": "system", "content": "You are a helpful bot."},
  {"role": "user", "content": "Hello"}
]

Plain string (treated as a system prompt).

---

Scoring Engine

Score = max(0, min(100, round(100 − total_penalty)))

Where total_penalty = redundancy + density + structure + concentration.

Penalty	Max	What It Detects
Redundancy	30	Lexical duplication via Jaccard + N-gram overlap
Density	30	Format overhead, whitespace waste, entropy compression
Structure	20	Retrieval dominance, system bloat, memory explosion
Concentration	20	Source dominance + entropy imbalance across chunks

Lexical only. ContextOps uses N-gram overlap, Jaccard similarity, and exact string matching — not semantic similarity. This is intentional: it is a structural analyser, not a semantic one.

---

ContextBench — The LeetCode for Context Engineering

If LeetCode is DSA for algorithms, ContextBench is DSA for context.

Just as competitive programming teaches you that the algorithm matters — not just the answer — ContextBench teaches you that context architecture matters, not just whether the LLM eventually gets it right.

A brute-force O(N²) solution that produces the correct answer still gets penalized for time complexity. A bloated, redundant context that still produces a good LLM response still gets penalized for structural waste. The leaderboard notices both.

---

The Parallel

LeetCode / DSA	ContextBench Leaderboard
Problem set	1,500 pre-built context windows across 5 failure categories
Judge	ContextOps engine — deterministic scorer, same output every time
Your solution	optimize_context(ctx) — takes a broken context, returns a better one
Score metric	Quality (50%) + Compression (35%) + Latency (15%)
Leaderboard	Ranked by final_score across all benchmark samples
Adversarial track	ContextSecBench — 9,500 adversarial attack payloads

---

The Problem Set

ContextBench contains 1,500 samples across 5 categories — think of these as your difficulty tiers:

Category	Samples	What It Tests
Optimal Architectures	300	Healthy pipelines — prove you don't produce false positives
Structural Failures	300	System prompt bloat, retrieval flooding, memory explosion
Redundancy Failures	300	Near-duplicate clusters, boilerplate explosion, paraphrase loops
Agent Architecture Failures	300	Multi-agent context explosion, recursive planning loops, tool chain bloat
Temporal Context Drift	300	Stale memory injection, retrieval drift, invalidated historical state

Every sample has a ground truth — just like a LeetCode problem has expected output:

{
  "ground_truth": {
    "failure_modes": ["system_prompt_bloat"],
    "expected_properties": {
      "contains_redundancy": false,
      "contains_density_bloat": true,
      "contains_structure_imbalance": true
    }
  }
}

Flag redundancy when contains_redundancy: false? False positive. Score drops. Miss the density bloat? False negative. Score drops.

---

Your Solution

Instead of writing a sorting algorithm, you write a context optimizer:

def optimize_context(ctx: dict) -> dict:
    # Your logic: prune, compress, deduplicate, rebalance
    return better_ctx

The harness runs your function across all 1,500 samples and scores:

final_score = (
    0.50 * quality_score     # ContextOps CHS must stay ≥ 78
  + 0.35 * compression       # Token reduction reward
  + 0.15 * latency_mult      # Speed penalty if you're slow
) * 100

The Quality Floor Gate (score ≥ 78) is non-negotiable — optimizers that destroy context quality to hit compression targets are disqualified.

---

The Security Track: ContextSecBench

ContextSecBench is the adversarial extension — the CTF track on top of LeetCode.

9,500 attack payloads covering:

• Prompt injection hiding — malicious instructions buried deep inside retrieved context
• Truncation smuggling — payloads designed to survive chunking with harmful content intact
• Semantic Denial of Service (SDoS) — padding the context window to exhaust the attention budget
• Context poisoning — subtle corruption of retrieval content to bias model behavior
• Format corruption — structural attacks that break parser assumptions

Your optimizer must handle malicious context the same way a secure sorting algorithm must handle adversarial inputs.

---

Why This Matters in Production

Poor context architecture causes problems long before the model itself becomes the bottleneck:

• Higher token costs and inference latency
• Lost-in-the-middle failures on long contexts
• Degraded reasoning quality from attention bandwidth collapse
• Context window exhaustion in long-running agent loops
• Hidden operational waste that compounds silently at scale

Better models don't solve poor context architecture. ContextBench measures the layer that does.

View the Leaderboard →

---

Core Principles

Deterministic

The same input always produces the same score, breakdown, findings, and recommendations — on any machine, at any time. No randomness.

Model-Independent

No embeddings, GPUs, inference APIs, or external AI services. The entire engine is pure Python math.

CI-Friendly

Guaranteed <2s for payloads ≤ 5,000 tokens. <10s for 50,000-token payloads. Works offline. Exit-code driven.

Structurally Scoped

Measures context architecture — not semantic quality, not model correctness, not factual truth.

---

Performance Guarantees

Payload Size	Max Execution Time
≤ 5,000 tokens	< 2 seconds
≤ 20,000 tokens	< 5 seconds
≤ 50,000 tokens	< 10 seconds

---

Vision

ContextOps aims to make LLM context as observable, measurable, and testable as source code.

Just as modern software uses compilers, linters, and static analysis before deployment, LLM systems should validate the structural quality of their context before inference.

Retrieve
      │
      ▼
 Build Context
      │
      ▼
  ContextOps   ←── structural gate
      │
      ▼
     LLM

---

Documentation

Document	Contents
HIGH_LEVEL_DOC.md	Full technical reference — scoring engine, all CLI flags, API, ContextBench, ContextSecBench
STABILITY.md	Formal stability contract, versioning policy, schema guarantees
USER_GUIDE.md	Getting started guide with practical examples
HOW_TO_GET_JSON.md	How to capture context from your running application
CONTRIBUTING.md	Contribution guide — setup, project structure, PR checklist
CHANGELOG.md	Release history

---

Contributing

Contributions are welcome — bug fixes, new ContextBench samples, documentation improvements, and new analyzer signals.

Read CONTRIBUTING.md for:

• Development setup and project structure
• How to run the test suite (including the chaos and signal contract tests)
• Rules for adding new signals or changing the scoring engine
• The PR checklist and commit style guide
• What the stability contract forbids

For anything non-trivial, open an issue first so we can align before you write code.

Repository: github.com/Abhijeet777/contextops

---

License

Released under the Sustainable Use License.