sift-gateway 0.3.4

Keep big MCP responses out of your context window. Query them.

Sift

Reliability gateway - Schema-stable, secret-safe, pagination-complete JSON for AI agents.

---

Sift is built for workflows where data correctness matters as much as model quality: enterprise automation, research pipelines, and long-running agent sessions.

For one-off CLI tasks, plain jq or Python can be enough. Sift adds value when you need guarantees: consistent schema handling, secret redaction before data re-enters model context, explicit pagination continuation, and an auditable artifact history.

Sift stores tool output as artifacts, infers schema metadata, and returns either inline payload (full) or an artifact reference (schema_ref). In schema_ref, Sift returns either a representative sample_item preview or verbose schemas fallback.

Keeping large payloads out of prompt context is still a core benefit, but it is one outcome of these guarantees rather than the only goal. See Why Sift exists for research and ecosystem context.

Sift works with MCP clients (Claude Desktop, Claude Code, Cursor, VS Code, Windsurf, Zed) and CLI agents (OpenClaw, terminal automation). Same artifact store, same query interface, two entry points.

                           ┌─────────────────────┐
  MCP tool call ──────────▶│                     │──────────▶ Upstream MCP Server
  CLI command   ──────────▶│        Sift         │──────────▶ Shell command
                           │                     │
                           │   ┌─────────────┐   │
                           │   │  Artifacts  │   │
                           │   │  (SQLite)   │   │
                           │   └─────────────┘   │
                           └─────────────────────┘
                                     │
                                     ▼
                           Small output? return inline
                           Large output? return schema reference
                           Agent queries what it needs via code

Quick start

MCP agents

pipx install sift-gateway
sift-gateway init --from claude

Restart your MCP client. Sift mirrors upstream tools, persists outputs as artifacts, and returns either the full payload (for small responses) or a schema reference (for large responses). The agent can query stored artifacts with artifact(action="query", query_kind="code", ...).

--from shortcuts: claude, claude-code, cursor, vscode, windsurf, zed, auto, or an explicit path.

CLI agents (OpenClaw, terminal automation)

pipx install sift-gateway
sift-gateway run -- kubectl get pods -A -o json

Use this flow when you need reproducibility and policy controls on command output (not just ad-hoc extraction). Large output is stored and returned as an artifact ID plus schema_ref metadata. Example:

sift-gateway code <artifact_id> '$.items' --code "def run(data, schema, params): return {'rows': len(data)}"

Another capture example:

sift-gateway run -- curl -s api.example.com/events

For OpenClaw, see the OpenClaw Integration Pack.

Example workflow

You ask an agent to check what is failing in prod:

datadog.list_monitors(tag="service:payments")

Without Sift, 70 KB of monitor configs and metadata can go straight into context. That is about 18,000 tokens before the next tool call.

With Sift, the agent gets a schema reference:

{
  "response_mode": "schema_ref",
  "artifact_id": "art_9b2c...",
  "sample_item": {
    "name": "Payments monitor",
    "status": "Alert",
    "type": "query alert"
  },
  "sample_item_source_index": 0,
  "sample_item_count": 120
}

If a representative sample is not valid for the result set, schema_ref falls back to schemas.

The agent can then run a focused query:

artifact(
    action="query",
    query_kind="code",
    artifact_id="art_9b2c...",
    root_path="$.monitors",
    code="def run(data, schema, params): return [m for m in data if m.get('status') == 'Alert']",
)

In this example, two calls use about 400 tokens and still leave room for follow-up steps.

How it works

Sift runs one processing pipeline for MCP and CLI:

1. Execute the tool call or command.
2. Parse JSON output.
3. Detect pagination from the raw response.
4. Redact sensitive values (enabled by default).
5. Persist the artifact to SQLite.
6. Map the schema (field types, sample values, cardinality).
7. Choose response mode: full (inline) or schema_ref (sample preview or schema fallback).
8. Return the artifact-centric response.

Response mode selection

Sift chooses between inline and reference automatically:

• If the response has upstream pagination: always schema_ref.
• If the full response exceeds the configured cap (default 8 KB): schema_ref.
• If the schema reference is at least 50% smaller than full: schema_ref.
• Otherwise: full (inline payload).

Pagination

When upstream tools or APIs paginate, Sift handles continuation explicitly.

MCP:

artifact(action="next_page", artifact_id="art_9b2c...")

CLI:

sift-gateway run --continue-from art_9b2c... -- gh api repos/org/repo/pulls --after NEXT_CURSOR

Each page creates a new artifact linked to the previous one through lineage metadata. The agent can run code queries across the full chain.

Code queries

Both MCP and CLI agents can analyze stored artifacts with Python.

MCP:

artifact(
    action="query",
    query_kind="code",
    artifact_id="art_123",
    root_path="$.items",
    code="def run(data, schema, params): return {'count': len(data)}",
)

CLI:

# Function mode
sift-gateway code art_123 '$.items' --code "def run(data, schema, params): return {'count': len(data)}"

# File mode
sift-gateway code art_123 '$.items' --file ./analysis.py

Multi-artifact query example:

artifact(
    action="query",
    query_kind="code",
    artifact_ids=["art_users", "art_orders"],
    root_paths={"art_users": "$.users", "art_orders": "$.orders"},
    code="""
def run(artifacts, schemas, params):
    users = {u["id"]: u["name"] for u in artifacts["art_users"]}
    return [{"user": users.get(o["user_id"]), "amount": o["amount"]}
            for o in artifacts["art_orders"]]
""",
)

Import allowlist

Code queries run with a configurable import allowlist. Default allowed import roots include math, json, re, collections, statistics, heapq, numpy, pandas, jmespath, datetime, itertools, functools, operator, decimal, csv, io, string, textwrap, copy, typing, dataclasses, enum, fractions, bisect, random, base64, and urllib.parse. Third-party modules are usable only when installed in Sift's runtime environment.

Install additional packages:

sift-gateway install scipy matplotlib

Security

Code queries use AST validation, an import allowlist, timeout enforcement, and memory limits. This is not a full OS-level sandbox.

Outbound secret redaction is enabled by default to reduce accidental leakage of API keys from upstream tool responses.

See SECURITY.md for the full security policy.

Configuration

Env var	Default	Description
SIFT_GATEWAY_DATA_DIR	.sift-gateway	Root data directory
SIFT_GATEWAY_PASSTHROUGH_MAX_BYTES	8192	Inline response cap
SIFT_GATEWAY_SECRET_REDACTION_ENABLED	true	Redact secrets from tool output
SIFT_GATEWAY_AUTH_TOKEN	unset	Required for non-local HTTP binds

Full reference: docs/config.md

Documentation

Doc	Covers
Why Sift Exists	Research and ecosystem context
Quick Start	Install, init, first artifact
Recipes	Practical usage patterns
OpenClaw Pack	OpenClaw skill, quickstart, templates
API Contracts	MCP + CLI public contract
Configuration	All settings and env vars
Deployment	Transport modes, auth, ops
Errors	Error codes and troubleshooting
Observability	Structured logging and metrics
Architecture	Design and invariants

Benchmarks

The Tier 1 benchmark compares Sift's schema_ref + codegen approach against naive full-JSON context stuffing across 12 real-world datasets and 103 factual questions.

Model	Condition	Accuracy	Input Tokens	Token Reduction
claude-sonnet-4-6	Baseline	31/103 (30.1%)	10,757,230	—
claude-sonnet-4-6	Sift	99/103 (96.1%)	501,639	95.3%
claude-opus-4-6	Baseline	34/103 (33.0%)	10,757,230	—
claude-opus-4-6	Sift	98/103 (95.1%)	508,496	95.3%

Sift achieves ~96% accuracy on both models while using ~95% fewer input tokens. The baseline struggles on large and deeply nested datasets (earthquakes, laureates, products) where payloads crowd the context window, while Sift handles them through artifact queries.

python benchmarks/tier1/fetch_data.py
python benchmarks/tier1/harness.py --model claude-sonnet-4-6

# or with uv (recommended for clean clones)
uv run python benchmarks/tier1/fetch_data.py
uv run python benchmarks/tier1/harness.py --model claude-sonnet-4-6

See benchmarks/tier1/ for the full suite and per-dataset breakdown.

Development

git clone https://github.com/lourencomaciel/sift-gateway.git
cd sift-gateway
uv sync --extra dev

uv run python -m pytest tests/unit/ -q
uv run python -m ruff check src tests
uv run python -m mypy src

See CONTRIBUTING.md for the full development guide.

License

MIT - see LICENSE.