LintLang

Catch agent failures before your agent runs.

Product page · Playground · PyPI · Docs · 简体中文

LintLang is a local, deterministic static linter for the instructions and tool interfaces an AI agent is given. It flags ambiguous tool choices, mixed output formats, schema gaps, missing bounds, and other setup defects before the agent runs.

Point it at a project directory. LintLang finds supported agent-facing content inside the files you already use: MCP and function-tool definitions nested in JSON/YAML, parameter schemas, system prompts, messages, output contracts, AGENTS.md, CLAUDE.md, GEMINI.md, SKILL.md, and supported Python prompt code.

uvx lintlang scan .

See LintLang in action. A stylized view of the real DeerFlow finding behind merged PR #5656.

Below, by contrast, is unedited terminal output on a bundled fixture: two of the ten findings from lintlang scan samples/bad_tool_descriptions.yaml, long lines wrapped.

  ❌ FAIL — 1 CRITICAL, 1 HIGH, 5 MEDIUM, 3 LOW
  Inspected: 5 tools (4 described, 5 with a schema), system prompt

    !! [CRITICAL] H1.1 samples/bad_tool_descriptions.yaml:34  tool:process_ticket
      Tool 'process_ticket' has no description.
      → Add a specific, disambiguating description that explains WHEN to use this tool, not just WHAT it does.

    ~ [MEDIUM] H1.6 samples/bad_tool_descriptions.yaml:10  tool:get_user_info vs tool:fetch_user_data
      Tools 'get_user_info' and 'fetch_user_data' carry no differentia — every meaning-bearing term in one is
      present, or has a synonym, in the other. Both descriptions may be accurate and still give a model nothing
      to choose between them.
      Evidence: "'Get user info' vs 'Get user data from the system'"
      → Name a condition that selects one over the other. State what each tool is for that the other is NOT
      for — e.g. 'use X for orders already placed, use Y for carts not yet submitted'.

How LintLang differs

• From LLM-as-judge reviewers: no model calls and no network access during a scan. pyyaml is the only runtime dependency, and the test suite runs the CLI with outbound sockets disabled (tests/test_first_run_recipe.py). Findings and verdicts are deterministic: the same tree produces byte-identical JSON and SARIF (tests/test_sarif.py); only the terminal summary's elapsed time varies. That is what lets it gate CI.
• From generic linters: agent-aware. It reads tool descriptions, system prompts, SKILL.md, MCP definitions, and parameter schemas as instructions a model will act on, not as prose.
• Every finding names the file and proposes a specific fix, and cites the line wherever the parser can justify one. LintLang never invents line numbers.

What LintLang catches

Agent configuration can be valid YAML, JSON, Markdown, or Python and still give a model bad instructions, bad choices, or an interface it cannot reliably use.

LintLang catches problems like:

• Ambiguous tools — sibling tools that overlap without a clear reason for the model to choose one over another.
• Missing bounds — retries, loops, or tool use without explicit stopping or progress conditions.
• Schema mismatches — missing required fields, unclear parameters, and schemas that do not communicate enough intent.
• Mixed output formats and missing priorities — a prompt that names more than one output format (H6 flags any two recognized formats, even when each is scoped to a case), and long instruction lists with no stated priority order (H5). LintLang does not detect semantic contradictions between two instructions, for example "always do X" next to "never do X".
• SKILL.md defects — missing or invalid metadata, unclear usage criteria, and skill names that do not match their directory.
• Context and message errors — stale project references, unbounded persistence, malformed roles, and broken tool-message sequences.
• Embedded agent logic — supported Python prompts, literal tool definitions, and selected pipeline thresholds.

Each result says what LintLang inspected. Content with no recognized agent-facing structures is reported as SKIPPED, never PASS. Tool comparisons are within one parsed input; a directory scan does not combine tools from separate files into one selection namespace.

For exact extraction rules and detector behavior, see the technical reference.

Quickstart

Requires Python 3.10+.

Run once without installing:

uvx lintlang scan .

Or name a specific configuration source:

uvx lintlang scan AGENTS.md
uvx lintlang scan SKILL.md
uvx lintlang scan agent.yaml

Install with pip:

pip install lintlang
lintlang scan .

Or with Homebrew on macOS:

brew install hermes-labs-ai/tap/lintlang
lintlang scan .

Findings are advisory by default.

Block on HIGH or CRITICAL findings:

lintlang scan . --fail-on fail

Include MEDIUM findings in the gate:

lintlang scan . --fail-on review

LintLang also emits JSON, SARIF, and GitLab Code Quality reports for automation. See the GitLab CI guide for a copyable Code Quality job.

Put it in CI

Generate a pinned GitHub Actions workflow that scans the repository directory:

lintlang init --github --path .

The generated Action gates HIGH or CRITICAL findings by default. Use a narrower path when CI should check only one configuration source.

For an existing repository with known findings, record a reviewed baseline:

lintlang scan . --write-baseline .lintlang-baseline.json

Then gate new or changed findings:

lintlang scan . \
  --baseline .lintlang-baseline.json \
  --fail-on review

See GitHub CI and Code Scanning and baseline adoption.

Measured

What we can claim today, and what we can't:

• More than 1,100 passing tests, run in CI on Python 3.10–3.13 on every pull request and push to main. Every tagged release from v0.3.1 through v0.8.2 points at a commit with a passing CI run; the release workflow checks tag/version parity and builds, it does not re-run the suite. Reproduce with pip install -e ".[dev]" && pytest -q.
• Regression corpus (evals/corpus/cases.jsonl, 2 cases and 23 variants today): immutable case IDs with positive/negative controls per phrase class, each linked to a focused test. It guards detector boundaries against drift — it does not estimate accuracy or false-positive rates.
• Sample detection check (evals/sample-detection-rate.sh): 4 deliberately-broken fixtures must fail, 1 clean fixture must pass. A release gate, not a benchmark.
• Daily proof loop (.github/workflows/proof-benchmark.yml): a clean clone scans a broken fixture to SARIF, swaps in the clean one, and must go fail → pass in under 300 seconds. It proves the wiring and timing, not accuracy.
• Field evidence: lintlang findings merged upstream, including bytedance/deer-flow PR #5656 (H1.9 skill name/directory mismatch — the skill declared vercel-deploy but lived under vercel-deploy-claimable, merged by the maintainer) and bytebase/dbhub PR #447 (H4.5 — a CLAUDE.md path that no longer existed in the tree).

What we don't publish yet: accuracy and false-positive rates on real-world projects. The corpus measures reproducible boundaries, not prevalence. Reproducible false positives are the most useful bug reports; see Contributing.

Integrations

LintLang works with GitHub Actions, GitHub Code Scanning, pre-commit, Claude Code, Cursor, GitHub Copilot CLI, Gemini CLI, Pi, OpenCode, Hermes Agent, and MegaLinter.

See the integration guide for setup and compatibility.

LintLang does not run models, observe runtime tool choices, or establish that an agent is production-safe. A clean scan means only that the selected static checks found no covered defects in the recognized content.

Documentation

• Technical reference — supported structures, detector behavior, CLI, JSON, and SARIF
• GitHub CI and Code Scanning
• Baselines
• Research: relational tool-description analysis (H1.6) — the Tool Differentia technical note and its scope, and the taxonomy of epistemic failure modes that motivates the tool
• Integrations
• Changelog

Contributing

Bug reports, disputed findings, reproducible false positives, documentation corrections, and focused contributions are welcome.

See CONTRIBUTING.md and SECURITY.md.

LintLang is developed by Hermes Labs.

License

Apache License 2.0