LintLang

LintLang statically analyzes the natural-language instructions that control AI agents, catching ambiguous tools, missing limits, and conflicting directives before runtime.

It flags patterns such as:

• empty, vague, or overlapping tool descriptions;
• tool pairs with no term that distinguishes one from the other (H1.6);
• missing stop conditions and unbounded retries;
• inconsistencies between tool schemas and their descriptions;
• unscoped context and vague instructions;
• conflicting output formats and malformed message roles;
• embedded prompts and uncalibrated thresholds in Python pipelines.

LintLang's default static checks are deterministic and local. They make no LLM, API, telemetry, or network calls.

LintLang was developed as the engineering offshoot of A Taxonomy of Epistemic Failure Modes in Large Language Models, but its bounded detectors do not claim to implement or validate every failure mode in the paper.

Technical note

Tool Differentia: Relational Static Analysis for AI Agent Tool Descriptions documents LintLang H1.6, the bounded pairwise check for tool descriptions that do not supply an analyzed distinction from a neighboring tool. It is a technical note, not a semantic-equivalence proof or a runtime-selection evaluation. Use its version-independent concept DOI, 10.5281/zenodo.21817243, for citation; the current archived release is Version 1.0.1.

Quick start

Run once without installing, using uv:

uvx lintlang scan AGENTS.md

For a persistent command in an isolated environment, use pipx:

pipx install lintlang
lintlang scan AGENTS.md

If pipx's app directory is not on PATH, run pipx ensurepath, open a new shell, and retry the scan.

Or install from PyPI into the current Python environment:

python -m pip install lintlang

Requires Python 3.10+.

From your project root, point LintLang at an actual instruction file:

lintlang scan AGENTS.md

If your project uses another filename, replace AGENTS.md with its prompt, tool-definition, agent-configuration, or supported directory path.

Character.AI's public Larch repository runs a pinned LintLang release in recurring CI. LintLang is also packaged for Gentoo through the third-party Haven overlay.

When you are ready to make HIGH or CRITICAL findings block CI:

lintlang scan AGENTS.md --fail-on fail

Each finding identifies the affected location, the detected pattern, its severity, and a suggested review action.

Try the bundled example

The source repository includes a deliberately broken example:

git clone --depth 1 https://github.com/hermes-labs-ai/lintlang.git
cd lintlang

lintlang scan samples/bad_tool_descriptions.yaml --fail-on fail

Excerpt from lintlang 0.4.1:

LINTLANG v0.4.1

FAIL — 1 CRITICAL, 2 HIGH, 7 MEDIUM, 3 LOW

H1: Tool Description Ambiguity

  [CRITICAL] H1.1 tool:process_ticket
  Tool 'process_ticket' has no description.

  [HIGH] H1.2 tool:get_user_info
  Tool 'get_user_info' has a very short description (13 chars):
  "Get user info"

…

H2: Missing Constraint Scaffolding

  [HIGH] system_prompt
  System prompt defines tools but contains no termination conditions,
  retry budgets, or progress checks.

The command exits with status 1 because it includes --fail-on fail.

Verdicts and CI behavior

PASS applies only to recognized content extracted from the requested inputs and the checks and severity filters selected for that run. It does not mean that every structure in an arbitrary JSON or YAML file was extracted. A clean LintLang scan is not evidence that an agent is safe or runtime-correct.

By default, findings are reported without failing the process.

• --fail-on fail blocks on FAIL.
• --fail-on review blocks on REVIEW or FAIL.
• Missing, malformed, unreadable, or otherwise unscannable requested inputs remain nonzero regardless of the chosen finding threshold.
• An invocation that finds no eligible files exits nonzero.

Filters such as --min-severity are applied before the verdict. For initial adoption, keep the full output visible and use --fail-on fail to block only the highest-severity findings.

Add it to CI

After choosing one real instruction path in your repository:

jobs:
  lint-agent-instructions:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0

      - name: Inspect agent instructions
        uses: hermes-labs-ai/lintlang@v0.4.0
        with:
          path: AGENTS.md

The release tag pins both the action and the LintLang source it installs. Upgrade that pin deliberately and inspect newly introduced findings before making them blocking.

Add it to pre-commit

Add the hook to .pre-commit-config.yaml with the explicit instruction paths to scan:

repos:
  - repo: https://github.com/hermes-labs-ai/lintlang
    rev: v0.4.0
    hooks:
      - id: lintlang
        args: [AGENTS.md]

Activate it and test the configured paths:

pre-commit install
pre-commit run lintlang

Replace or extend args with the prompt, tool-definition, agent-configuration, or supported directory paths your repository owns. The hook scans only those configured paths and reports findings without blocking on a verdict by default. After reviewing the repository's baseline, opt into blocking FAIL findings:

hooks:
  - id: lintlang
    args: [AGENTS.md, --fail-on, fail]

Missing, unreadable, or malformed configured inputs still return nonzero.

Machine-readable output and GitHub Code Scanning

For machine-readable output:

lintlang scan AGENTS.md --format json --fail-on fail

For deterministic SARIF 2.1.0 output on stdout:

lintlang scan AGENTS.md --format sarif --fail-on fail > lintlang.sarif

Relative inputs are resolved from the current directory. Artifact URIs are URI-encoded paths relative to the nearest Git worktree root (or the current directory when there is no Git worktree). A resolved source outside that root is a fatal output error rather than an absolute-path leak. Python AST findings carry supported line spans; YAML, JSON, and text findings intentionally remain file-level.

The composite Action can write the same report with its optional sarif-file input. In that mode SARIF stdout is redirected to the requested file, while verdict messages remain on stderr and fail-on keeps its normal exit status. Directory creation or file-write errors are fatal.

To ask GitHub to ingest the report, keep generation and upload as separate steps so the upload still runs after a blocking LintLang verdict:

permissions:
  contents: read
  security-events: write

steps:
  - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
  - name: Run LintLang
    uses: hermes-labs-ai/lintlang@v0.4.0
    with:
      path: AGENTS.md
      fail-on: fail
      sarif-file: lintlang.sarif
  - name: Upload LintLang SARIF
    if: always() && (github.event_name == 'push' || (github.actor != 'dependabot[bot]' && github.event.pull_request.head.repo.full_name == github.repository))
    uses: github/codeql-action/upload-sarif@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
    with:
      sarif_file: lintlang.sarif

The complete copy-paste workflow is examples/github-code-scanning.yml. LintLang emits code-quality/static-language results without security tags, security severity, source snippets, or custom fingerprints. GitHub's upload Action may calculate fingerprints during ingestion.

What it inspects

LintLang currently accepts:

• JSON and YAML objects using recognized top-level agent fields such as system_prompt, instructions, tools, functions, messages, and selected response-schema fields;
• .txt, .md, and .prompt instruction files;
• Python files, using AST extraction for prompt-like strings and threshold assignments.

Nested vendor-specific layouts and raw top-level YAML arrays are not automatically normalized. A syntactically valid input must still match a recognized shape for its structured tools or messages to be inspected.

The checks cover reader-facing categories including tool clarity, execution bounds, schema-description alignment, context boundaries, instruction specificity, output contracts, message-role structure, and Python pipeline hygiene.

H1.6: tool descriptions without a differentia

Per-tool schema validation assesses one definition at a time. Within one parsed input, H1.6 instead compares tool definitions with each other and reports a pair when, under LintLang's term-and-synonym model, one or both descriptions provide no distinguishing term. Both tools can be individually valid, so per-tool validation has nothing to report. A mutual finding means neither description distinguishes itself; domination means one tool's terms are all covered by the other, and the finding names which description to repair. Directory scans do not aggregate tool definitions across files or infer a shared namespace.

Findings print the sub-code: ~ [MEDIUM] H1.6 tool:find_tickets vs tool:search_tickets. pattern_id stays H1; JSON output adds a code field holding the most specific identifier.

H1.6 is MEDIUM, so --fail-on fail does not block on it. Matching uses a finite English synonym lexicon, so pairs that say the same thing in different words or a different sentence shape are missed. The absence of an H1.6 finding is not evidence that no such pair exists.

Use narrow, intentional paths. Directory scans can discover Markdown and Python files that were not written as agent configuration; use .lintlangignore or --exclude where needed.

For the exact H-series identifiers:

lintlang patterns

lintlang patterns lists the H1-H7 structural detectors only. Python pipeline findings report as P1 and P2 in scan, JSON, and SARIF output.

See the full technical reference for detector details.

Where it fits

syntax and schema validation
        ↓
LintLang static language checks
        ↓
runtime agent evaluation
        ↓
domain and security review

LintLang is useful during authoring and pull-request review, before runtime testing. It does not:

• determine whether an instruction is factually or semantically correct;
• observe an agent selecting or executing tools;
• prove that a finding causes a runtime failure;
• certify an agent as safe or production-ready;
• replace runtime evaluation or human review.

Suggestions are review aids, not guaranteed meaning-preserving fixes.

Optional instruction preflight

Secondary capability: provider-neutral instruction preflight inspects one present instruction plus explicit context.

More

• Technical reference
• Product scope and invariants
• Changelog
• Contributing
• Report an issue or disputed finding
• Security policy

License

Apache License 2.0

LintLang is maintained by Hermes Labs.