semlint

semlint is a small, local Python CLI for finding redundant information in agent instruction corpora: SKILL.md, AGENTS.md, prompts, rules, and other Markdown/text files.

It answers: how many tokens are consuming context without adding unique information, and where are they?

Quick start

uv sync
uv run semlint ./skills
uv run semlint ./skills ./rules/AGENTS.md
uv run semlint './skills/**/*.md' './prompts/*.txt'
uv run semlint ./skills --threshold 0.90 --min-block-tokens 10
uv run semlint ./skills --split markdown
uv run semlint ./skills --output json > report.json
uv run semlint ./skills --limit 10
uv run semlint ./skills -s asc similarity
uv run semlint ./skills --fail-above 0.10
uv run semlint self list

The command exits with 1 when --fail-above is exceeded and 2 for invalid input/errors.

Stronger semantic matching

SemHash with local Model2Vec embeddings is the default semantic engine. It loads its small CPU model on first run (and caches it locally). For an explicit alternate Model2Vec model, use:

uv run semlint ./skills --model minishlab/potion-base-8M --threshold 0.90

SemHash and Model2Vec run locally on CPU. No API, LLM, vector database, server, or GPU is needed.

Agent capability discovery

semlint self list prints one full command synopsis and help description per line. Optional arguments and options are bracketed inline. The listing is generated by inspecting the registered Typer command definitions, so it stays aligned with the CLI.

What it scans

• Accepts one or more file paths, directories, or glob patterns; recursively scans .md and .txt files and de-duplicates overlapping matches.
• By default (--split auto), uses Markdown block boundaries for .md files and blank-line paragraph splitting for other supported text files. Markdown blocks include paragraphs (including list and blockquote paragraphs), fenced/indented code, HTML blocks, and tables. Headings and front matter are not treated as instruction blocks.
• Use --split paragraph to force blank-line splitting, or --split markdown to force Markdown parsing.
• All modes preserve original source text and line ranges; none infer sections or rule IDs.
• Ignores blocks below --min-block-tokens.
• Clusters likely duplicates without modifying source files.

Similarity is intentionally configurable because generic phrases can be false positives. Review clusters before replacing instructions.

Output formats

Text is the default and uses Rich-formatted summary metrics, cluster separators, cluster summaries, then a blank line followed by blank-line-separated path:start-end locations and numbered source lines. By default, clusters sort by estimated redundant tokens (largest first); -s|--sort <asc|desc> <occurrences|similarity|estimated-savings> selects a different order. --limit N displays only the first N clusters while keeping corpus-wide metrics unchanged. Linters commonly use this compact location notation with a source code frame; Rich provides readable terminal rendering without introducing a separate diagnostic framework.

Choose JSON with --output json. Top-level fields include files_scanned, blocks_scanned, total_tokens, exact_duplicate_percentage, semantic_redundancy_percentage, estimated_unique_tokens, estimated_redundant_tokens, estimated_context_token_savings, and clusters. Clusters include similarity, all source blocks, and token/savings estimates—no canonical or action suggestions.

The savings estimate assumes retaining the shortest occurrence in each cluster; review the source blocks yourself before deciding what to consolidate. context_redundancy is token-weighted:

estimated_redundant_tokens / total_tokens

This is an estimate, not a tokenizer-specific bill. The initial version counts word tokens for transparent, consistent reporting.

Development

just build uses the installed doxxer binary to inject the next Git-derived version into pyproject.toml, runs uv build, then restores the 0.0.0 source placeholder. Install doxxer before building.

just build

Install the project and development tools with uv sync, then run static checks:

uv sync
uv run ruff check src
uv run ty check src

The project uses Python 3.13+, Pydantic BaseModels for domain and report models, and a modular src/semlint package (parser/scanner, SemHash clustering, metrics, reporting, and Typer CLI). Structured SemHash inputs and JSON report output are validated with Pydantic. It does not automatically edit files.