promptlint 0.1.0

A comprehensive prompt analyzer for LLM optimization, security scanning, and cost estimation

PromptLint

A developer tool for analyzing, scoring, and optimizing LLM prompts with built-in security scanning and cost estimation.

Overview

PromptLint is a CLI tool that helps developers write better prompts by providing actionable feedback on clarity, cost efficiency, and security. It's designed as Phase 1 of a larger vision (PromptIR) but delivers immediate value as a standalone linting tool.

Think of it as: ESLint for prompts, with diff capabilities and security scanning.

---

Quick Start

Installation

# Install from source
poetry install

# Make CLI available
poetry run promptlint --help

Basic Usage

# Score a prompt
poetry run promptlint score examples/good_prompt.txt

# Compare two prompt versions
poetry run promptlint diff examples/basic_prompt.txt examples/good_prompt.txt

# Security analysis
poetry run promptlint security examples/injection_prompt.txt

# Cost estimation across models
poetry run promptlint estimate examples/good_prompt.txt

# List supported models
poetry run promptlint models

---

Features

1. Prompt Scoring

Scores prompts on three dimensions:

• ✨ Clarity (0-10): How well-structured and unambiguous the prompt is

  • Checks for clear instructions
  • Detects ambiguous phrases ("as needed", "maybe", "try to")
  • Validates output format specification
  • Identifies conflicting instructions

• 💰 Cost Efficiency (0-10): Token count and API cost estimates

  • Counts input tokens using tiktoken
  • Estimates output tokens based on complexity
  • Calculates cost for multiple models
  • Identifies expensive patterns

• 🛡️ Security (0-10): Detects prompt injection vulnerabilities

  • Finds injection patterns (e.g., "ignore all previous instructions")
  • Detects unvalidated variables
  • Checks for risky operations
  • Rates overall security risk

2. Prompt Diffing

Compare two prompt versions and see impact:

promptlint diff old_prompt.txt new_prompt.txt

Shows:

• Line-by-line changes
• Clarity, cost, and security deltas
• Cost impact across models
• Recommendation on whether the change improves the prompt

3. Security Analysis

Dedicated security scanning:

promptlint security prompt.txt

Detects:

• Prompt injection attempts
• Unguarded user input
• Risky patterns (code execution, information disclosure)
• Provides severity ratings and remediation suggestions

4. Cost Estimation

Estimate costs across models:

promptlint estimate my_prompt.txt
# Shows costs for GPT-4o, GPT-4o-mini, and Claude 3.5 Sonnet by default

# Or specify models:
promptlint estimate my_prompt.txt --models gpt-4o,gpt-4-turbo,claude-3.5-haiku

---

Architecture

Project Structure

promptlint/
├── promptlint/
│   ├── __init__.py
│   ├── cli.py                    # Main CLI interface (Typer)
│   │
│   ├── core/
│   │   ├── __init__.py
│   │   ├── models.py             # Pydantic data models
│   │   ├── parser.py             # Extract prompt structure
│   │   ├── analyzer.py           # Orchestrate analyzers
│   │   └── differ.py             # Compare prompts
│   │
│   ├── analyzers/
│   │   ├── __init__.py
│   │   ├── clarity.py            # Clarity scoring
│   │   ├── cost.py               # Cost analysis
│   │   └── security.py           # Security scanning
│   │
│   ├── reporters/
│   │   ├── __init__.py
│   │   ├── console.py            # Rich terminal output
│   │   ├── json_reporter.py      # JSON format
│   │   └── markdown.py           # Markdown format
│   │
│   └── utils/
│       ├── __init__.py
│       ├── tokenizer.py          # Token counting
│       └── pricing.py            # Model pricing data
│
├── tests/
│   ├── test_promptlint.py        # Comprehensive tests
│   └── fixtures/
│       ├── good_prompt.txt
│       ├── bad_prompt.txt
│       └── injection_prompt.txt
│
├── examples/
│   ├── basic_prompt.txt
│   ├── good_prompt.txt
│   ├── ambiguous_prompt.txt
│   └── injection_prompt.txt
│
├── pyproject.toml                # Poetry configuration
└── README.md                      # This file

Data Models (Pydantic)

Core models form the foundation for prompt analysis:

• ParsedPrompt: Structured representation of a prompt
• Instruction: Individual instruction or directive
• Variable: Placeholder/variable in prompt
• ScoreResult: Complete analysis result
• Issue: Found problem in prompt
• CostEstimate: Cost prediction for a model
• DiffReport: Comparison between two prompts

---

Module Descriptions

Core Modules

core/parser.py

Extracts structure from raw prompt text:

parsed = PromptParser.parse_file("my_prompt.txt")
# or
parsed = PromptParser.parse(raw_text)

Detects:

• Instructions (imperative statements)
• Variables ({name}, {{name}}, , ${name})
• Conditionals (if/then logic)
• Output format requirements
• Examples
• Metadata

core/analyzer.py

Orchestrates all analyzers:

result = Analyzer.analyze(parsed_prompt, models=['gpt-4o', 'gpt-4o-mini'])

Returns ScoreResult with:

• Scores for clarity, cost, security
• List of issues found
• Suggestions for improvement
• Cost estimates for each model

core/differ.py

Compares two prompts:

report = Differ.diff_prompts("old.txt", "new.txt", models=['gpt-4o'])
# or
report = Differ.diff(parsed_old, parsed_new, models=['gpt-4o'])

Analyzers

analyzers/clarity.py

Scores prompt clarity using heuristics:

• Clear structure detection
• Instruction counting
• Ambiguous phrase detection
• Output format validation
• Conflict detection

Scoring: 5.0 base + adjustments for various factors, clamped to 0-10

analyzers/cost.py

Analyzes token count and costs:

• Token counting using tiktoken (with fallback)
• Output token estimation based on complexity
• Cost calculation for multiple models
• Expensive pattern detection

Scoring: 10.0 base, reduced for high token counts

analyzers/security.py

Detects security vulnerabilities:

High-risk patterns:

• Prompt override attempts ("ignore all previous instructions")
• System prompt disclosure ("reveal your system prompt")
• Role confusion attacks
• Code execution risks

Medium-risk patterns:

• Unvalidated variables
• Debug mode references
• Potential information disclosure

Scoring: 10.0 base, reduced by severity of findings

Reporters

reporters/console.py

Beautiful terminal output using Rich:

reporter = ConsoleReporter()
reporter.print_score_report(result)
reporter.print_diff_report(diff_report)
reporter.print_security_report(result)
reporter.print_cost_estimate(result)

reporters/json_reporter.py

JSON format for machine parsing:

json_str = JSONReporter.print_score_report(result)

reporters/markdown.py

Markdown format for documentation:

markdown_str = MarkdownReporter.score_to_markdown(result)

Utils

utils/tokenizer.py

Token counting:

tokenizer = Tokenizer()
tokens = tokenizer.count_tokens(text, model='gpt-4o')
estimated_output = Tokenizer.estimate_output_tokens(prompt_text, complexity='normal')

Supports automatic detection with tiktoken, fallback to word-based estimation.

utils/pricing.py

Model pricing data:

pricing = PricingData.get_pricing('gpt-4o')
cost = PricingData.calculate_cost('gpt-4o', input_tokens=100, output_tokens=200)
models = PricingData.get_supported_models()

---

CLI Commands

promptlint score [PROMPT_FILE]

Score a single prompt.

Options:

• --model, -m: Primary model (default: gpt-4o)
• --format, -f: Output format (console, json, markdown)
• --output, -o: Save to file

Examples:

promptlint score my_prompt.txt
promptlint score my_prompt.txt --model gpt-4o-mini --format json
promptlint score my_prompt.txt --format markdown --output report.md

promptlint diff [OLD] [NEW]

Compare two prompts.

Options:

• --format, -f: Output format (console, json, markdown)
• --output, -o: Save to file
• --models, -m: Comma-separated models to compare

Examples:

promptlint diff old.txt new.txt
promptlint diff old.txt new.txt --models gpt-4o,claude-3.5-sonnet
promptlint diff old.txt new.txt --format json --output diff.json

promptlint security [PROMPT_FILE]

Security analysis only.

Options:

• --format, -f: Output format (console, json, markdown)
• --output, -o: Save to file

Examples:

promptlint security risky_prompt.txt
promptlint security risky_prompt.txt --format markdown --output security_report.md

promptlint estimate [PROMPT_FILE]

Cost estimation across models.

Options:

• --models, -m: Comma-separated models

Examples:

promptlint estimate my_prompt.txt
promptlint estimate my_prompt.txt --models gpt-4o,gpt-4o-mini,claude-3.5-sonnet

promptlint models

List supported models and pricing.

---

Testing

Run the comprehensive test suite:

# Using pytest
pytest tests/

# Or with poetry
poetry run pytest tests/

# With coverage
poetry run pytest tests/ --cov=promptlint

Test categories:

• Parser tests: Variable/instruction extraction, example detection
• Analyzer tests: Clarity, cost, and security scoring
• Differ tests: Prompt comparison and impact analysis
• Integration tests: File I/O and end-to-end workflows

---

Example Usage

Example 1: Analyze a Prompt

$ poetry run promptlint score examples/good_prompt.txt

📊 Prompt Analysis Report
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓
┃ Metric        ┃ Score       ┃ Details            ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩
│ ✨ Clarity     │ 9.2/10 ✓    │ Excellent          │
│ 💰 Cost        │ 8.1/10 ✓    │ Reasonable         │
│ 🛡️ Security    │ 9.5/10 ✓    │ Low risk           │
└───────────────┴─────────────┴────────────────────┘

💡 Suggestions:
  • Adding examples would improve output quality

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Overall Score: 8.9/10

Example 2: Compare Prompts

$ poetry run promptlint diff examples/basic_prompt.txt examples/good_prompt.txt

📊 Prompt Comparison
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Added: +8 | Removed: -1 | Modified: 0

┏━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Metric   ┃ Delta    ┃ Change %  ┃
┡━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━┩
│ Clarity  │ ↑ +2.5   │ +25%      │
│ Security │ → +0.0   │ 0%        │
│ Cost     │ ↑ +0.02  │ +8%       │
└──────────┴──────────┴───────────┘

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ 📋 Recommendation                                                              ┃
┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ ✅ Clarity improved significantly                                             ┃
┃ ✅ This change improves the prompt overall                                    ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

Example 3: Security Analysis

$ poetry run promptlint security examples/injection_prompt.txt

🛡️ Security Analysis
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

❌ High Severity Issues:
  • HIGH RISK: Prompt override attempt (line 2)
    → Remove or rephrase this instruction to prevent prompt injection

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Overall Security Score: 3/10

---

Configuration

Future support for .promptlint.yaml:

default_model: gpt-4o
output_format: console

rules:
  clarity:
    min_score: 7.0
  cost:
    max_tokens: 2000
    warn_threshold: 1000
  security:
    min_score: 8.0

ignore_patterns:
  - "# promptlint-ignore"

---

Technical Stack

• Python 3.11+
• Typer: CLI framework
• Rich: Terminal output formatting
• Pydantic: Data validation and models
• tiktoken: Token counting (OpenAI tokenizer)
• PyYAML: Configuration (optional, future)

Dev Dependencies

• pytest: Testing
• black: Code formatting
• ruff: Linting
• mypy: Type checking

---

Roadmap

Phase 1 (Current): MVP ✅

• ✅ Prompt scoring (clarity, cost, security)
• ✅ Diff functionality
• ✅ CLI interface
• ✅ Multiple output formats
• ✅ Comprehensive tests
• ✅ Example prompts

Phase 2 (Next): Enhancement

• LLM-based consistency testing
• Regression testing with snapshots
• VS Code extension
• Web UI for analysis

Phase 3: PromptIR Evolution

• Graph visualization of prompt logic
• Optimization layer
• Execution simulation
• Multi-model comparison

Phase 4: SaaS

• Team collaboration
• API access
• Enterprise features
• Prompt versioning

---

Contributing

Contributions welcome! Areas for improvement:

• Enhanced heuristics for clarity scoring
• Support for more models and pricing data
• Plugin system for custom rules
• Performance optimizations
• Additional output formats

---

License

MIT

---

Notes

• Model pricing is current as of November 2025. Check utils/pricing.py for updates.
• Token counting uses tiktoken when available for accuracy. Falls back to word-based estimation.
• Security patterns are based on common OWASP prompt injection techniques.
• Clarity heuristics combine structural analysis with common best practices.

---

Troubleshooting

"tiktoken not found" warning

Token counting will use word-based fallback. Install tiktoken for accuracy:

poetry add tiktoken

No cost estimates shown

Check that models are supported:

promptlint models

Add unsupported models to utils/pricing.py.

Parser not finding expected instructions

The parser uses heuristics. Help by:

• Using imperative verbs (generate, create, analyze, etc.)
• Starting instructions on new lines
• Being explicit about the task

---

Made with ❤️ for prompt engineers everywhere.