vallm 0.1.16

A complete toolkit for validating LLM-generated code

vallm

A complete toolkit for validating LLM-generated code.

vallm validates code proposals through a four-tier pipeline — from millisecond syntax checks to LLM-as-judge semantic review — before a single line ships.

Features

• Multi-language AST parsing via tree-sitter (165+ languages)
• Syntax validation with ast.parse (Python) and tree-sitter error detection
• Import resolution checking for Python, JavaScript/TypeScript, Go, Rust, Java, C/C++
• Complexity metrics via radon (Python) and lizard (16 languages)
• Security scanning with language-specific patterns and optional bandit integration
• LLM-as-judge semantic review via Ollama, litellm, or direct HTTP
• Code graph analysis — import/call graph diffing for structural regression detection
• AST similarity scoring with normalized fingerprinting
• Pluggy-based plugin system for custom validators
• Rich CLI with JSON/text output formats

Supported Languages

Language	Syntax	Imports	Complexity	Security
Python	✅ AST + tree-sitter	✅ Full resolution (22 methods)	✅ radon + lizard	✅ bandit + patterns
JavaScript	✅ tree-sitter	✅ Node.js builtins	✅ lizard	✅ XSS, eval patterns
TypeScript	✅ tree-sitter	✅ Node.js builtins	✅ lizard	✅ XSS, eval patterns
Go	✅ tree-sitter	✅ stdlib + modules	✅ lizard	✅ SQL injection, exec
Rust	✅ tree-sitter	✅ crates	✅ lizard	✅ unsafe, unwrap
Java	✅ tree-sitter	✅ stdlib packages	✅ lizard	✅ Runtime.exec, SQL
C/C++	✅ tree-sitter	✅ std headers	✅ lizard	✅ buffer overflow, system
Ruby	✅ tree-sitter	⚠️ Limited	✅ lizard	⚠️ Limited
PHP	✅ tree-sitter	⚠️ Limited	✅ lizard	⚠️ Limited
Swift	✅ tree-sitter	⚠️ Limited	✅ lizard	⚠️ Limited
Kotlin	✅ tree-sitter	⚠️ Limited	✅ lizard	⚠️ Limited
Scala	✅ tree-sitter	⚠️ Limited	✅ lizard	⚠️ Limited

Installation

pip install vallm

With optional dependencies:

pip install vallm[all]        # Everything
pip install vallm[llm]        # Ollama + litellm for semantic review
pip install vallm[security]   # bandit integration
pip install vallm[semantic]   # CodeBERTScore
pip install vallm[graph]      # NetworkX graph analysis

Quick Start

Validate Entire Project

# Install with LLM support
pip install vallm[llm]

# Setup Ollama (for semantic review)
ollama pull qwen2.5-coder:7b
ollama serve

# Validate entire project recursively
vallm batch . --recursive --semantic --model qwen2.5-coder:7b

Python API

from vallm import Proposal, validate, VallmSettings

code = """
def fibonacci(n: int) -> list[int]:
    if n <= 0:
        return []
    fib = [0, 1]
    for i in range(2, n):
        fib.append(fib[i-1] + fib[i-2])
    return fib
"""

proposal = Proposal(code=code, language="python")
result = validate(proposal)
print(f"Verdict: {result.verdict.value}")  # pass / review / fail
print(f"Score: {result.weighted_score:.2f}")

CLI Commands Reference

# Batch validation (best for entire projects)
vallm batch . --recursive --semantic --model qwen2.5-coder:7b
vallm batch src/ --recursive --include "*.py,*.js" --exclude "*/test/*"
vallm batch . --recursive --format json --fail-fast
vallm batch . --recursive --verbose --show-issues  # Detailed per-file results

# Output formats for batch results
vallm batch . --recursive --format json   # Machine-readable JSON
vallm batch . --recursive --format yaml   # YAML format
vallm batch . --recursive --format toon   # Compact TOON format
vallm batch . --recursive --format text   # Plain text

# Single file validation
vallm validate --file mycode.py --semantic --model qwen2.5-coder:7b
vallm validate --file app.js --security
vallm validate --file mycode.py --format json  # JSON output

# Quick syntax check only
vallm check mycode.py
vallm check src/main.go

# Configuration and info
vallm info

Generate Validation Summary File

# JSON summary for entire project (with per-file details and issues)
vallm batch . --recursive --format json > validation-summary.json

# YAML summary for src/ directory (with per-file details and issues)
vallm batch src/ --recursive --format yaml > validation-summary.yaml

# TOON format (compact, human-readable) with per-file details
vallm batch . --recursive --format toon > validation-summary.toon

# Text format with security checks
vallm batch . --recursive --format text --security > validation-report.txt

# Full validation with semantic review - save to file
vallm batch . --recursive --semantic --model qwen2.5-coder:7b --format json > full-validation.json

# Tee output to both console and file
vallm batch . --recursive --format json | tee validation-summary.json

Output Structure (JSON/YAML/TOON formats now include per-file details):

{
  "summary": {
    "total_files": 146,
    "passed": 145,
    "failed": 1
  },
  "files": [
    {
      "path": "src/proxym/config.py",
      "language": "python",
      "verdict": "fail",
      "score": 0.45,
      "issues_count": 3,
      "issues": [
        {
          "validator": "syntax",
          "severity": "error",
          "message": "Invalid syntax at line 42",
          "line": 42,
          "column": 15
        },
        {
          "validator": "imports",
          "severity": "error", 
          "message": "Module 'requests' not found",
          "line": 5,
          "column": 0
        }
      ]
    }
  ],
  "failed_files": [
    {"path": "src/proxym/config.py", "error": "Validation fail"}
  ]
}

---
summary:
  total_files: 146
  passed: 145
  failed: 1

files:
  - path: src/proxym/config.py
    language: python
    verdict: fail
    score: 0.45
    issues_count: 3
    issues:
      - validator: syntax
        severity: error
        message: "Invalid syntax at line 42"
        line: 42
        column: 15
      - validator: imports
        severity: error
        message: "Module 'requests' not found"
        line: 5

# vallm batch | 146f | 145✓ 1✗

SUMMARY:
  total: 146
  passed: 145
  failed: 1

FILES:
  [python]
    ✗ src/proxym/config.py
      verdict: fail
      score: 0.45
      issues: 2
        [error] syntax: Invalid syntax at line 42@42
        [error] imports: Module 'requests' not found@5
    ✓ src/proxym/ctl.py
      verdict: pass
      score: 0.92
      issues: 0

FAILED:
  ✗ src/proxym/config.py: Validation fail

Batch Command Options

Option	Short	Description
--recursive	-r	Recurse into subdirectories
--include		File patterns to include (e.g., ".py,.js")
--exclude		File patterns to exclude
--use-gitignore		Respect .gitignore patterns (default: true)
--format	-f	Output format: rich, json, yaml, toon, text
--fail-fast	-x	Stop on first failure
--semantic		Enable LLM-as-judge semantic review
--security		Enable security checks
--model	-m	LLM model for semantic review
--verbose	-v	Show detailed validation results for each file
--show-issues	-i	Show issues for failed files

With Ollama (LLM-as-judge)

# 1. Install and start Ollama
ollama pull qwen2.5-coder:7b

# 2. Run with semantic review
vallm validate --file mycode.py --semantic

from vallm import Proposal, validate, VallmSettings

settings = VallmSettings(
    enable_semantic=True,
    llm_provider="ollama",
    llm_model="qwen2.5-coder:7b",
)

proposal = Proposal(
    code=new_code,
    language="python",
    reference_code=existing_code,  # optional: compare against reference
)
result = validate(proposal, settings)

Validation Pipeline

Tier	Speed	Validators	What it catches
1	ms	syntax, imports	Parse errors, missing modules
2	seconds	complexity, security	High CC, dangerous patterns
3	seconds	semantic (LLM)	Logic errors, poor practices
4	minutes	regression (tests)	Behavioral regressions

The pipeline fails fast — Tier 1 errors stop execution immediately.

Configuration

Via environment variables (VALLM_*), vallm.toml, or pyproject.toml [tool.vallm]:

# vallm.toml
pass_threshold = 0.8
review_threshold = 0.5
max_cyclomatic_complexity = 15
enable_semantic = true
llm_provider = "ollama"
llm_model = "qwen2.5-coder:7b"

Plugin System

Write custom validators using pluggy:

from vallm.hookspecs import hookimpl
from vallm.scoring import ValidationResult

class MyValidator:
    tier = 2
    name = "custom"
    weight = 1.0

    @hookimpl
    def validate_proposal(self, proposal, context):
        # Your validation logic
        return ValidationResult(validator=self.name, score=1.0, weight=self.weight)

Register via pyproject.toml:

[project.entry-points."vallm.validators"]
custom = "mypackage.validators:MyValidator"

Multi-Language Support

vallm supports 30+ programming languages via tree-sitter parsers:

Auto-Detection

from vallm import detect_language, Language

# Auto-detect from file path
lang = detect_language("main.rs")  # → Language.RUST
print(lang.display_name)  # "Rust"
print(lang.is_compiled)     # True

CLI with Auto-Detection

# Language auto-detected from file extension
vallm validate --file script.py      # → Python
vallm check main.go                   # → Go  
vallm validate --file lib.rs          # → Rust

# Batch validation with mixed languages
vallm batch src/ --recursive --include "*.py,*.js,*.ts,*.go,*.rs"

Supported Languages

Language	Category	Complexity	Syntax
Python	Scripting	✓ radon + lizard	✓ ast + tree-sitter
JavaScript	Web/Scripting	✓ lizard	✓ tree-sitter
TypeScript	Web/Scripting	✓ lizard	✓ tree-sitter
Go	Compiled	✓ lizard	✓ tree-sitter
Rust	Compiled	✓ lizard	✓ tree-sitter
Java	Compiled	✓ lizard	✓ tree-sitter
C/C++	Compiled	✓ lizard	✓ tree-sitter
Ruby	Scripting	✓ lizard	✓ tree-sitter
PHP	Web	✓ lizard	✓ tree-sitter
Swift	Compiled	✓ lizard	✓ tree-sitter
+ 20 more via tree-sitter		✓ tree-sitter	✓ tree-sitter

See examples/07_multi_language/ for a comprehensive demo.

Examples

Each example lives in its own folder with main.py and README.md. Run all at once:

cd examples && ./run.sh

Example	What it demonstrates
01_basic_validation/	Default pipeline — good, bad, and complex code
02_ast_comparison/	AST similarity scoring, tree-sitter multi-language parsing
03_security_check/	Security pattern detection (eval, exec, hardcoded secrets)
04_graph_analysis/	Import/call graph building and structural diffing
05_llm_semantic_review/	Ollama Qwen 2.5 Coder 7B LLM-as-judge review
06_multilang_validation/	JavaScript and C validation via tree-sitter
07_multi_language/	Comprehensive multi-language support — 8+ languages with auto-detection
08_code2llm_integration/	Project analysis integration with code2llm
09_code2logic_integration/	Call graph analysis with code2logic
10_mcp_ollama_demo/	MCP (Model Context Protocol) demo with Ollama
11_claude_code_autonomous/	Autonomous refactoring with Claude Code
12_ollama_simple_demo/	Simplified Ollama integration example

Architecture

src/vallm/
├── cli/                    # 🆕 Modular CLI package
│   ├── __init__.py         # Command registration and app export
│   ├── command_handlers.py # CLI command implementations
│   ├── output_formatters.py # Output formatting utilities
│   ├── settings_builders.py # Settings configuration logic
│   └── batch_processor.py  # Batch processing logic
├── cli.py                  # 🆕 Simplified main entry point (9L)
├── config.py               # pydantic-settings (VALLM_* env vars)
├── hookspecs.py            # pluggy hook specifications
├── scoring.py              # Weighted scoring + verdict engine (CC=18 validate function)
├── core/
│   ├── languages.py        # Language enum, auto-detection, 30+ languages
│   ├── proposal.py         # Proposal model
│   ├── ast_compare.py      # tree-sitter + Python AST similarity
│   ├── graph_builder.py    # Import/call graph construction
│   └── graph_diff.py       # Before/after graph comparison
├── validators/
│   ├── syntax.py           # Tier 1: ast.parse + tree-sitter (multi-lang)
│   ├── imports/            # 🆕 Modular import validators
│   │   ├── base.py         # 🆕 Enhanced base class with shared validate()
│   │   ├── factory.py      # Validator factory
│   │   ├── python_imports.py
│   │   ├── go_imports.py   # 🆕 Uses shared validation logic
│   │   ├── rust_imports.py # 🆕 Uses shared validation logic
│   │   └── java_imports.py # 🆕 Uses shared validation logic
│   ├── complexity.py       # Tier 2: radon (Python) + lizard (16+ langs)
│   ├── security.py         # Tier 2: patterns + bandit
│   └── semantic.py         # Tier 3: LLM-as-judge
└── sandbox/
    └── runner.py           # subprocess / Docker execution

🆕 Code Health Improvements

Recent Refactoring Achievements:

✅ CLI Modularization - Split 850L god module into focused packages:

• cli/command_handlers.py - Command implementations
• cli/output_formatters.py - Output formatting logic
• cli/settings_builders.py - Settings configuration
• cli/batch_processor.py - Batch processing logic
• cli/__init__.py - Command registration and app export

✅ Import Validator Cleanup - Removed 653L legacy module:

• Enhanced BaseImportValidator with shared validation logic
• Eliminated duplicate validate() methods across language validators
• Improved maintainability through template method pattern

✅ Code Deduplication - Removed 469 lines of duplicated code:

• Shared validation runners for examples (154 lines saved)
• Centralized analysis data saving (66 lines saved)
• Common demo utilities (60 lines saved)
• LLM response parsing utilities (40 lines saved)
• Import validator logic consolidation (40 lines saved)
• Additional utility function consolidation (109 lines saved)

Updated Code Metrics:

Metric	Before	After	Improvement
God Modules (>500L)	2	0	✅ 100% eliminated
Max Cyclomatic Complexity	42	~18	✅ 57% reduction
Code Duplication	504 lines	35 lines	✅ 93% eliminated
CLI Module Size	850 lines	9 lines	✅ 99% reduction

Remaining Critical Functions:

Function	Location	CC	Status
validate	scoring.py:122	18	🟡 Acceptable
_check_lizard	complexity.py	12	🟡 Acceptable
_parse_response	semantic.py	12	🟡 Acceptable

Roadmap

v0.2 — Completeness ✅ MAJOR PROGRESS

• ✅ CLI modularization - Split 850L god module into focused packages
• ✅ Import validator cleanup - Removed 653L legacy module
• ✅ Code deduplication - Eliminated 469 lines of duplicate code
• ✅ God module elimination - 100% reduction in god modules
• ✅ Complexity reduction - 57% reduction in max cyclomatic complexity
• Wire pluggy plugin manager (entry_point-based validator discovery)
• Add LogicalErrorValidator (pyflakes) and LintValidator (ruff)
• TOML config loading (vallm.toml, [tool.vallm])
• Pre-commit hook integration
• GitHub Actions CI/CD

v0.3 — Depth

• AST edit distance via apted/zss
• CodeBERTScore embedding similarity
• NetworkX cycle detection and centrality in graph analysis
• RegressionValidator (Tier 4) with pytest-json-report
• TypeCheckValidator (mypy/pyright)
• Extract output formatters

v0.4 — Intelligence

• --fix auto-repair mode (LLM-based retry loop)
• hypothesis/crosshair property-based test generation
• E2B cloud sandbox backend
• Streaming LLM output

See TODO.md for the full task breakdown.

License

Apache License 2.0 - see LICENSE for details.

Author

Created by Tom Sapletta - [e-mail]