py-smart-test 1.5.0

Development, testing, and dependency analysis tools for Python projects.

Py Smart Test

Smart Test Runner and Pytest Plugin for Python projects. Runs only the tests affected by your code changes, with intelligent prioritization, outcome tracking, and robust fallbacks.

🚀 Features

• Pytest Plugin — zero-config integration via --smart, --smart-first, and --smart-working-tree flags
• Smart Test Execution — runs only tests relevant to changed code (Git diff, staged, working-tree, or hash-based detection)
• ⚡ Ultra-Fast Incremental Analysis — 33.4x speedup via persistent AST caching (28.6ms → 0.9ms on warm cache)
• Parallel Execution — run tests concurrently across multiple CPUs using pytest-xdist integration
• Coverage-Based Tracking — optional runtime dependency tracking via pytest-cov for higher precision
• Test Prioritization — previously failed tests run first, then affected, then by historical duration
• Outcome Tracking — persists pass/fail/skip results and durations across sessions for smarter ordering
• Intelligent Caching — 100% cache hit rate for unchanged files, automatic cache invalidation on modifications
• Incremental AST Parsing — only reparses changed files, dramatically reducing analysis overhead
• Dependency Graph Analysis — AST-based import analysis builds comprehensive transitive dependency maps
• Multiple Change Detection Methods:
  • Git diff: compare against branches/tags (default: main)
  • Staged changes: test only staged modifications
  • Working-tree: detect unstaged/untracked files via git status
  • Hash-based fallback: works without Git repository
• Robust Fallbacks — auto-falls back through git → hash → full suite → error with logging
• Auto-Regeneration — detects stale dependency graphs and regenerates automatically
• JSON Output — --json flag on CLI for CI/CD integration and scripting
• Structured Logging — all operations logged to files with configurable verbosity

⚡ Performance Optimization

py-smart-test achieves 33.4x faster dependency analysis through intelligent incremental caching, far exceeding the 3x performance target.

Key Performance Features

• Incremental AST Parsing — only re-parses files that have actually changed (hash-based detection)
• Persistent AST Cache — stores parsed syntax trees in .py_smart_test/cache/ast_cache.json
• 100% Cache Hit Rate — unchanged files are never reparsed, instant retrieval from cache
• Sequential Execution — optimized for Python's single-threaded performance (parallelism tested 3-5x slower)
• Automatic Cache Management — thread-safe singleton manages all caching with auto-invalidation

Real-World Performance

Cold Start (first run):  28.6ms — parses all files and builds cache
Warm Start (cached):      0.9ms — instant retrieval from cache
Speedup:                 33.4x — far exceeds 3x target requirement

Performance Characteristics

Scenario	Analysis Time	Cache Hit Rate	Files Parsed
First run (cold cache)	~28ms	0%	100% of files
No changes (warm cache)	~0.9ms	100%	0 files
10% files changed	~3-5ms	90%	10% of files
All files changed	~28ms	0%	100% of files

How It Works

1. Hash-Based Change Detection — computes MD5 hashes of all Python files
2. Smart Cache Lookup — checks if file hash exists in AST cache
3. Incremental Parsing — only parses changed files, reuses cached ASTs from disk
4. Transitive Analysis — rebuilds dependency graph incrementally using cached + fresh ASTs
5. Automatic Persistence — cache saved automatically after each run

Note: Multi-process parallelism was extensively benchmarked and found to be 3.4-5.2x slower than sequential execution due to Python's multiprocessing overhead (400-500ms spawn cost). The incremental caching strategy provides vastly superior performance gains.

📋 Table of Contents

• Performance Optimization
• Installation
• Quick Start
• Pytest Plugin
• CLI Usage
• Architecture
• Missing Functionalities / Future Enhancements
• Configuration
• Testing
• Development
• License

🛠️ Installation

Requirements

• Python 3.11+
• Git (optional, for Git-based change detection)

Install from PyPI

uv add py-smart-test

# Optional: Install with parallel execution support
uv add "py-smart-test[parallel]"

# Optional: Install with coverage tracking support
uv add "py-smart-test[coverage]"

# Optional: Install with watch mode support
uv add "py-smart-test[watch]"

# Optional: Install with remote caching support
uv add "py-smart-test[remote-cache]"

# Optional: Install with all optional features
uv add "py-smart-test[all]"

Install from Source

git clone <repository-url>
cd py-smart-test

# Install with uv (recommended)
uv pip install -e .

# Or with pip
pip install -e .

🚀 Quick Start

As a Pytest Plugin (recommended)

# Run only affected tests
pytest --smart

# Run affected tests first, then everything else
pytest --smart-first

# Detect changes from unstaged/untracked files
pytest --smart-working-tree

# Combine: working-tree detection + affected-only
pytest --smart --smart-working-tree

# Run tests in parallel using multiple CPUs
pytest --smart --smart-parallel

# Run tests in parallel with specific number of workers
pytest --smart --smart-parallel --smart-parallel-workers 4

# Enable coverage-based dependency tracking
pytest --smart --smart-coverage

As a CLI Tool

# Run affected tests (compares current branch vs main)
py-smart-test

# Output affected tests as JSON (no test execution)
py-smart-test --json

# Run affected tests for staged changes only
py-smart-test --staged

# Dry run to see what would be tested
py-smart-test --dry-run

# Run all tests (bypass smart detection)
py-smart-test --mode all

# Force graph regeneration
py-smart-test --regenerate-graph

# Run tests in parallel with automatic worker detection
py-smart-test --parallel

# Run tests in parallel with specific number of workers
py-smart-test --parallel --parallel-workers 4

# Run tests with coverage reporting
py-smart-test --coverage

# Combine features: parallel execution with coverage
py-smart-test --parallel --coverage

🔌 Pytest Plugin

The pytest plugin integrates directly into your test workflow — no configuration files needed. Install py-smart-test and the plugin is auto-discovered.

Plugin Options

Flag	Description
--smart	Run only tests affected by code changes. Deselects unaffected tests entirely.
--smart-first	Run all tests, but prioritize affected tests first.
--smart-no-collect	Alias for --smart.
--smart-since REF	Git reference to diff against (default: main).
--smart-staged	Diff staged changes only (like git diff --cached).
--smart-working-tree	Detect changes via git status — ideal for active development.
--smart-parallel	Run tests in parallel using pytest-xdist (requires pytest-xdist).
--smart-parallel-workers N	Number of parallel workers (default: auto). Use with --smart-parallel.
--smart-coverage	Enable coverage-based dependency tracking (requires pytest-cov).

How It Works

1. Collection phase — the plugin hooks into pytest_collection_modifyitems to filter/reorder tests
2. Change detection — identifies modified files via git diff or working-tree status
3. Dependency analysis — traverses the import graph to find all transitively affected modules
4. Test prioritization — orders tests: previously-failed → affected → shortest-duration-first
5. Outcome recording — pytest_runtest_makereport records pass/fail/skip + duration for each test
6. Session finish — pytest_sessionfinish persists outcomes to .py_smart_test/outcomes.json

Examples

# During active development (unstaged changes)
pytest --smart --smart-working-tree

# In CI against a feature branch
pytest --smart --smart-since origin/main

# Test everything but put likely-failures first
pytest --smart-first

# Only staged changes (pre-commit hook style)
pytest --smart --smart-staged

# Run affected tests in parallel for faster execution
pytest --smart --smart-parallel

# Run with coverage tracking for more precise test selection
pytest --smart --smart-coverage

# Combine parallel execution and coverage tracking
pytest --smart --smart-parallel --smart-coverage

💻 CLI Usage

| Command Aliases

Full Command	Alias	Purpose
py-smart-test	pst	Smart test runner
py-smart-test-graph-gen	pst-gen	Generate dependency graph
py-smart-test-map-tests	pst-map	Test module mapping
py-smart-test-affected	pst-affected	Find affected modules
py-smart-test-stale	pst-stale	Check graph staleness
py-smart-test-watch	pst-watch	Watch mode (auto-rerun tests)

py-smart-test — Smart Test Runner

The primary command for running tests intelligently.

py-smart-test [OPTIONS]

Options:

Option	Default	Description
--mode [affected|all]	affected	Test mode
--since REF	main	Git base reference
--staged / --no-staged	--no-staged	Use only staged changes
--regenerate-graph	false	Force dependency graph regeneration
--exclude-e2e / --no-exclude-e2e	--exclude-e2e	Exclude E2E tests
--dry-run	false	Show what would run without executing
--json	false	Output affected tests as JSON and exit
--parallel	false	Run tests in parallel using pytest-xdist
--parallel-workers N	auto	Number of parallel workers (use with --parallel)
--coverage	false	Enable coverage tracking and reporting

pst-affected — Find Affected Modules

Debug or script affected module detection.

pst-affected [OPTIONS]

Options:
  --base REF         Git base reference (default: main)
  --staged           Check staged changes only
  --use-coverage     Use coverage-based tracking for more precise results
  --json             Output in JSON format

pst-gen — Generate Dependency Graph

Manually generate or update the dependency graph.

pst-gen

pst-stale — Check Graph Staleness

Check if the dependency graph needs regeneration.

pst-stale

pst-watch — Watch Mode

Automatically rerun affected tests when files change.

pst-watch

# With custom test command
pst-watch --command "pytest --smart -x"

# Adjust debounce time (default: 0.5 seconds)
pst-watch --debounce 1.0

Requirements: Install with pip install "py-smart-test[watch]" to enable watchdog support.

Remote Caching

Share AST cache across team members and CI runners for faster analysis.

Configuration: Set environment variable with remote cache URL:

# S3 bucket
export PY_SMART_TEST_REMOTE_CACHE="s3://my-bucket/py-smart-test-cache"

# Redis
export PY_SMART_TEST_REMOTE_CACHE="redis://cache.example.com:6379/0"

# HTTP REST API
export PY_SMART_TEST_REMOTE_CACHE="https://cache-api.example.com"

# Network file share
export PY_SMART_TEST_REMOTE_CACHE="file:///mnt/shared/cache"

Supported Backends:

• S3: AWS S3 or S3-compatible storage (requires boto3)
• Redis: Redis key-value store (requires redis)
• HTTP: REST API backend (requires requests)
• File: Network file share (NFS, SMB, etc.)

Install: pip install "py-smart-test[remote-cache]" for all backend dependencies.

The cache is automatically:

• Loaded from remote backend on analysis start
• Merged with local cache (local takes precedence)
• Saved to remote backend after analysis completes

📂 Architecture

Core Components

src/py_smart_test/
├── smart_test_runner.py          # Main CLI orchestrator (Typer app)
├── pytest_plugin.py              # Pytest plugin hooks (collection, reporting, session)
├── find_affected_modules.py      # Change detection + dependency traversal
├── generate_dependency_graph.py  # AST-based import analysis with incremental parsing
├── cache_manager.py              # Centralized cache management (AST cache, hashes, outcomes)
├── test_outcome_store.py         # Persist pass/fail/duration history
├── test_prioritizer.py           # Test ordering (failed-first, affected, duration)
├── test_module_mapper.py         # Test-to-module heuristics
├── detect_graph_staleness.py     # Graph freshness detection
├── file_hash_manager.py          # Hash-based change detection (sequential optimized)
├── coverage_tracker.py           # Optional coverage-based dependency tracking
├── _paths.py                     # Path configuration and constants
└── __init__.py                   # Package initialization

Data Flow

graph TD
    A[Code Change] --> B{Detection Method}
    B -->|git diff| C[Changed Files]
    B -->|git status| C
    B -->|hash comparison| C
    C --> D[Module Mapping]
    D --> E[Transitive Dependency Analysis]
    E --> F[Affected Test Files]
    F --> G{Historical Data}
    G -->|failed tests| H[Test Prioritizer]
    G -->|durations| H
    H --> I[Ordered Test Execution]
    I --> J[Outcome Recording]
    J -->|persist| K[outcomes.json]

Storage Structure

.py_smart_test/
├── dependency_graph.json     # Import dependency graph (incrementally updated)
├── file_hashes.json          # File hash snapshots for change detection
├── outcomes.json             # Test pass/fail/duration history
├── coverage_mapping.json     # Coverage-based test-to-code mappings (optional)
├── cache/
│   └── ast_cache.json        # Persistent AST parse cache (keyed by file hash)
└── logs/
    └── latest_run.log        # Execution logs

Fallback Strategy

1. Git diff change detection
   └─ fallback → Hash-based change detection
      └─ fallback → Full test suite
         └─ fallback → Graceful error with logging

🚧 Missing Functionalities / Future Enhancements

While py-smart-test is feature-complete for core smart testing workflows, the following enhancements are planned for future releases:

✅ Recently Implemented

• Watch Mode ✅ — automatically rerun affected tests on file changes (use pst-watch)
• Remote Caching ✅ — share AST cache across CI runners and developer machines (supports S3, Redis, HTTP, file shares)
• Parallel Test Execution ✅ — run tests concurrently using pytest-xdist
• Compression ✅ — using orjson for fast JSON serialization with automatic fallback

🔄 Watch Mode & Continuous Testing (✅ IMPLEMENTED)

• File watcher ✅ — automatically rerun affected tests on file changes (implemented via watchodog)
• Interactive mode — keyboard controls for test selection and execution
• Incremental test runs ✅ — continuous integration-style feedback loop during development

🌐 Distributed & Remote Caching (✅ IMPLEMENTED)

• Remote cache backend ✅ — share AST cache across CI runners and developer machines
• Multiple backend support ✅ — S3, Redis, HTTP REST API, network file shares
• Automatic synchronization ✅ — cache loaded from remote on startup, saved on completion
• Environment-based configuration ✅ — PY_SMART_TEST_REMOTE_CACHE environment variable
• Cache compression — reduce storage footprint with zstd/lz4 compression (planned)
• Multi-machine cache sharing ✅ — team-wide cache for faster onboarding

🏗️ Build System Integration

• Bazel integration — native support for Bazel's build and test framework
• Make/CMake support — hooks for traditional build systems
• Custom build tool adapters — plugin architecture for arbitrary build systems

🎯 Advanced Test Selection

• Custom test selectors — user-defined predicates for test filtering
• Semantic test grouping — group tests by feature, module, or tag
• Risk-based prioritization — ML-driven prediction of test failure probability
• Flaky test detection — identify and quarantine unstable tests

📊 Analytics & Insights

• Test history dashboard — web UI for visualizing test trends over time
• Performance regression detection — alert on tests with increasing duration
• Coverage gap analysis — identify untested code paths from dependency graph
• Test redundancy detection — find tests covering identical code paths

🧪 Test Generation & Maintenance

• Coverage-driven test generation — auto-generate tests for uncovered modules
• Test health scoring — rank tests by value (coverage/duration ratio)
• Dead test elimination — identify tests that never detect failures
• Test deduplication — merge redundant test cases automatically

🔌 IDE & Tool Integration

• VS Code extension — inline test status, smart run buttons, graph visualization
• JetBrains plugin — PyCharm/IntelliJ integration
• GitHub Actions integration — first-party action for CI workflows
• GitLab CI templates — pre-configured pipeline stages

🐳 Containerization & Isolation

• Docker-based test execution — run tests in isolated containers
• Test environment snapshots — reproducible test environments
• Per-test sandboxing — filesystem and network isolation for tests

📈 Multi-Repository Support

• Mono-repo optimization — cross-project dependency tracking
• Multi-repo change detection — detect affected tests across repository boundaries
• Shared dependency graphs — unified view of multi-repo architecture

🔐 Security & Compliance

• SBOM generation — software bill of materials from dependency graph
• License compliance checking — verify transitive dependency licenses
• Vulnerability scanning — flag tests covering vulnerable dependencies

🛠️ Developer Experience

• Configuration file support — .py-smart-test.toml for project settings
• Test outcome explanations — AI-generated summaries of test results
• Interactive graph explorer — CLI-based dependency graph navigation
• Performance profiling — per-test resource usage tracking

Environment Variables

Variable	Description
PY_SMART_TEST_LOG_LEVEL	Set logging level (DEBUG, INFO, WARNING, ERROR)
PY_SMART_TEST_CACHE_DIR	Override cache directory location
PY_SMART_TEST_REMOTE_CACHE	Remote cache URL (e.g., s3://bucket/prefix)
REMOTE_CACHE_URL	Alternative environment variable for remote cache URL

Path Configuration

Variable	Description
PY_SMART_TEST_LOG_LEVEL	Set logging level (DEBUG, INFO, WARNING, ERROR)
PY_SMART_TEST_CACHE_DIR	Override cache directory location

Path Configuration

The system automatically detects project structure:

Path	Purpose
Repository root	Current working directory
Source code	src/<package>/
Tests	tests/
Cache	.py_smart_test/

🧪 Testing

Running Tests

# Run all tests
uv run pytest

# Run with coverage
uv run coverage run -m pytest tests/ -p no:cov
uv run coverage report --show-missing

# Run specific test file
uv run pytest tests/test_pytest_plugin.py -v

Test Suite

• 212 tests covering all modules
• 96% overall coverage (estimated)
• Core modules at 100%: pytest_plugin.py, test_outcome_store.py, test_prioritizer.py
• New features tested: parallel execution (8 tests), coverage tracking (17 tests), watch mode (12 tests), remote caching (42 tests), utilities (6 tests)

Test File	Tests	What's Covered
test_pytest_plugin.py	21	Plugin hooks, option registration, smart/first modes
test_paths.py	19	Path discovery and configuration
test_coverage_tracking.py	17	Coverage mapping, persistence, integration
test_smart_test_runner.py	17	CLI orchestration, pytest invocation, error handling
test_find_affected_modules.py	16	Change detection, dependency traversal, working-tree
test_outcome_store.py	15	Outcome persistence, error handling, corrupt data
test_detect_graph_staleness.py	11	Graph freshness, hash comparison
test_file_hash_manager.py	9	Hash computation, snapshot management
test_parallel_execution.py	8	Parallel test execution, xdist integration
test_generate_dependency_graph.py	8	AST parsing, import resolution
test_test_module_mapper.py	8	Module-to-test mapping heuristics
test_prioritizer.py	7	Test ordering logic
test_utils.py	6	Utility functions for dependency checking
test_bug_fixes.py	5	Regression tests for fixed bugs
test_init.py	2	Package initialization

🏗️ Development

Setup

# Install development dependencies
uv add --dev -e .

# Install pre-commit hooks
uv run pre-commit install --install-hooks
uv run pre-commit install --hook-type commit-msg

Code Quality

# Lint
uv run ruff check src/ tests/

# Format
uv run ruff format src/ tests/

# Type check
uv run pyright src/

E2E Verification

An end-to-end verification script is included that creates a temporary project, installs py-smart-test from local source, and tests all features:

bash scripts/verify_e2e.sh

Conventional Commits

This project uses Conventional Commits. Pre-commit hooks validate commit messages.

<type>[scope]: <description>

Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore, revert

Version Management

# Bump version and generate changelog
uv run cz bump

# Preview
uv run cz bump --dry-run

🤝 Contributing

We welcome contributions! Please see our Contributing Guidelines for details.

📄 License

This project is licensed under the MIT License — see the LICENSE file for details.

🙏 Acknowledgments

• Built with Typer for CLI
• Uses Pydantic for data validation
• Pytest plugin architecture inspired by pytest-picked