pytest-fastcollect 0.4.0

A high-performance pytest plugin that replaces test collection with a Rust-based implementation

pytest-fastcollect

A high-performance pytest plugin that uses Rust to accelerate test collection. This plugin leverages rustpython-parser to parse Python test files in parallel, with incremental caching to skip unchanged files.

Performance: Up to 2.4x faster collection on large projects (tested on Django's 1977 test files). Best for codebases with 200+ test files.

Features

• 🦀 Rust-Powered Parsing: Uses rustpython-parser for blazing-fast Python AST parsing
• ⚡ Parallel Processing: Leverages Rayon for parallel file processing
• 💾 Incremental Caching: Caches parsed results with file modification tracking
• 🎯 Smart Filtering: Pre-filters test files before pytest's collection phase
• 🔧 Drop-in Replacement: Works as a pytest plugin with no code changes required
• 🎛️ Configurable: Enable/disable fast collection and caching with command-line flags
• 📈 Scales with Size: Performance improvements scale with project size (2-4x on 500+ files)

Installation

From Source

# Clone the repository
git clone https://github.com/yourusername/pytest-fastcollect.git
cd pytest-fastcollect

# Create a virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install maturin and build
pip install maturin
maturin develop --release

# Or install in production mode
maturin build --release
pip install target/wheels/pytest_fastcollect-*.whl

Requirements

• Python 3.8+
• Rust 1.70+
• pytest 7.0+

Usage

Once installed, the plugin is automatically activated for all pytest runs:

# Run pytest as normal - fast collection is enabled by default
pytest

# Collect tests only (useful for benchmarking)
pytest --collect-only

# Disable fast collection
pytest --no-fast-collect

# Clear cache and reparse all files
pytest --fastcollect-clear-cache --collect-only

# Disable caching (always parse)
pytest --no-fastcollect-cache

# Experimental: Parallel module import (2.33x faster on pytest itself!)
pytest --parallel-import --parallel-workers=4

# Experimental: Collection Daemon (instant re-collection)
pytest --daemon-start tests/        # Start daemon
pytest --daemon-status              # Check status
pytest --daemon-stop                # Stop daemon

# Run benchmarks
python benchmark.py --synthetic
python benchmark_incremental.py  # Shows cache effectiveness
python benchmark_parallel.py     # Test parallel import performance

Configuration Options

• --use-fast-collect: Enable Rust-based fast collection (default: True)
• --no-fast-collect: Disable fast collection and use standard pytest collection
• --fastcollect-cache: Enable incremental caching (default: True)
• --no-fastcollect-cache: Disable caching and parse all files
• --fastcollect-clear-cache: Clear the cache before collection
• --benchmark-collect: Benchmark collection time (fast vs standard)
• --parallel-import: [Experimental] Pre-import modules in parallel (default: False)
• --parallel-workers=N: Number of parallel import workers (default: CPU count)
• --daemon-start: [Experimental] Start collection daemon for instant re-collection
• --daemon-stop: Stop the collection daemon
• --daemon-status: Show daemon status (running/not running, PID, uptime, cached modules)

Architecture

How It Works

1. Rust Collector (FastCollector):

   • Walks the directory tree to find Python test files
   • Uses rustpython-parser to parse each file's AST in parallel
   • Extracts file modification times for cache validation
   • Returns collected metadata to Python

2. Incremental Caching:

   • Caches parsed test data with file mtimes in .pytest_cache/v/fastcollect/
   • On subsequent runs, checks file mtimes
   • Only reparses files that have changed
   • Shows cache statistics (hits/misses) after collection

3. pytest Plugin Integration:

   • Hooks into pytest's pytest_ignore_collect to filter files
   • Uses cached data when available
   • Falls back to standard collection on errors

4. File Detection:

   • Test files: test_*.py or *_test.py
   • Ignored directories: .git, __pycache__, .tox, .venv, venv, .eggs, *.egg-info

Components

pytest-fastcollect/
├── src/
│   └── lib.rs                    # Rust implementation (FastCollector)
├── pytest_fastcollect/
│   ├── __init__.py               # Python package init
│   ├── plugin.py                 # pytest plugin hooks
│   ├── cache.py                  # Incremental caching layer
│   ├── daemon.py                 # Collection daemon server
│   ├── daemon_client.py          # Daemon client communication
│   └── filter.py                 # Selective import filtering
├── tests/
│   └── sample_tests/             # Sample tests for validation
├── benchmark.py                  # Performance benchmarking
├── benchmark_incremental.py      # Cache effectiveness benchmark
├── benchmark_parallel.py         # Parallel import benchmarking
├── Cargo.toml                    # Rust dependencies
└── pyproject.toml                # Python package metadata

Benchmarks

v0.4.0 - Selective Import (Latest) ⭐

THE BREAKTHROUGH: Selective import based on -k and -m filters!

How it works:

1. Parse all test files with Rust (parallel, fast)
2. Extract test names and markers from AST
3. Apply -k and -m filters BEFORE importing modules
4. Only import files containing matching tests
5. Result: Massive speedups for filtered runs!

Benchmark Results (100 files, 10 tests/file):

Scenario                        Time      Speedup
Full collection (no filter)     0.98s     baseline
With -k filter (10% files)      0.57s     1.71x faster ⚡
With -m filter (20% files)      0.64s     1.55x faster ⚡
Combined filters                0.55s     1.78x faster ⚡

Real-World Impact:

# Before v0.4.0: imports ALL 100 test files
pytest -k test_user  # 0.98s

# With v0.4.0: imports only 10 matching files
pytest -k test_user  # 0.57s (1.71x faster!)

When it helps most:

• ✅ Running specific tests: pytest -k test_user_login
• ✅ Running marked tests: pytest -m smoke
• ✅ Development workflow (constantly filtering tests)
• ✅ CI/CD with test splits
• ✅ Large test suites with good organization

Key Features:

• Marker detection from decorators (@pytest.mark.slow)
• Keyword matching (function names, class names, file names)
• Supports and, or, not in expressions
• Shows file selection stats with -v
• Fully compatible with pytest's filter syntax

Multi-Project Real-World Benchmarks 📊

Tested on 5 popular Python projects to validate real-world performance:

Project	Files	Baseline	FastCollect	Speedup	Grade
Django	~1977	10.85s	4.49s	2.42x	⚡⚡⚡ Excellent
SQLAlchemy	~219	0.68s	0.63s	1.07x	✓ Minor
Pytest	~108	2.40s	2.54s	0.94x	⚠️ Overhead
Requests	~9	0.61s	0.54s	1.13x	✓ Minor
Flask	~22	0.55s	0.55s	1.00x	→ Neutral

Key Finding: Performance scales with project size! 🚀

Selective Import Performance (additional speedup with -k filters):

• Pytest: Up to 2.75x faster with specific filters (-k test_basic)
• Django: Additional 1.32x faster with keyword filters
• Small projects: Minimal additional benefit

Break-Even Analysis:

• ✅ Large projects (500+ files): 2-4x speedup - highly recommended
• ⚠️ Medium projects (100-300 files): 0.9-1.5x - evaluate first
• → Small projects (< 50 files): ~1.0x - not necessary

Bottom Line: pytest-fastcollect is ideal for large codebases (200+ test files) where collection time becomes a bottleneck. For projects with < 50 files, the overhead roughly equals the benefit.

📄 See REALWORLD_BENCHMARKS.md for comprehensive analysis across all projects.

Parallel Import (Experimental) ⚡⚡⚡

NEW: Pre-import test modules in parallel for additional speedup!

pytest --parallel-import --parallel-workers=4

Benchmark Results (with --parallel-import):

Project	Baseline	With Parallel	Speedup	Grade
Pytest	2.40s	1.03s	2.33x faster	⚡⚡⚡ Excellent
SQLAlchemy	0.69s	0.64s	1.07x faster	✓ Minor
Django	4.80s	4.90s	0.98x slower	⚠️ Overhead

Key Finding: Parallel import works great for projects with simple, independent test modules (like pytest itself!), but can hurt projects with complex interdependent imports (like Django).

When to use:

• ✅ Medium projects (100-300 files) with simple imports → 2-2.5x speedup
• ⚠️ Projects with complex imports → Benchmark first
• ❌ Small projects (< 100 files) → Overhead not worth it

Optimal configuration: 4 workers seems to be the sweet spot for most projects.

ProcessPoolExecutor (experimental):

# Bypass GIL with true process parallelism
pytest --parallel-import --use-processes --parallel-workers=4

Results: ProcessPoolExecutor tested but not recommended

• ❌ Slower than ThreadPoolExecutor in most cases (0.88-1.10x)
• Process overhead > GIL bypass benefit
• Must import twice (subprocess + main process)
• ThreadPoolExecutor remains the better choice

📄 See PARALLEL_IMPORT_RESULTS.md for threading details. 📄 See PROCESS_POOL_RESULTS.md for process pool analysis.

Collection Daemon (Experimental - Phase 1) 🚀

NEW: Long-running daemon process that keeps test modules imported in memory for instant re-collection!

# Start the daemon (imports all modules once)
pytest --daemon-start tests/

# Check daemon status
pytest --daemon-status

# Stop the daemon
pytest --daemon-stop

Expected Performance (Phase 2 - Full Integration):

• First run: ~10s (cold start, imports all modules)
• Subsequent runs: ~0.01s (instant! modules already in memory)
• 100-1000x speedup on subsequent test runs

Current Status (Phase 1 - MVP):

• ✅ Daemon server with Unix socket communication
• ✅ Module pre-importing and caching in memory
• ✅ Start/stop/status management commands
• ⏳ Full pytest collection integration (Phase 2)
• ⏳ File watching for auto-reload (Phase 2)

Architecture:

• Long-running Python process
• Unix socket IPC for client-daemon communication
• Keeps modules in sys.modules across pytest runs
• Background process management with forking
• Per-project daemon instances (separate socket per root)

When to use:

• 🎯 TDD workflows: Constantly re-running tests during development
• 🎯 Watch mode: Instant collection on file changes
• 🎯 Large codebases: Where collection time > 5 seconds
• ⚠️ Not for CI/CD: Designed for development, not one-shot runs

Current Limitations:

• Unix/Linux only (uses Unix domain sockets)
• Phase 1: Infrastructure only, not yet integrated with pytest collection
• Requires manual daemon management (start/stop)
• May require reload after significant code changes

📄 See COLLECTION_DAEMON_PLAN.md for full implementation roadmap.

Django Real-World Benchmark 🚀

The ultimate test: Django's massive test suite with ~1977 Python test files!

Full Collection Performance:

Scenario                        Time      Speedup
Baseline (no plugin)            36.59s    -
FastCollect                     9.16s     3.99x faster ⚡⚡⚡

Selective Import Performance:

Filter Type                     Time      Speedup vs Full
Full collection                 9.16s     baseline
-k test_get                     4.12s     2.22x faster ⚡⚡
-k test_forms                   3.80s     2.41x faster ⚡⚡
-k "test_view or test_model"    4.19s     2.19x faster ⚡⚡

Combined Impact: FastCollect + Selective Import = 9.6x faster than baseline pytest!

• Baseline: 36.59s → FastCollect + filter: 3.80s

Key Takeaways:

• ✅ 4x faster on full collection (real-world production codebase)
• ✅ 2-2.4x additional speedup with keyword filters
• ✅ Nearly 10x overall when combining both optimizations
• ✅ Production-ready on Django's complex test infrastructure
• ✅ Zero configuration required - works out of the box

📄 See DJANGO_BENCHMARK_RESULTS.md for detailed analysis.

v0.3.0 - Better Integration

Architecture Improvements:

Early initialization:             Collection happens in pytest_configure
File filtering:                   Simplified pytest_ignore_collect hook
Collection overhead:              Reduced hook call complexity
Code quality:                     Cleaner separation of concerns

Performance (comparable to v0.2.0):

• Maintains incremental caching benefits (~5% improvement on warm cache)
• No duplicate collection issues
• Cleaner initialization flow

Key Changes:

• Moved Rust collection from lazy (first pytest_ignore_collect call) to eager (in pytest_configure)
• Simplified pytest_ignore_collect to only use pre-collected data
• Removed custom collector class to avoid pytest hook conflicts
• More predictable and testable architecture

v0.2.0 - Incremental Caching

Incremental Collection Benchmark (500 files, 20 tests/file, 5 files modified):

Cold start (no cache):              3.34s  (baseline)
Warm cache (no changes):            3.18s  (1.05x faster)
Incremental (5 files changed):      3.50s  (cache + reparse 1%)

Cache effectiveness:
  - 100% cache hit rate on unchanged files
  - Only modified files are reparsed
  - ~5% improvement on warm cache
  - Persistent across pytest runs

Cache Statistics (displayed after collection with -v):

FastCollect Cache: 2 files from cache, 0 parsed (100.0% hit rate)

v0.1.0 - File Filtering Only

Synthetic Benchmarks:

• 200 files, 50 tests/file: ~1.01x speedup
• 500 files, 100 tests/file: ~1.01x speedup

Real-World (pandas, 969 test files):

• File filtering: 0.75x (slower due to overhead)

Performance Analysis

Why Limited Speedup in Pure Collection?

1. Pytest Import Bottleneck: Even with caching, pytest must import Python modules to get actual function/class objects
2. File Filtering Overhead: The pytest_ignore_collect hook is called for every path
3. Duplicate Work: Both Rust (for discovery) and Python (for import) process files

Where Caching Provides Real Value:

1. ✅ Incremental Workflows: Only reparse modified files (5-10% improvement)
2. ✅ Large Codebases: Faster discovery in deep directory trees
3. ✅ CI/CD Pipelines: Cache persists across runs
4. ✅ Development Workflow: Repeated pytest --collect-only calls
5. ✅ Watch Mode: Quick re-collection when files change

Recent Improvements (v0.2.0)

✅ Incremental Caching - Cache parsed results with file modification tracking

• Persists to .pytest_cache/v/fastcollect/cache.json
• Only reparses files that have changed
• Shows cache statistics after collection
• ~5% improvement on repeated runs

Future Optimizations

To achieve even greater speedup:

1. Direct Item Creation: Create pytest Item objects directly from Rust (complex, see IMPLEMENTATION_NOTES.md)
2. Lazy Loading: Only parse files when their tests are actually executed
3. Better Integration: Use pytest's lower-level APIs to bypass standard collection
4. Parallel Imports: Import Python modules in parallel

Technical Details

Rust Dependencies

• pyo3: Python bindings for Rust
• rustpython-parser: Python AST parser in Rust
• walkdir: Recursive directory traversal
• rayon: Data parallelism library

Python API

from pytest_fastcollect import FastCollector

# Create a collector for a directory
collector = FastCollector("/path/to/tests")

# Collect all tests (basic mode)
results = collector.collect()
# Returns: {"file_path": [{"name": "test_foo", "line": 10, "type": "Function"}, ...]}

# Collect with metadata (includes file mtimes for caching)
metadata = collector.collect_with_metadata()
# Returns: {"file_path": {"mtime": 1234567890.0, "items": [...]}}

Development

Building from Source

# Development build (faster compilation, slower runtime)
maturin develop

# Release build (slower compilation, faster runtime)
maturin develop --release

# Build wheel
maturin build --release

Running Tests

# Run sample tests
pytest tests/sample_tests -v

# Run with fast collection disabled
pytest tests/sample_tests --no-fast-collect -v

# Collect only (no execution)
pytest tests/sample_tests --collect-only

# View cache statistics
pytest tests/sample_tests --collect-only -v

Running Benchmarks

# Synthetic benchmark with custom parameters
python benchmark.py --synthetic --num-files 200 --tests-per-file 100

# Incremental caching benchmark (shows cache effectiveness)
python benchmark_incremental.py

# Benchmark on a real project
python benchmark.py --project /path/to/project

# Run all benchmarks
python benchmark.py --all

Cache Management

# Clear the cache
pytest --fastcollect-clear-cache --collect-only

# Disable caching for one run
pytest --no-fastcollect-cache

# View cache contents
cat .pytest_cache/v/fastcollect/cache.json

Contributing

Contributions are welcome! Areas for improvement:

1. Performance Optimization: Implement direct Item creation from Rust data
2. Advanced Caching: Add file content hashing for more reliable cache validation
3. Test Discovery: Support more complex test patterns (fixtures, parameterization)
4. Configuration: Add support for custom test patterns and ignore rules
5. Documentation: Add more examples and use cases

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Built with PyO3 for seamless Rust-Python integration
• Uses RustPython Parser for Python AST parsing
• Inspired by the need for faster test collection in large Python codebases

Technical Notes

Why Rust?

• Speed: Rust's zero-cost abstractions and lack of GIL make it ideal for CPU-intensive parsing
• Parallelism: Rayon makes it trivial to parse files in parallel
• Safety: Rust's type system ensures memory safety without garbage collection overhead

Current Limitations

1. Limited Pure Collection Speedup: Pytest still needs to import modules (~5% improvement)
2. Simple Test Detection: Only detects test_* functions and Test* classes
3. No Fixture Support: Doesn't analyze pytest fixtures or dependencies
4. No Parametrization: Doesn't expand parametrized tests

Changelog

v0.3.0 (Current)

• 🏗️ Better Integration: Refactored plugin architecture for cleaner code
• ⚡ Early initialization in pytest_configure instead of lazy loading
• 🔧 Simplified pytest_ignore_collect hook to only use cached data
• 🐛 Fixed duplicate collection issues from custom collector conflicts
• 📊 Maintains all caching benefits from v0.2.0
• 🧹 Cleaner separation of concerns and more predictable behavior

v0.2.0

• ✨ Added incremental caching with file modification tracking
• 💾 Cache persists to .pytest_cache/v/fastcollect/cache.json
• 📊 Shows cache statistics after collection
• 🚀 ~5% improvement on repeated runs with warm cache
• 📝 Added benchmark_incremental.py for cache effectiveness testing

v0.1.0

• 🎉 Initial release
• 🦀 Rust-based parallel AST parsing
• 🎯 File filtering via pytest_ignore_collect hook
• ⚡ Parallel processing with Rayon
• 📚 Comprehensive documentation

Contact

For issues, questions, or contributions, please open an issue on GitHub.