pytest-cream 0.1.2

The cream of test execution - smooth pytest workflows with intelligent orchestration

pytest-cream

The cream of test execution - smooth pytest workflows with intelligent orchestration.

pytest-cream is a Python package designed to streamline the process of fetching tests from repositories, running them with advanced dependency management, and gathering statistics for parallel execution.

Features

• Fetch tests from remote repositories or local directories
• Advanced dependency management with custom installation commands
• Support for uv, poetry, pip, and other package managers
• Python version control for project compatibility
• Parallel test execution with intelligent orchestration
• Duration statistics and performance optimization
• Environment isolation for complex projects (Rust extensions, etc.)

Installation

pip install pytest-cream

Usage

Quick Start (All-in-One)

The easiest way to use pytest-cream is with the quick-run command, which handles everything automatically:

# Complete workflow: fetch, run, and analyze tests in one command
pytest-cream quick-run --repo owner/project --branch main --python python3.11 --ws ~/workspace

Step-by-Step Workflow (Advanced)

For more control, you can run each step separately using a shared workspace:

# Step 1: Fetch tests from repository
pytest-cream fetch --repo owner/project --branch main --ws ~/my-workspace

# Step 2: Run tests and collect duration statistics
pytest-cream run --ws ~/my-workspace/owner_project_main --python python3.11

# Step 3: Filter tests by duration threshold
pytest-cream filter --ws ~/my-workspace/owner_project_main --threshold 0.5

# All files (tests, logs, results) are stored in the same workspace for easy access

Basic Usage Examples

# Run tests with standard dependency installation (specify Python version to avoid errors)
pytest-cream quick-run --repo owner/project --branch main --python python3.11 --ws ~/workspace

# Use custom installation commands for complex projects
pytest-cream quick-run --repo owner/project --branch main \
  --install-cmd "uv sync --extra cpu" \
  --python python3.10 \
  --ws ~/workspace

Advanced Usage Examples

# Multiple installation commands with fallback
pytest-cream quick-run --repo owner/complex-project --branch main \
  --install-cmd "uv sync --extra gpu; pip install torch" \
  --install-fallback \
  --python python3.11 \
  --ws ~/workspace

# Exclude problematic tests
pytest-cream quick-run --repo owner/project --branch main \
  --exclude-tests "test_slow.py,test_integration/" \
  --python python3.11 \
  --ws ~/workspace

Command Line Options

Core Commands

quick-run - Complete Workflow (Recommended)

Run the entire test workflow in one command: fetch, analyze, and execute tests with intelligent orchestration.

pytest-cream quick-run --repo owner/project --branch main --python python3.11 --ws ~/workspace

fetch - Fetch Tests

Clone a repository to a workspace for testing.

pytest-cream fetch --repo owner/project --branch main --ws ~/workspace

Required Parameters:

• --repo: Repository to fetch (format: 'owner/repo' or GitHub URL)
• --ws / --workspace: Directory where the repository will be cloned

Optional Parameters:

• --branch: Git branch to checkout (default: main)

Output: Creates a directory in workspace named owner_project_branch/ containing the cloned repository.

run - Run Tests

Execute tests from a workspace and collect duration statistics.

pytest-cream run --ws ~/workspace/owner_project_main --python python3.11

Required Parameters:

• --ws / --workspace: Workspace directory containing the tests (from fetch command)

Optional Parameters:

• --python: Python executable to use (e.g., python3.11)
• --tests-dir: Subdirectory containing tests (default: tests)
• --output: Output filename for duration log (default: test_durations.log)

Output: Creates test_durations.log in the workspace directory.

filter - Filter Tests by Duration

Parse duration logs and categorize tests as long/short.

pytest-cream filter --ws ~/workspace/owner_project_main --threshold 0.5

Required Parameters:

• --ws / --workspace: Workspace directory containing duration logs
• --threshold: Duration threshold in seconds

Optional Parameters:

• --input: Input duration log filename (default: test_durations.log)
• --output: Output filename for filtered results (default: filtered_tests.txt)

Output: Creates filtered_tests.txt in the workspace directory.

quick-run Options

For the complete workflow command, additional options are available:

Basic Options:

• --repo: Repository to test (owner/repo or URL) - Required
• --branch: Branch to test (default: main)
• --ws / --workspace: Directory to use as workspace - Required
• --python: Python executable (e.g., python3.11) - Highly recommended
• --threshold: Duration threshold in seconds for splitting long/short tests (default: 0.5)
• --workers: Number of parallel workers for short tests (default: 4)

Installation Options:

• --install-repo: Repository to install dependencies from before running tests
• --install-branch: Branch to install from
• --install-mode: How to install (pip, editable, wheel) - default: pip
• --install-uv: Use uv package manager instead of pip
• --install-cmd: Custom install command (overrides --install-mode)
• --install-fallback: Fallback to standard installation if custom command fails

Test Execution Options:

• --exclude-tests: Comma-separated patterns of test files/directories to exclude

Workspace Organization

All commands use a shared workspace for easy file management:

~/workspace/
└── owner_project_main/           # Created by fetch command
    ├── src/                       # Your project files
    ├── tests/                     # Test files
    ├── test_durations.log         # Created by run command
    └── filtered_tests.txt         # Created by filter command

This organization makes it easy to:

• Find all test-related files in one place
• Chain commands together using the same workspace
• Review logs and results after test execution

Installation Modes

When using --install-repo in quick-run, you have several options:

Standard Modes:

• pip: Direct install from git+https URL (fastest)
• editable: Clone repository and install in editable mode (for development)
• wheel: Build wheel and install (most reproducible)

Custom Commands:

Use --install-cmd for complex dependency management that standard modes can't handle:

# uv with extras
--install-cmd "uv sync --extra cpu"

# Poetry with multiple extras
--install-cmd "poetry install --extras dev,test"

# pip with specific requirements file
--install-cmd "pip install -r requirements-dev.txt"

# Conda environment setup
--install-cmd "conda env update -f environment.yml"

Note: --install-cmd overrides --install-mode when provided. The command runs in the cloned repository directory.

Usage Priority:

1. If --install-cmd is provided → Use custom command
2. If --install-uv is provided → Use uv instead of pip
3. Otherwise → Use standard --install-mode (pip/editable/wheel)

Troubleshooting

Python Version Issues

Some projects require specific Python versions, especially those with Rust extensions or compiled components:

# Use Python 3.10 for projects with Rust extensions
pytest-cream quick-run \
  --repo owner/rust-project \
  --python python3.10 \
  --ws ~/workspace

# Use Python 3.11 for newer projects
pytest-cream quick-run \
  --repo owner/modern-project \
  --python python3.11 \
  --ws ~/workspace

Common Python version issues:

• Rust extensions often require Python 3.10 or specific versions
• Some packages may not be compatible with Python 3.13+
• Use --python to specify the exact Python executable to use

Excluding Problematic Tests

When some tests fail due to missing dependencies or environment issues:

# Exclude specific test files
pytest-cream quick-run --repo owner/project --branch main \
  --exclude-tests "test_integration.py,test_slow.py" \
  --python python3.11 --ws ~/workspace

# Exclude test directories
pytest-cream quick-run --repo owner/project --branch main \
  --exclude-tests "tests/integration,tests/gpu" \
  --python python3.11 --ws ~/workspace

Pip Issues

If you encounter "No module named pip" errors when using virtualenv contexts, try using the --install-uv flag to use uv instead of pip:

pytest-cream quick-run --repo owner/repo --install-repo owner/repo \
  --install-uv --install-mode editable --python python3.11 --ws ~/workspace

This requires uv to be installed on your system.

Custom Command Issues

If your --install-cmd fails:

1. Check the command works manually:

   git clone https://github.com/owner/repo
   cd repo
   uv sync --extra cpu  # or your command
   

2. Common fixes:

   • For uv: Ensure pyproject.toml has the required extras defined
   • For Poetry: Make sure pyproject.toml exists and has proper dependencies
   • For build errors: Check if system dependencies (like Rust, C compilers) are needed

3. Debugging: Check the workspace directory for logs and cloned repository to investigate build issues

Complete Workflow Examples

Example 1: Step-by-Step Testing

Execute each phase separately for maximum control:

# Set up variables for easier reuse
WORKSPACE=~/my-test-workspace
REPO=owner/my-project
BRANCH=main

# Step 1: Fetch the repository
pytest-cream fetch --repo $REPO --branch $BRANCH --ws $WORKSPACE

# Step 2: Run tests and collect durations
pytest-cream run --ws $WORKSPACE/owner_my-project_$BRANCH --python python3.11

# Step 3: Analyze and filter tests
pytest-cream filter --ws $WORKSPACE/owner_my-project_$BRANCH --threshold 0.5

# Step 4: Review results
echo "Tests are in: $WORKSPACE/owner_my-project_$BRANCH"
cat $WORKSPACE/owner_my-project_$BRANCH/filtered_tests.txt

Example 2: Quick Testing with One Command

For rapid testing without manual steps:

pytest-cream quick-run \
  --repo owner/project \
  --branch develop \
  --python python3.11 \
  --threshold 0.3 \
  --workers 8 \
  --ws ~/quick-workspace

Example 3: Testing with Custom Dependencies

When your project needs special installation steps:

pytest-cream quick-run \
  --repo owner/ml-project \
  --branch main \
  --install-cmd "uv sync --extra gpu,training" \
  --install-fallback \
  --python python3.10 \
  --exclude-tests "test_slow_training.py,tests/integration/" \
  --ws ~/ml-workspace

Example 4: Development Testing

Test changes in a development branch with editable install:

pytest-cream quick-run \
  --install-repo owner/my-library \
  --install-branch feature-branch \
  --install-mode editable \
  --repo owner/my-library \
  --branch feature-branch \
  --python python3.11 \
  --ws ~/dev-workspace

Example 5: CI/CD Pipeline

Suitable for automated testing environments:

#!/bin/bash
set -e  # Exit on error

# Run tests and capture results
pytest-cream quick-run \
  --repo $CI_REPO \
  --branch $CI_BRANCH \
  --python python3.11 \
  --exclude-tests "tests/manual/,test_interactive.py" \
  --ws /tmp/ci-workspace-$CI_JOB_ID

# Check if tests passed (quick-run exit code indicates success/failure)
if [ $? -eq 0 ]; then
  echo "✅ All tests passed!"
  exit 0
else
  echo "❌ Tests failed!"
  exit 1
fi

Tips and Best Practices

1. Always specify --python: Avoid "python command not found" errors by explicitly setting the Python executable (e.g., --python python3.11)

2. Use meaningful workspace names: Organize workspaces by project or purpose (e.g., ~/workspaces/myproject-dev, ~/workspaces/myproject-prod)

3. Keep workspaces outside project directories: Store workspaces in a separate location (e.g., ~/pytest-cream-workspaces/) to avoid pytest collection issues. Never create workspaces inside your pytest-cream installation directory.

4. Reuse workspaces for iteration: When testing the same project multiple times, you can reuse the workspace or let pytest-cream create timestamped directories

5. Check workspace contents: After running commands, inspect the workspace directory to review logs, test results, and any generated files

6. Exclude flaky tests: Use --exclude-tests to skip tests that are environment-dependent or have external dependencies

7. Start with quick-run: Use the all-in-one command first, then switch to individual commands if you need more control

8. Clean up old workspaces: Periodically remove old workspace directories to free up disk space

License

MIT License