pytest-cream 0.1.4

The cream of test execution - smooth pytest workflows with intelligent orchestration

pytest-cream

The cream of test execution - smooth pytest workflows with intelligent orchestration.

pytest-cream is a Python package designed to streamline the process of fetching tests from repositories, running them with advanced dependency management, and gathering statistics for parallel execution.

Features

• Fetch tests from remote repositories or local directories
• Advanced dependency management with custom installation commands
• Uses uv by default for faster and more reliable installations
• Support for uv, pip, poetry, and other package managers
• Python version control for project compatibility
• Parallel test execution with intelligent orchestration
• Duration statistics and performance optimization
• Environment isolation for complex projects (Rust extensions, etc.)

Installation

pip install pytest-cream

Usage

Quick Start (Automated Setup)

The easiest way to get started is with the init command, which fetches tests and installs dependencies in one step:

# Initialize: fetch tests and install dependencies from the same repository
pytest-cream init --repo owner/project --branch main \
  --install-repo owner/project --install-mode editable \
  --ws ~/workspace

# Then profile tests to collect duration statistics
pytest-cream profile --ws ~/workspace/owner_project_main --python python3.11

# Finally run tests with intelligent orchestration
pytest-cream run --ws ~/workspace/owner_project_main --python python3.11 --threshold 0.5

Step-by-Step Workflow (Manual Install)

For projects requiring custom installation steps, handle dependencies manually:

# Step 1: Fetch tests from repository
pytest-cream fetch --repo owner/project --branch main --ws ~/my-workspace

# Step 2: Install dependencies manually
cd ~/my-workspace/owner_project_main
pip install -e .  # or your preferred installation method

# Step 3: Profile tests - run all tests once to collect duration statistics
pytest-cream profile --ws ~/my-workspace/owner_project_main --python python3.11

# Step 4: Run tests with intelligent orchestration
pytest-cream run --ws ~/my-workspace/owner_project_main --python python3.11 --threshold 0.5

# (Optional) Step 5: Filter tests by duration threshold for analysis
pytest-cream filter --ws ~/my-workspace/owner_project_main --threshold 0.5

# All files (tests, logs, results) are stored in the same workspace for easy access

Basic Usage Examples

# Initialize with automatic dependency installation (uses uv by default)
pytest-cream init --repo owner/project --branch main \
  --install-repo owner/project --install-mode editable \
  --ws ~/workspace

# Then profile and run tests
pytest-cream profile --ws ~/workspace/owner_project_main --python python3.11
pytest-cream run --ws ~/workspace/owner_project_main --python python3.11

# Use pip instead of uv (if needed for compatibility)
pytest-cream init --repo owner/project --branch main \
  --install-repo owner/project \
  --install-mode editable \
  --use-pip \
  --ws ~/workspace

# Use custom installation commands for complex projects
pytest-cream init --repo owner/project --branch main \
  --install-repo owner/project \
  --install-cmd "uv sync --extra cpu" \
  --ws ~/workspace

Advanced Usage Examples

# Multiple installation commands with fallback
pytest-cream init --repo owner/complex-project --branch main \
  --install-repo owner/complex-project \
  --install-cmd "uv sync --extra gpu; pip install torch" \
  --install-fallback \
  --ws ~/workspace

# Then profile and run
pytest-cream profile --ws ~/workspace/owner_complex-project_main --python python3.11
pytest-cream run --ws ~/workspace/owner_complex-project_main --python python3.11 \
  --exclude-tests "test_slow.py,test_integration/"

Command Line Options

Core Commands

init - Initialize Workspace

Fetch tests from a repository and optionally install dependencies. This is the first step in the workflow.

pytest-cream init --repo owner/project --branch main \
  --install-repo owner/project --install-mode editable \
  --ws ~/workspace

Note: After running init, you need to run profile and then run to actually execute tests.

fetch - Fetch Tests

Clone a repository to a workspace for testing.

pytest-cream fetch --repo owner/project --branch main --ws ~/workspace

Required Parameters:

• --repo: Repository to fetch (format: 'owner/repo' or GitHub URL)
• --ws / --workspace: Directory where the repository will be cloned

Optional Parameters:

• --branch: Git branch to checkout (default: main)

Output: Creates a directory in workspace named owner_project_branch/ containing the cloned repository.

profile - Profile Tests

Run all tests once to collect duration statistics. This is essential before using the run command.

pytest-cream profile --ws ~/workspace/owner_project_main --python python3.11

Required Parameters:

• --ws / --workspace: Workspace directory containing the tests (from fetch command)

Optional Parameters:

• --python: Python executable to use (e.g., python3.11)
• --tests-dir: Subdirectory containing tests (default: tests)
• --output: Output filename for duration log (default: test_durations.log)

Output: Creates test_durations.log in the workspace directory.

filter - Filter Tests by Duration

Parse duration logs and categorize tests as long/short.

pytest-cream filter --ws ~/workspace/owner_project_main --threshold 0.5

Required Parameters:

• --ws / --workspace: Workspace directory containing duration logs
• --threshold: Duration threshold in seconds

Optional Parameters:

• --input: Input duration log filename (default: test_durations.log)
• --output: Output filename for filtered results (default: filtered_tests.txt)

Output: Creates filtered_tests.txt in the workspace directory.

run - Run Tests with Orchestration

Execute tests with intelligent orchestration based on duration analysis. Long tests run sequentially, short tests run in parallel.

pytest-cream run --ws ~/workspace/owner_project_main --python python3.11 --threshold 0.5

Required Parameters:

• --ws / --workspace: Workspace directory containing the tests and duration logs

Optional Parameters:

• --python: Python executable to use (e.g., python3.11)
• --durations: Duration log file (default: test_durations.log)
• --threshold: Duration threshold in seconds for splitting long/short tests (default: 0.5)
• --workers: Number of parallel workers for short tests (default: 4)
• --limit: Maximum number of tests to run
• --exclude-tests: Comma-separated patterns of test files/directories to exclude

Output: Runs tests with intelligent orchestration and displays results.

init Options

For the workspace initialization command, these options are available:

Basic Options:

• --repo: Repository to fetch (owner/repo or URL) - Required
• --branch: Branch to checkout (default: main)
• --ws / --workspace: Directory to use as workspace - Required

Installation Options:

• --install-repo: Repository to install dependencies from (same as --repo typically)
• --install-branch: Branch to install from (default: same as --branch)
• --install-mode: How to install (pip, editable, wheel) - default: pip
• --use-pip: Use pip instead of uv (by default, uv is used for faster installations)
• --install-cmd: Custom install command (overrides --install-mode)
• --install-fallback: Fallback to standard installation if custom command fails
• --python: Python executable for installation (e.g., python3.11)

Workspace Organization

All commands use a shared workspace for easy file management:

~/workspace/
└── owner_project_main/           # Created by fetch command
    ├── src/                       # Your project files
    ├── tests/                     # Test files
    ├── test_durations.log         # Created by run command
    └── filtered_tests.txt         # Created by filter command

This organization makes it easy to:

• Find all test-related files in one place
• Chain commands together using the same workspace
• Review logs and results after test execution

Installation Modes

When using --install-repo in init, you have several options. By default, uv is used for faster and more reliable installations (automatically detects virtual environments and uses appropriate flags).

Standard Modes:

• pip: Direct install from git+https URL (fastest)
• editable: Clone repository and install in editable mode (for development)
• wheel: Build wheel and install (most reproducible)

All standard modes use uv by default. To use pip instead, add --use-pip.

Note: uv automatically detects if you're in a virtual environment. Outside a venv, it uses --system flag; inside a venv, it installs normally.

Using pip Instead of uv:

If you encounter compatibility issues with uv, use --use-pip to fallback to pip:

# Standard install with pip (slower but maximum compatibility)
pytest-cream init --repo owner/project --branch main \
  --install-repo owner/project \
  --install-mode pip \
  --use-pip \
  --ws ~/workspace

# Editable install with pip
pytest-cream init --repo owner/project --branch main \
  --install-repo owner/project \
  --install-mode editable \
  --use-pip \
  --ws ~/workspace

Custom Commands:

Use --install-cmd for complex dependency management that standard modes can't handle:

# uv with extras
--install-cmd "uv sync --extra cpu"

# Poetry with multiple extras
--install-cmd "poetry install --extras dev,test"

# pip with specific requirements file
--install-cmd "pip install -r requirements-dev.txt"

# Conda environment setup
--install-cmd "conda env update -f environment.yml"

Note: --install-cmd overrides --install-mode when provided. The command runs in the cloned repository directory.

Usage Priority:

1. If --install-cmd is provided → Use custom command
2. If --use-pip is provided → Use pip instead of uv
3. Otherwise → Use uv with standard --install-mode (pip/editable/wheel)

Note: uv is used by default for faster installations. Install uv with: pip install uv or curl -LsSf https://astral.sh/uv/install.sh | sh

Troubleshooting

Python Version Issues

Some projects require specific Python versions, especially those with Rust extensions or compiled components:

# Use Python 3.10 for projects with Rust extensions
pytest-cream init \
  --repo owner/rust-project \
  --python python3.10 \
  --ws ~/workspace

# Use Python 3.11 for newer projects
pytest-cream init \
  --repo owner/modern-project \
  --python python3.11 \
  --ws ~/workspace

Common Python version issues:

• Rust extensions often require Python 3.10 or specific versions
• Some packages may not be compatible with Python 3.13+
• Use --python to specify the exact Python executable to use

Excluding Problematic Tests

When some tests fail due to missing dependencies or environment issues, exclude them using the run command:

# Exclude specific test files during test execution
pytest-cream run --ws ~/workspace/owner_project_main \
  --exclude-tests "test_integration.py,test_slow.py" \
  --python python3.11

# Exclude test directories
pytest-cream run --ws ~/workspace/owner_project_main \
  --exclude-tests "tests/integration,tests/gpu" \
  --python python3.11

Pip Issues

If you encounter "No module named pip" errors or issues with uv, try using the --use-pip flag to use pip instead of uv:

pytest-cream init --repo owner/repo --install-repo owner/repo \
  --use-pip --install-mode editable --python python3.11 --ws ~/workspace

By default, pytest-cream uses uv for faster and more reliable installations.

Custom Command Issues

If your --install-cmd fails:

1. Check the command works manually:

   git clone https://github.com/owner/repo
   cd repo
   uv sync --extra cpu  # or your command
   

2. Common fixes:

   • For uv: Ensure pyproject.toml has the required extras defined
   • For Poetry: Make sure pyproject.toml exists and has proper dependencies
   • For build errors: Check if system dependencies (like Rust, C compilers) are needed

3. Debugging: Check the workspace directory for logs and cloned repository to investigate build issues

Complete Workflow Examples

Example 1: Step-by-Step Testing

Execute each phase separately for maximum control:

# Set up variables for easier reuse
WORKSPACE=~/my-test-workspace
REPO=owner/my-project
BRANCH=main

# Step 1: Initialize - fetch the repository and install dependencies
pytest-cream init --repo $REPO --branch $BRANCH \
  --install-repo $REPO --install-mode editable \
  --ws $WORKSPACE

# Step 2: Profile - collect test duration statistics
pytest-cream profile --ws $WORKSPACE/owner_my-project_$BRANCH --python python3.11

# Step 3: Run - execute tests with intelligent orchestration
pytest-cream run --ws $WORKSPACE/owner_my-project_$BRANCH \
  --python python3.11 --threshold 0.5

# Step 4: (Optional) Filter - analyze test durations
pytest-cream filter --ws $WORKSPACE/owner_my-project_$BRANCH --threshold 0.5

# Step 5: Review results
echo "Tests are in: $WORKSPACE/owner_my-project_$BRANCH"
cat $WORKSPACE/owner_my-project_$BRANCH/filtered_tests.txt

Example 2: Quick Testing

For rapid testing with automated setup:

# Initialize workspace
pytest-cream init \
  --repo owner/project \
  --branch develop \
  --install-repo owner/project \
  --install-mode editable \
  --ws ~/quick-workspace

# Profile tests
pytest-cream profile \
  --ws ~/quick-workspace/owner_project_develop \
  --python python3.11

# Run with custom settings
pytest-cream run \
  --ws ~/quick-workspace/owner_project_develop \
  --python python3.11 \
  --threshold 0.3 \
  --workers 8

Example 3: Testing with Custom Dependencies

When your project needs special installation steps:

# Initialize with custom install command
pytest-cream init \
  --repo owner/ml-project \
  --branch main \
  --install-repo owner/ml-project \
  --install-cmd "uv sync --extra gpu,training" \
  --install-fallback \
  --ws ~/ml-workspace

# Profile tests
pytest-cream profile \
  --ws ~/ml-workspace/owner_ml-project_main \
  --python python3.10

# Run tests with exclusions
pytest-cream run \
  --ws ~/ml-workspace/owner_ml-project_main \
  --python python3.10 \
  --exclude-tests "test_slow_training.py,tests/integration/"

Example 4: Development Testing

Test changes in a development branch with editable install:

# Initialize with editable install
pytest-cream init \
  --repo owner/my-library \
  --branch feature-branch \
  --install-repo owner/my-library \
  --install-branch feature-branch \
  --install-mode editable \
  --ws ~/dev-workspace

# Profile and run
pytest-cream profile --ws ~/dev-workspace/owner_my-library_feature-branch --python python3.11
pytest-cream run --ws ~/dev-workspace/owner_my-library_feature-branch --python python3.11

Example 5: CI/CD Pipeline

Suitable for automated testing environments:

#!/bin/bash
set -e  # Exit on error

# Initialize workspace
pytest-cream init \
  --repo $CI_REPO \
  --branch $CI_BRANCH \
  --install-repo $CI_REPO \
  --install-mode pip \
  --ws /tmp/ci-workspace-$CI_JOB_ID

WORKSPACE="/tmp/ci-workspace-$CI_JOB_ID/$(echo $CI_REPO | tr '/' '_')_$CI_BRANCH"

# Profile tests
pytest-cream profile --ws $WORKSPACE --python python3.11

# Run tests and capture results
pytest-cream run \
  --ws $WORKSPACE \
  --python python3.11 \
  --exclude-tests "tests/manual/,test_interactive.py"

# Check if tests passed
if [ $? -eq 0 ]; then
  echo "✅ All tests passed!"
  exit 0
else
  echo "❌ Tests failed!"
  exit 1
fi

Tips and Best Practices

1. Always specify --python: Avoid "python command not found" errors by explicitly setting the Python executable (e.g., --python python3.11)

2. Use meaningful workspace names: Organize workspaces by project or purpose (e.g., ~/workspaces/myproject-dev, ~/workspaces/myproject-prod)

3. Keep workspaces outside project directories: Store workspaces in a separate location (e.g., ~/pytest-cream-workspaces/) to avoid pytest collection issues. Never create workspaces inside your pytest-cream installation directory.

4. Reuse workspaces for iteration: When testing the same project multiple times, you can reuse the workspace or let pytest-cream create timestamped directories

5. Check workspace contents: After running commands, inspect the workspace directory to review logs, test results, and any generated files

6. Exclude flaky tests: Use --exclude-tests with the run command to skip tests that are environment-dependent or have external dependencies

7. Use init for setup: The init command handles fetching and installing, then use profile and run for test execution

8. Clean up old workspaces: Periodically remove old workspace directories to free up disk space

9. Profile before running: Always run profile before run to collect test duration statistics for intelligent orchestration

License

MIT License