pytest-proofy

Pytest plugin for Proofy test reporting with real-time results and rich metadata support.

Features

• Multiple Reporting Modes: Live, Lazy, and Batch reporting
• Rich Metadata: Decorators for test description, severity, and custom attributes
• Attachment Support: Add screenshots, logs, and other files to test results
• Flexible Configuration: CLI, environment variables, and pytest.ini support
• Local Backup: Automatic fallback to local JSON export

Installation

pip install pytest-proofy

Quick Start

Basic Usage

pytest --proofy \
       --proofy-token YOUR_TOKEN \
       --proofy-project-id 123

Note: The --proofy flag is required to activate the Proofy plugin. Without it, the plugin will not register or report test results.

Configuration

Command Line Options

# Activation
--proofy                            # Enable Proofy plugin (required)

# Core options
--proofy-mode {live,batch,lazy}     # Reporting mode
--proofy-api-base URL               # Proofy API base URL
--proofy-token TOKEN                # API authentication token
--proofy-project-id ID              # Project ID

# Run options
--proofy-run-id ID                  # Existing run ID to append to
--proofy-run-name NAME              # Custom run name
--proofy-run-attributes ATTRS       # Custom run attributes. Repeatable flag; accepts "k=v"

# Batch options
--proofy-batch-size N               # Results per batch (default: 10)

# Output options
--proofy-output-dir DIR             # Local backup directory
--proofy-backup                     # Create local backup files

Environment Variables

export PROOFY_MODE=live
export PROOFY_API_BASE=https://api.proofy.dev
export PROOFY_TOKEN=your-token-here
export PROOFY_PROJECT_ID=123

pytest.ini Configuration

[pytest]
proofy = true                       # Enable Proofy plugin
proofy_mode = lazy
proofy_api_base = https://api.proofy.dev
proofy_token = your-token-here
proofy_project_id = 123
proofy_batch_size = 20
proofy_output_dir = test-artifacts
proofy_run_attributes =
    env=staging
    version=1.2.3
    team=backend

Note: When proofy = true is set in pytest.ini, you don't need to use the --proofy flag. CLI flag has priority over ini configuration.

Reporting Modes

Live Mode (Default)

Real-time test reporting with immediate server updates:

pytest --proofy --proofy-mode live

• Creates test result when test starts (IN_PROGRESS status)
• Updates result when test finishes with final outcome
• Uploads attachments immediately
• Best for interactive development and debugging

Lazy Mode

Sends complete results after test execution:

pytest --proofy --proofy-mode lazy

• Collects results during execution
• Sends all results in batches at session end
• Best for CI/CD environments

Batch Mode

Groups results and sends in configurable batches:

pytest --proofy --proofy-mode batch --proofy-batch-size 50

• Collects results during execution
• Sends results in batches
• Optimized for large test suites
• Configurable batch size

Run Attributes

Run attributes allow you to add metadata to your test runs, such as environment information, version numbers, and other custom data. Proofy automatically collects system information (Python version, OS, framework version) and allows you to add custom attributes.

Automatic System Attributes

The following attributes are automatically collected for every run:

• python_version - Python version (e.g., "3.11.0")
• platform - Platform details (e.g., "macOS-14.0-arm64")
• framework - Test framework (e.g., "pytest")
• framework_version - Framework version (e.g., "7.4.0")

Adding Custom Run Attributes

Via Command Line

pytest --proofy \
  --proofy-run-attributes env=prod \
  --proofy-run-attributes version=1.2 \
  --proofy-run-attributes team=qa

Via Environment Variable

export PROOFY_RUN_ATTRIBUTES="environment=staging,version=2.0.0"
pytest --proofy

Via pytest.ini

[pytest]
proofy_run_attributes =
    environment=development
    team=backend

Via conftest.py

# conftest.py
import proofy

def pytest_sessionstart(session):
    """Set run attributes at session start."""
    proofy.add_run_attributes(
        environment="staging",
        version="1.2.3",
        build_number="456",
        branch="feature/new-api"
    )

Via Runtime API in Tests

import proofy

def test_example():
    # You can also set run attributes from within tests
    # (though this is less common - usually set at session start)
    proofy.set_run_attribute("custom_key", "custom_value")
    proofy.add_run_attributes(
        environment="production",
        region="us-east-1"
    )

    # Get all run attributes
    attrs = proofy.get_run_attributes()
    assert "environment" in attrs

Using Decorators and Runtime API

Decorators

from proofy import name, description, severity, attributes

@name("User Login Test")
@description("Validates user authentication with valid credentials")
@severity("critical")
@attributes(component="auth", browser="chrome")
def test_user_login():
    # Test implementation
    assert login("user", "pass") == True

Runtime API

from proofy import (
    set_name, set_description, set_severity,
    add_attributes, add_attachment, ArtifactType
)


def test_dynamic_metadata():
    set_name("Dynamic Test Name")
    set_description("This description is set at runtime")
    set_severity("high")

    # Test logic here
    result = perform_test()

    if result.screenshot:
        add_attachment(
            result.screenshot,
            name="test_screenshot",
            mime_type="image/png",
            artifact_type=ArtifactType.SCREENSHOT
        )

    add_attributes(
        execution_time=result.duration,
        environment="staging"
    )

Attachments

Add files to test results for better debugging:

from proofy import add_attachment, ArtifactType


def test_with_attachments():
    # Your test code
    take_screenshot("failure.png")
    save_logs("test.log")

    # Add attachments
    add_attachment("failure.png", name="Failure Screenshot", artifact_type=ArtifactType.SCREENSHOT)
    add_attachment("test.log", name="Test Logs", mime_type="text/plain")

Troubleshooting

Common Issues

1. Authentication Errors

   # Verify token is correct
   curl -H "Authorization: Bearer YOUR_TOKEN" https://api.proofy.dev/health
   

2. Large Test Suites

   # Use batch mode with larger batches
   pytest --proofy --proofy-mode batch --proofy-batch-size 100
   

Debug Mode

pytest --proofy --proofy-mode lazy -v -s

Local Backup

pytest --proofy --proofy-backup --proofy-output-dir ./test-results

Status mappings:

• pytest passed → PASSED (1)
• pytest failed → FAILED (2)
• pytest error → BROKEN (3)
• pytest skipped → SKIPPED (4)

Development

# In monorepo root
uv venv .venv && source .venv/bin/activate
uv pip install -e ../proofy-commons -e .[dev]
uv run -q pytest -n auto

License

Apache-2.0 — see LICENSE file for details.