testdino 1.0.24

CLI tool for uploading Playwright test reports to TestDino platform with TestDino storage support

testdino

TestDino CLI - Cache test metadata and upload Playwright test reports to TestDino platform

Quick Start

⚠️ Important: The testdino upload command requires a JSON report to be generated during test execution. Make sure to:

1. Install required plugins: pip install pytest-playwright-json pytest-html
2. Run your tests with --playwright-json flag: pytest --playwright-json=test-results/report.json
3. Optionally generate HTML reports: pytest --html=test-results/index.html --self-contained-html

Cache Command

# Cache test execution metadata after Playwright runs
testdino cache --token="your-api-token"

# With custom directory (recommended when test results are in a specific folder)
testdino cache --working-dir test-results --token="your-api-token"

# With verbose logging
testdino cache --working-dir test-results --verbose --token="your-api-token"

Last Failed Command

# Get last failed test cases for Playwright reruns
testdino last-failed --token="your-api-token"

# Get failed tests for specific shard
testdino last-failed --shard="2/5" --token="your-api-token"

# Run only last failed tests
pytest $(testdino last-failed --token="your-api-token")

Upload Command

# Upload test reports with attachments
testdino upload ./test-results --token="your-api-token"

# Upload with environment tag
testdino upload ./test-results --environment="staging" --token="your-api-token"

# Upload all attachments
testdino upload ./test-results --token="your-api-token" --upload-full-json

Features

• Test Metadata Caching - Store test execution metadata after Playwright runs
• Last Failed Tests - Retrieve and rerun only failed tests for faster CI/CD pipelines
• Shard Support - Get failed tests for specific test shards (e.g., shard 2 of 5)
• Environment Tagging - Tag uploads with environment labels (staging, production, qa)
• Zero Configuration - Auto-discovers Playwright reports and configuration
• Smart Shard Detection - Automatically detects Playwright shard information
• CI/CD Ready - Works seamlessly with GitHub Actions, GitLab CI, Jenkins, Azure DevOps
• Secure Authentication - Token-based API authentication
• Usage Limit Handling - Clear error messages with upgrade guidance
• Last Failed Tests - Retrieve and rerun only failed tests for faster CI/CD pipelines
• Zero Configuration - Auto-discovers Playwright reports and configuration
• Smart Shard Detection - Automatically detects Playwright shard information
• CI/CD Ready - Works seamlessly with GitHub Actions, GitLab CI, Jenkins, Azure DevOps
• Secure Authentication - Token-based API authentication

Commands

Cache Command

Store test execution metadata after Playwright runs.

# Basic usage
testdino cache --token="your-token"

# With custom working directory
testdino cache --working-dir ./test-results --token="your-token"

# With verbose logging
testdino cache --verbose --token="your-token"

Options:

Option	Description	Default
--working-dir <path>	Directory to scan for test results	Current directory
--cache-id <value>	Custom cache ID override	Auto-detected
-t, --token <value>	TestDino API token	Required
-v, --verbose	Enable verbose logging	false

Last Failed Command

Retrieve cached test failures for intelligent reruns.

# Basic usage
testdino last-failed --token="your-token"

# Run only last failed tests
pytest $(testdino last-failed --token="your-token")

# With custom branch and commit
testdino last-failed --branch="main" --commit="abc123" --token="your-token"

Options:

Option	Description	Default
--cache-id <value>	Custom cache ID override	Auto-detected
--branch <value>	Custom branch name override	Auto-detected
--commit <value>	Custom commit hash override	Auto-detected
-t, --token <value>	TestDino API token	Required
-v, --verbose	Enable verbose logging	false

Upload Command

Upload test reports with attachments.

⚠️ Important: The testdino upload command requires a JSON report to function. You must:

1. Install required plugins: pip install pytest-playwright-json pytest-html
2. Run your tests with the --playwright-json flag: pytest --playwright-json=test-results/report.json
3. Optionally generate HTML reports: pytest --html=test-results/index.html --self-contained-html

# Basic upload
testdino upload ./test-results --token="your-token"

# Upload with attachments
testdino upload ./test-results --token="your-token" --upload-images --upload-videos

# Upload all attachments
testdino upload ./test-results --token="your-token" --upload-full-json

Options:

Option	Description
<report-directory>	Directory containing Playwright reports (required)
-t, --token <value>	TestDino API token (required)
--upload-images	Upload image attachments
--upload-videos	Upload video attachments
--upload-html	Upload HTML reports
--upload-traces	Upload trace files
--upload-files	Upload file attachments (.md, .pdf, .txt, .log)
--upload-full-json	Upload all attachments
-v, --verbose	Enable verbose logging

Environment Variables

export TESTDINO_TOKEN="your-api-token"
export TESTDINO_TARGET_ENV="staging"  # Optional: Set target environment

CI/CD Integration

GitHub Actions

Basic Example:

name: Playwright Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          pip install pytest pytest-playwright pytest-playwright-json pytest-html testdino
          playwright install chromium --with-deps

      - name: Run Playwright Tests
        shell: bash
        env:
          TESTDINO_TOKEN: ${{ secrets.TESTDINO_TOKEN }}
        run: |
          mkdir -p test-results

          # Case 1: Re-run failed jobs → run only failed tests
          if [[ "${{ github.run_attempt }}" -gt 1 ]]; then
            testdino last-failed --token="$TESTDINO_TOKEN" > last-failed-flags.txt
            FAILED_TESTS="$(cat last-failed-flags.txt | tail -1)"

            if [[ -z "$FAILED_TESTS" ]]; then
              exit 0
            fi

            # IMPORTANT: Use eval to preserve quotes in the -k expression
            eval "pytest $FAILED_TESTS --playwright-json=test-results/report.json --html=test-results/index.html --self-contained-html -p no:selenium -v" || true
            exit 0
          fi

          # Case 2: Normal execution (first run)
          pytest \
            --playwright-json=test-results/report.json \
            --html=test-results/index.html \
            --self-contained-html \
            -p no:selenium \
            -v || true

      - name: Cache rerun metadata
        if: always()
        run: testdino cache --working-dir test-results --token="${{ secrets.TESTDINO_TOKEN }}" -v

      - name: Upload test reports
        if: always()
        run: testdino upload ./test-results --token="${{ secrets.TESTDINO_TOKEN }}"

Advanced Example with Sharding and Intelligent Reruns:

Note: Sharding is optional. The basic example above works perfectly without sharding. Use the advanced example below only if you want to run tests in parallel across multiple shards for faster execution. The CLI automatically detects shard configuration from environment variables (SHARD_INDEX, SHARD_TOTAL) or Playwright config, and defaults to a single shard (1/1) if none is detected.

name: Playwright Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4, 5]
        shardTotal: [5]
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install pytest pytest-playwright pytest-playwright-json pytest-html pytest-xdist pytest-shard
          pip install testdino
          playwright install --with-deps chromium

      - name: Run Playwright Tests
        shell: bash
        env:
          TESTDINO_TOKEN: ${{ secrets.TESTDINO_TOKEN }}
          SHARD_INDEX: ${{ matrix.shardIndex }}
          SHARD_TOTAL: ${{ matrix.shardTotal }}
        run: |
          mkdir -p test-results

          # Case 1: Re-run failed jobs → run only failed tests
          if [[ "${{ github.run_attempt }}" -gt 1 ]]; then

            testdino last-failed --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --token="$TESTDINO_TOKEN" > last-failed-flags.txt
            FAILED_TESTS="$(cat last-failed-flags.txt | tail -1)"

            if [[ -z "$FAILED_TESTS" ]]; then
              exit 0
            fi

            # IMPORTANT: Use eval to preserve quotes in the -k expression
            # pytest-shard uses 0-indexed shard IDs, so subtract 1 from SHARD_INDEX
            # Use shard-specific report name
            SHARD_ID=$(( $SHARD_INDEX - 1 ))
            eval "pytest $FAILED_TESTS --shard-id=$SHARD_ID --num-shards=$SHARD_TOTAL --playwright-json=test-results/report-${{ matrix.shardIndex }}.json --html=test-results/index.html --self-contained-html -p no:selenium -v" || true
            exit 0
          fi

          # Case 2: Normal execution (first run) with sharding
          # pytest-shard uses 0-indexed shard IDs, so subtract 1 from SHARD_INDEX
          SHARD_ID=$(( $SHARD_INDEX - 1 ))
          # Use shard-specific report name to prevent overwriting when artifacts are merged
          pytest \
            --shard-id=$SHARD_ID \
            --num-shards=$SHARD_TOTAL \
            --playwright-json=test-results/report-${{ matrix.shardIndex }}.json \
            --html=test-results/index.html \
            --self-contained-html \
            -p no:selenium \
            -v || true

      - name: Upload HTML report
        if: ${{ !cancelled() }}
        uses: actions/upload-artifact@v4
        with:
          name: test-results-${{ matrix.shardIndex }}
          path: test-results/
          retention-days: 1

      - name: Cache testdino last failed metadata
        if: always()
        env:
          TESTDINO_TOKEN: ${{ secrets.TESTDINO_TOKEN }}
          SHARD_INDEX: ${{ matrix.shardIndex }}
          SHARD_TOTAL: ${{ matrix.shardTotal }}
        run: |
          if [ -n "$TESTDINO_TOKEN" ]; then
            testdino cache --working-dir test-results --token="$TESTDINO_TOKEN"
          fi

  upload-to-testdino:
    name: Upload Test Results to TestDino
    if: ${{ always() }}
    needs: test
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install testdino
          pip install pytest-playwright-json

      - name: Download all test results
        uses: actions/download-artifact@v4
        with:
          pattern: test-results-*
          merge-multiple: true
          path: combined-test-results

      - name: Merge JSON reports from all shards
        run: |
          python3 -m pytest_playwright_json.merge -d combined-test-results -o combined-test-results/report.json -v

      - name: Upload to TestDino
        env:
          TESTDINO_TOKEN: ${{ secrets.TESTDINO_TOKEN }}
        run: |
          if [ -z "$TESTDINO_TOKEN" ]; then
            echo "TESTDINO_TOKEN not set, skipping upload"
            exit 0
          fi
          if [ -d "combined-test-results" ] && [ -n "$(ls -A combined-test-results 2>/dev/null)" ]; then
            # Basic upload - uploads merged JSON report
            testdino upload ./combined-test-results --token="$TESTDINO_TOKEN"
          else
            echo "No test results found to upload"
          fi

GitLab CI

image: python:3.11

stages:
  - test

playwright-tests:
  stage: test
  script:
    - pip install pytest pytest-playwright pytest-playwright-json pytest-html testdino
    - playwright install chromium --with-deps
    - pytest --playwright-json=test-results/report.json --html=test-results/index.html --self-contained-html
    - testdino upload ./test-results --token="$TESTDINO_TOKEN"
  when: always

Jenkins

pipeline {
    agent any

    environment {
        TESTDINO_TOKEN = credentials('testdino-token')
    }

    stages {
        stage('Test') {
            steps {
                sh 'pip install pytest pytest-playwright pytest-playwright-json pytest-html testdino'
                sh 'playwright install chromium --with-deps'
                sh 'pytest --playwright-json=test-results/report.json --html=test-results/index.html --self-contained-html'
                sh 'testdino upload ./test-results --token="$TESTDINO_TOKEN"'
            }
        }
    }
}

Authentication

Getting Your Token

1. Sign up at TestDino
2. Navigate to Settings > API Tokens
3. Generate a new token
4. Store it securely in your CI/CD secrets

Token Format:

trx_{environment}_{64-character-hex-string}

Security Best Practices:

• Never commit tokens to version control
• Use environment variables or CI/CD secrets
• Rotate tokens regularly

Examples

Basic Workflow

# Run tests with JSON and HTML reports
pytest \
  --playwright-json=test-results/report.json \
  --html=test-results/index.html \
  --self-contained-html

# Cache metadata (specify directory where test results are located)
testdino cache --working-dir test-results --token="your-token"

Intelligent Test Reruns

# Run tests with JSON and HTML reports, then cache results
pytest \
  --playwright-json=test-results/report.json \
  --html=test-results/index.html \
  --self-contained-html
testdino cache --working-dir test-results --token="your-token"

# On next run, execute only previously failed tests
pytest $(testdino last-failed --token="your-token")

Complete CI/CD Workflow

# Run all tests with JSON and HTML reports
pytest \
  --playwright-json=test-results/report.json \
  --html=test-results/index.html \
  --self-contained-html

# Cache test metadata (specify directory where test results are located)
testdino cache --working-dir test-results --token="$TESTDINO_TOKEN"

# Rerun only failed tests
if [ $? -ne 0 ]; then
  FAILED=$(testdino last-failed --token="$TESTDINO_TOKEN")
  [ -n "$FAILED" ] && pytest $FAILED
fi

Support

• Documentation: docs.testdino.com
• Issues: GitHub Issues
• Email: [e-mail]

---

Made with love by the TestDino team