smarttest-cli 0.1.3

SmartTest CLI - Execute test scenarios with secure credential handling

SmartTest CLI

Enterprise-ready CLI for executing test scenarios with secure, zero-credential-exposure architecture.

Features

🔒 Zero Credential Exposure: Auth tokens never leave your network ⚡ Concurrent Execution: Run up to 5 scenarios simultaneously 🎯 Continue-on-Error: Individual failures don't stop execution 📊 Real-time Progress: Live progress updates with rich terminal output 📄 CI Integration: JUnit XML reports for CI/CD pipelines 🌐 Network Aware: Proxy and custom CA support for enterprise networks

Installation

Install the CLI from PyPI:

pip install smarttest-cli

Get Your PAT Token

1. Visit the SmartTest platform: https://ai-smart-test.vercel.app/
2. Sign up or log in to your account
3. Navigate to Settings → API Tokens
4. Click Generate PAT Token
5. Copy your Personal Access Token (PAT)

What is a PAT Token? A Personal Access Token (PAT) authenticates the CLI with the SmartTest backend. It allows the CLI to fetch test scenarios and submit results securely. This token is separate from any credentials needed to test your APIs.

Quick Start

1. Set Your PAT Token

export SMARTTEST_TOKEN=your_pat_token_here

2. Run Test Scenarios

# Run a specific scenario
python smarttest.py run --scenario-id 123

# Run all scenarios for an endpoint
python smarttest.py run --endpoint-id 456

# Run all scenarios for a system
python smarttest.py run --system-id 789

# With JUnit XML report for CI
python smarttest.py run --system-id 789 --report junit.xml

Configuration

Optional Configuration File

Create .smarttest.yml in your project root for advanced configuration:

# API Configuration
api_url: "https://api.smarttest.com"

# Execution Settings
concurrency: 5
timeout: 30

# Enterprise Network Settings
proxy:
  http_proxy: "http://proxy.company.com:8080"
  https_proxy: "https://proxy.company.com:8080"

tls:
  ca_bundle_path: "/path/to/ca-bundle.pem"
  verify_ssl: true

Configuration Options:

• api_url: SmartTest API endpoint (default: https://api.smarttest.com)
• concurrency: Number of scenarios to run in parallel (default: 5)
• timeout: Request timeout in seconds (default: 30)
• proxy: HTTP/HTTPS proxy settings for corporate networks
• tls: Custom CA bundle and SSL verification options

Authentication

SmartTest Authentication (Required)

The CLI requires a Personal Access Token (PAT) to authenticate with SmartTest:

export SMARTTEST_TOKEN=your_pat_token_here

How to get your PAT token:

1. Visit https://ai-smart-test.vercel.app/
2. Go to Settings → API Tokens
3. Generate a new PAT token
4. Copy and save it securely

Zero-Credential Exposure (Advanced)

When testing APIs that require authentication, SmartTest uses a zero-credential-exposure model to keep your API credentials secure.

How It Works

1. SmartTest backend sends auth config references (metadata only, no actual credentials)
2. CLI resolves credentials locally from your environment variables
3. Credentials never leave your network or reach SmartTest servers
4. Requests are made directly from your environment to the target API

Setting Up Target API Credentials

Only needed if you're testing APIs that require authentication:

# Bearer Token Authentication
export AUTH_CONFIG_123_TOKEN=your_api_bearer_token

# Basic Authentication
export AUTH_CONFIG_456_USERNAME=api_username
export AUTH_CONFIG_456_PASSWORD=api_password

# API Key Authentication
export AUTH_CONFIG_789_API_KEY=your_api_key

Pattern: AUTH_CONFIG_{ID}_{CREDENTIAL_TYPE}

• {ID}: Auth configuration ID from SmartTest dashboard
• {CREDENTIAL_TYPE}: TOKEN, USERNAME, PASSWORD, or API_KEY

Example Scenario:

• You're testing your company's API at https://api.company.com
• The API requires a bearer token for authentication
• SmartTest knows the API needs auth (config ID: 123)
• You set: export AUTH_CONFIG_123_TOKEN=company_bearer_token_xyz
• CLI resolves the token locally and includes it in requests
• SmartTest never sees company_bearer_token_xyz

CI/CD Integration

GitHub Actions

- name: Run SmartTest
  env:
    SMARTTEST_TOKEN: ${{ secrets.SMARTTEST_TOKEN }}
    API_TOKEN: ${{ secrets.API_TOKEN }}
  run: |
    python smarttest.py run --system-id 123 --report junit.xml

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Install SmartTest CLI
        run: pip install smarttest-cli

      - name: Run API Tests
        env:
          # PAT token for SmartTest authentication
          SMARTTEST_TOKEN: ${{ secrets.SMARTTEST_TOKEN }}
        run: |
          smarttest --system-id 123 --report junit.xml

      - name: Publish Test Results
        uses: dorny/test-reporter@v1
        if: always()
        with:
          name: SmartTest Results
          path: junit.xml
          reporter: java-junit

Jenkins

Basic Pipeline:

pipeline {
    agent any
    environment {
        // PAT token for SmartTest authentication
        SMARTTEST_TOKEN = credentials('smarttest-token')
        API_TOKEN = credentials('api-token')
    }
    stages {
        stage('Test') {
            steps {
                sh 'python smarttest.py run --system-id 123 --report junit.xml'
            }
            post {
                always {
                    junit 'junit.xml'
                }
            }
        }
    }
}

Example Output

$ python smarttest.py run --system-id 123
🔍 Found 25 scenarios (3 skipped - no validations)
⚡ Executing scenarios... ✅ 18 passed, ❌ 3 failed, ⚠️ 1 errors [████████████████████████] 22/22 • 0:00:15

Results:
✅ 18 passed
❌ 3 failed (validation errors)
⚠️  1 error (network timeout)

Failed scenarios:
  - user_login_invalid: Expected status 401, got 200
  - payment_process: Response missing required field 'transaction_id'

Error scenarios:
  - webhook_callback: Connection timeout after 30s

Summary: 18/22 scenarios passed (81.8% success rate)
Execution time: 15.2s

Error Handling

The CLI provides comprehensive error classification:

• ✅ Success: HTTP request succeeded, all validations passed
• ❌ Failed: HTTP request succeeded, but validations failed
• ⚠️ Network Timeout: Request timed out
• ⚠️ Network Error: Connection failed
• ⚠️ Auth Error: Authentication resolution failed
• ⚠️ Unknown Error: Unexpected error occurred

Architecture

┌─ Scenario Discovery (API with rate limiting)
├─ Skip scenarios without validations
├─ Concurrent Execution (max 5)
│  ├─ Fetch definition (auth config references only)
│  ├─ Resolve auth locally (zero credential exposure)
│  ├─ Execute HTTP request (with comprehensive error handling)
│  └─ Submit results (continue on any error)
└─ Generate reports and exit

Troubleshooting

Authentication Issues

SmartTest Authentication:

# Verify your PAT token is set
echo $SMARTTEST_TOKEN

# Test connection to SmartTest
curl -H "Authorization: Bearer $SMARTTEST_TOKEN" https://api.smarttest.com/health

Target API Authentication:

# Verify target API credentials are set
echo $AUTH_CONFIG_123_TOKEN

# Check which auth configs your scenarios use in the SmartTest dashboard

Network Issues

# Test with custom config
python smarttest.py run --scenario-id 123 --config .smarttest.yml

# Enable request debugging
SMARTTEST_DEBUG=1 python smarttest.py run --scenario-id 123

Rate Limiting

The CLI automatically handles rate limiting with exponential backoff. If you encounter persistent rate limiting:

1. Reduce concurrency in .smarttest.yml
2. Contact support to increase rate limits
3. Spread execution across longer time periods

Support

• 🌐 Platform: https://ai-smart-test.vercel.app/
• 📚 Documentation: https://ai-smart-test.vercel.app/docs
• 💬 Support: [e-mail]