Highflame Forge

Unified sandbox platform for ML training, model scanning, agent testing, and secure execution.

Highflame Forge provides a single API for creating sandboxed GPU environments across multiple cloud backends (Modal, RunPod, GCP). Use it for training ML models, running security scans, testing AI agents, or any workload that needs isolated execution with GPUs.

Features

• Multi-Backend Support: Modal (primary), RunPod (cheap GPUs), GCP/Vertex AI
• Unified API: Same code works across all backends
• GPU Orchestration: A40, A100, H100, T4, and more
• Training Jobs: Built-in support for ML training with metric streaming
• Hyperparameter Sweeps: Parallel parameter search across multiple sandboxes
• Cost Estimation: Compare pricing across backends before running
• Network Isolation: Configurable network policies for security testing
• Presets: Ready-to-use configurations for common workloads

Installation

Using uv (Recommended)

# Install uv if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install base package
uv pip install highflame-forge

# Install with specific backend support
uv pip install highflame-forge[modal]      # Modal backend
uv pip install highflame-forge[runpod]     # RunPod backend
uv pip install highflame-forge[gcp]        # GCP backend
uv pip install highflame-forge[all]        # All backends

# Or add to your project
uv add highflame-forge
uv add highflame-forge[modal]

Using pip

# Install base package
pip install highflame-forge

# Install with specific backend support
pip install highflame-forge[modal]      # Modal backend
pip install highflame-forge[runpod]     # RunPod backend
pip install highflame-forge[gcp]        # GCP backend
pip install highflame-forge[all]        # All backends

Quick Start

Python API

from highflame_forge import Forge

forge = Forge()

# Create a sandbox with context manager (auto-cleanup)
with forge.sandbox(gpu="A40", memory_gb=32) as sb:
    sb.run_sync("python train.py")
    sb.download_sync("/workspace/model", "./results")

# Async usage
async with forge.sandbox_async(gpu="A40") as sb:
    await sb.run("python train.py")
    await sb.download("/workspace/model", "./results")

Using Presets

from highflame_forge import Forge

forge = Forge()

# Use predefined configurations
with forge.sandbox(preset="training:medium") as sb:
    sb.run_sync("python train.py")

# Available presets:
# - training:small   (T4, 16GB RAM)
# - training:medium  (A40, 32GB RAM)
# - training:large   (A100-80GB, 80GB RAM)
# - training:xlarge  (4x A100-80GB, 320GB RAM)
# - inference:small  (T4, 8GB RAM)
# - inference:large  (A100-40GB, 40GB RAM)
# - security:scan    (CPU only, no network)
# - agent:standard   (CPU, restricted network)

Training Jobs

from highflame_forge import Forge
from highflame_forge.jobs import TrainingJob

forge = Forge()

# Run a training job with automatic metric extraction
job = TrainingJob(
    forge=forge,
    script="train.py",
    preset="training:medium",
    env={"WANDB_PROJECT": "my-project"},
)

result = await job.run()
print(f"Training completed: {result.metrics}")

Hyperparameter Sweeps

from highflame_forge.jobs import HyperparameterSweep

sweep = HyperparameterSweep(
    forge=forge,
    script="train.py",
    preset="training:medium",
    param_space={
        "learning_rate": [1e-4, 1e-3, 1e-2],
        "batch_size": [8, 16, 32],
    },
    max_parallel=4,
)

results = await sweep.run()
print(f"Best config: {results.best_config}")
print(f"Best loss: {results.best_result.metrics.get('loss')}")

Cost Estimation

from highflame_forge import Forge

forge = Forge()

# Compare costs across backends
estimates = forge.estimate_cost(gpu="A40", memory_gb=32, duration_hours=2)

for backend, estimate in estimates.items():
    print(f"{backend}: ${estimate.estimated_total:.2f}")

# Find cheapest backend
cheapest = forge.cheapest_backend(gpu="A40", duration_hours=2)
print(f"Cheapest: {cheapest}")

CLI Usage

# Create a sandbox interactively
forge create --gpu A40 --memory 32

# Run a training job
forge train --script train.py --gpu A40 --backend modal

# Run hyperparameter sweep
forge sweep --script train.py --param learning_rate=1e-4,1e-3 --max-parallel 4

# Estimate costs
forge estimate --gpu A40 --hours 2

# List running sandboxes
forge list

# Terminate a sandbox
forge terminate <sandbox-id>

# Show available GPUs
forge gpus

# List available presets
forge presets

Configuration

Environment Variables

# Modal (primary backend)
MODAL_TOKEN_ID=...
MODAL_TOKEN_SECRET=...

# RunPod
RUNPOD_API_KEY=...

# GCP
GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json
GCP_PROJECT_ID=your-project

# Optional
HF_TOKEN=...              # HuggingFace access
WANDB_API_KEY=...         # Experiment tracking

Backend Selection

from highflame_forge import Forge, BackendSelector

# Use specific backend
forge = Forge(backend="modal")

# Auto-select cheapest backend
forge = Forge(selector=BackendSelector.COST_OPTIMIZED)

# Auto-select fastest startup
forge = Forge(selector=BackendSelector.SPEED_OPTIMIZED)

# Limit to specific backends
forge = Forge(backends=["modal", "runpod"])

Network Policies

from highflame_forge import Forge, SandboxConfig, NetworkConfig

# Fully isolated (for security scanning)
config = SandboxConfig(
    gpu="T4",
    network=NetworkConfig(
        allow_outbound=False,
        allow_internet=False,
    ),
)

# Restricted access (for agents)
config = SandboxConfig(
    gpu="none",
    network=NetworkConfig(
        allow_outbound=True,
        allowed_hosts=[
            "api.anthropic.com",
            "github.com",
            "pypi.org",
        ],
    ),
)

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                      Highflame Forge                            │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  from highflame_forge import Forge                              │
│                                                                 │
│  forge = Forge()  # Auto-selects best backend                   │
│                                                                 │
│  with forge.sandbox(gpu="A40") as sb:                           │
│      sb.run("python train.py")                                  │
│                                                                 │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Backends:                                                      │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐            │
│  │  Modal  │  │ RunPod  │  │   GCP   │  │ Vertex  │            │
│  │(primary)│  │ (cheap) │  │(credits)│  │  (mgd)  │            │
│  └─────────┘  └─────────┘  └─────────┘  └─────────┘            │
│                                                                 │
│  High-Level APIs:                                               │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐             │
│  │TrainingJob  │  │InferenceScan│  │WhiteboxTest │             │
│  │   Sweep     │  │ (Palisade)  │  │  (Redteam)  │             │
│  └─────────────┘  └─────────────┘  └─────────────┘             │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Use Cases

ML Training

GPU-accelerated model training with workflow orchestration, metric streaming, and hyperparameter sweeps.

job = TrainingJob(
    forge=forge,
    script="train.py",
    preset="training:large",
    requirements=["torch", "transformers"],
)
result = await job.run()

Palisade Integration

Safe inference-time backdoor detection with network isolation and syscall monitoring.

with forge.sandbox(preset="security:inference") as sb:
    sb.upload_sync("./model", "/workspace/model")
    result = sb.run_sync("python scan.py --model /workspace/model")

Agent Testing

Isolated execution environments for testing AI agents with controlled network access.

with forge.sandbox(preset="agent:standard") as sb:
    sb.run_sync("python agent.py --task 'implement feature X'")

Backend Comparison

Development

Using uv (Recommended)

# Clone the repository
git clone https://github.com/highflame-ai/highflame-forge.git
cd highflame-forge

# Install from lockfile (reproducible)
uv sync --group dev

# Run tests
uv run pytest tests/

# Type checking
uv run mypy src/highflame_forge

# Linting
uv run ruff check src/

# Update dependencies and regenerate lockfile
uv lock --upgrade

Note: The uv.lock file is committed to ensure reproducible builds across environments.

Using pip

# Clone the repository
git clone https://github.com/highflame-ai/highflame-forge.git
cd highflame-forge

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install development dependencies
pip install -e ".[dev]"

# Run tests
pytest tests/

# Type checking
mypy src/highflame_forge

# Linting
ruff check src/

License

MIT License - see LICENSE for details.

Related Projects

• Palisade - ML model security scanning
• Redteam - Agent security testing
• Overwatch - Agent monitoring