pdtrain 0.2.0

Pipedream Training Orchestrator CLI - Train ML models on AWS SageMaker

pdtrain

Pipedream Training Orchestrator CLI - Train ML models on AWS SageMaker with ease.

Installation

pip install pdtrain

Quick Start

1. Configure

pdtrain configure

This will prompt you for:

• API URL (default: http://localhost:8000)
• API Key (from Pipedream dashboard)

2. Upload Training Code

# Upload a directory (will auto-create tar.gz)
pdtrain bundle upload ./my-training-code --wait

# Or upload existing tar.gz file
pdtrain bundle upload ./my-training-code.tar.gz --wait

3. Upload Dataset

pdtrain dataset upload ./data.csv --name "train-data" --wait

4. Create and Run Training

# Get bundle and dataset IDs first
pdtrain bundle list
pdtrain dataset list

# Create run using IDs
pdtrain run create \
  --bundle ca8912d6-79a4-4ea9-8570-234ec1baeef1 \
  --dataset ds_abc123-def456-7890 \
  --framework pytorch \
  --entry train.py \
  --submit --wait

5. View Results

# View logs
pdtrain logs run-abc123

# List artifacts
pdtrain artifacts list run-abc123

# Download artifacts
pdtrain artifacts download run-abc123 --output ./results/

Commands

Bundle Management

# Upload directory (auto-creates tar.gz)
pdtrain bundle upload ./my-training-code --wait

# Upload existing tar.gz
pdtrain bundle upload ./code.tar.gz --name "my-model" --wait

# List bundles
pdtrain bundle list

# Show bundle details
pdtrain bundle show abc-123

Dataset Management

# Upload dataset
pdtrain dataset upload ./data.csv --name "train-data" --wait

# List datasets
pdtrain dataset list

# Download dataset
pdtrain dataset download ds-456 --version 1 --output ./data/

Training Runs

# Create run
pdtrain run create \
  --bundle my-model:v1.0.0 \
  --dataset train-data:1 \
  --framework pytorch \
  --entry train.py

# Submit run
pdtrain run submit run-abc123

# List runs
pdtrain run list

# Show run details
pdtrain run show run-abc123

# Watch run progress
pdtrain run watch run-abc123

# Stop run
pdtrain run stop run-abc123

Logs & Artifacts

# View logs (last 300 lines by default)
pdtrain logs run-abc123 --lines 500

# Follow logs in real-time
pdtrain logs run-abc123 --follow --interval 5

# List artifacts
pdtrain artifacts list run-abc123

# Download artifacts (defaults to ./artifacts/<run_id>/)
pdtrain artifacts download run-abc123 --output ./results/

Quota

# Check storage quota
pdtrain quota

Configuration

Configuration is stored in ~/.pdtrain/config.json.

You can also use environment variables:

export PDTRAIN_API_URL=https://ml-orchestrator.pipedream.in
export PDTRAIN_API_KEY=sdk_xxxxx

Examples

Complete Workflow

# 1. Upload code (from directory)
pdtrain bundle upload ./resnet-training --wait

# 2. Upload dataset
pdtrain dataset upload ./cifar10.csv --name "cifar10" --wait

# 3. Get IDs
pdtrain bundle list    # Copy bundle ID
pdtrain dataset list   # Copy dataset ID

# 4. Run training (use IDs from step 3)
pdtrain run create \
  --bundle <bundle-id> \
  --dataset <dataset-id> \
  --framework pytorch \
  --entry train.py \
  --submit --wait

# 5. Download results
pdtrain artifacts download <run-id> --output ./results/

Docker Mode

pdtrain run create \
  --bundle my-code:latest \
  --image 763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:2.2.0-cpu \
  --entry train.py \
  --submit

Script Mode with Hyperparameters

pdtrain run create \
  --bundle my-code:latest \
  --framework pytorch \
  --framework-version 2.2.0 \
  --hyperparameter epochs=10 \
  --hyperparameter batch_size=32 \
  --submit

Development

# Clone repository
git clone https://github.com/pipedream/pdtrain
cd pdtrain

# Install in development mode
pip install -e ".[dev]"

# Run tests
pytest

# Format code
black pdtrain/

License

MIT License - see LICENSE file for details.

Support

• API KEYS:https://pipedream.in/api-keys
• Issues: https://github.com/pipedream/pdtrain/issues
• Email: [e-mail]