All environments support both spot (leverage=1) and futures (leverage>1) trading via config.

See Offline Environments Documentation for detailed guides.

TorchTrade includes implementations of multiple RL algorithms, all usable across any environment via Hydra config switching:

• PPO - examples/online_rl/ppo/
• PPO + Chronos (time series embeddings) - examples/online_rl/ppo_chronos/
• DQN - examples/online_rl/dqn/
• IQL - examples/online_rl/iql/
• DSAC - examples/online_rl/dsac/
• GRPO - examples/online_rl/grpo/
• CTRL - Research

Run Training Examples

# PPO with default environment (sequential SLTP)
uv run python examples/online_rl/ppo/train.py

# PPO with different environments (switch via command-line)
uv run python examples/online_rl/ppo/train.py env=sequential_futures
uv run python examples/online_rl/ppo/train.py env=onestep_futures
uv run python examples/online_rl/ppo/train.py env=sequential_spot

# GRPO with default (one-step futures)
uv run python examples/online_rl/grpo/train.py

# GRPO with spot trading
uv run python examples/online_rl/grpo/train.py env=onestep_spot

# Customize with Hydra overrides
uv run python examples/online_rl/ppo/train.py \
    env=sequential_futures \
    env.symbol="ETH/USD" \
    env.leverage=10 \
    optim.lr=1e-4 \
    loss.gamma=0.95

Available environment configs (env=<name>):

• sequential_spot - Basic spot trading
• sequential_futures - Basic futures trading
• sequential_sltp - Spot with bracket orders
• sequential_futures_sltp - Futures with bracket orders
• onestep_spot - Contextual bandit (spot)
• onestep_futures - Contextual bandit (futures)

See Examples Documentation for all available examples.

Prerequisites

• Python 3.11+
• CUDA (optional, for GPU acceleration)
• UV - Fast Python package installer

Full Installation

# 1. Install UV
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows: powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

# 2. Clone repository
git clone https://github.com/TorchTrade/torchtrade.git
cd torchtrade

# 3. Install dependencies
uv sync

# Optional: Install with extra features
# uv sync --extra examples         # Deps the scripts under examples/ need
# uv sync --extra llm              # LLM actors (OpenAI API + local vLLM/transformers)
# uv sync --extra chronos          # Chronos forecasting transforms
# uv sync --extra dev              # Development/testing tools
# uv sync --extra docs             # Documentation building
# uv sync --all-extras             # Install all optional dependencies

# 4. Activate virtual environment
source .venv/bin/activate  # Unix/macOS
# .venv\Scripts\activate  # Windows

# 5. For live trading, create .env file
cat > .env << EOF
ALPACA_API_KEY=your_alpaca_api_key
ALPACA_SECRET_KEY=your_alpaca_secret_key
BINANCE_API_KEY=your_binance_api_key
BINANCE_SECRET_KEY=your_binance_secret_key
BYBIT_API_KEY=your_bybit_api_key
BYBIT_API_SECRET=your_bybit_api_secret
OKX_API_KEY=your_okx_api_key
OKX_API_SECRET=your_okx_api_secret
OKX_PASSPHRASE=your_okx_passphrase
EOF

# 6. Verify installation
uv run pytest tests/ -v

Training PPO on Backtesting Data

from torchtrade.envs.offline import SequentialTradingEnv, SequentialTradingEnvConfig
import datasets
import pandas as pd

# Load historical data from HuggingFace
df = datasets.load_dataset("Torch-Trade/btcusdt_spot_1m_03_2023_to_12_2025")
df = df["train"].to_pandas()
df['timestamp'] = pd.to_datetime(df['timestamp'])

# Configure multi-timeframe environment
config = SequentialTradingEnvConfig(
    leverage=1,             # spot
    action_levels=[0, 1],   # long-only; the default [-1, 0, 1] warns under leverage=1
    time_frames=["1min", "5min", "15min", "60min"],
    window_sizes=[12, 8, 8, 24],
    execute_on="5Min",
    initial_cash=[1000, 5000],  # Domain randomization
    transaction_fee=0.0025,
    slippage=0.001
)

env = SequentialTradingEnv(df, config)
# Train with PPO - see examples/online_rl/ppo/train.py

Live Trading with Alpaca

import os

from dotenv import load_dotenv
from torchtrade.envs.live.alpaca import AlpacaTorchTradingEnv, AlpacaTradingEnvConfig
from alpaca.data.timeframe import TimeFrame, TimeFrameUnit

load_dotenv()  # reads the .env written above; os.getenv does not read files

config = AlpacaTradingEnvConfig(
    symbol="BTC/USD",
    time_frames=[
        TimeFrame(1, TimeFrameUnit.Minute),
        TimeFrame(5, TimeFrameUnit.Minute),
    ],
    window_sizes=[12, 8],
    execute_on=TimeFrame(5, TimeFrameUnit.Minute),
    paper=True  # Start with paper trading!
)

env = AlpacaTorchTradingEnv(
    config,
    api_key=os.getenv("ALPACA_API_KEY"),
    api_secret=os.getenv("ALPACA_SECRET_KEY"),
)
# See examples/online_rl/ppo/live.py

LLM-Based Trading

from torchtrade.actor import FrontierLLMActor

# Use GPT as trading policy
policy = FrontierLLMActor(
    model="gpt-4o-mini",
    market_data_keys=env.market_data_keys,
    account_state_labels=env.account_state,  # list of label strings, e.g. ["exposure_pct", ...]
    action_levels=env.action_levels,
    debug=True,
)

tensordict = env.reset()
action = policy(tensordict)
# See examples/llm/frontier/offline.py

Rule-Based Trading Strategies

from torchtrade.actor import MeanReversionActor

# Use as baseline or for imitation learning
actor = MeanReversionActor(
    market_data_keys=["market_data_5Minute_24"],
)

Feature Engineering

import ta

def custom_preprocessing(df):
    """Add technical indicators as features"""
    df["features_open"] = df["open"]
    df["features_close"] = df["close"]
    df["features_rsi_14"] = ta.momentum.RSIIndicator(
        df["close"], window=14
    ).rsi()
    df.fillna(0, inplace=True)
    # timestamp must come back as a COLUMN, not an index, or the sampler raises
    # KeyError: ['timestamp'] not in index
    return df.reset_index()

config = SequentialTradingEnvConfig(
    leverage=1,
    action_levels=[0, 1],   # long-only; the default [-1, 0, 1] warns under leverage=1
    time_frames=["1min", "5min"],
    window_sizes=[12, 8],
)

# feature_preprocessing_fn is a CONSTRUCTOR argument, not a config field.
env = SequentialTradingEnv(df, config, custom_preprocessing)

See Advanced Customization for more examples.

Multi-Timeframe Observations

config = SequentialTradingEnvConfig(
    leverage=1,
    action_levels=[0, 1],   # long-only; the default [-1, 0, 1] warns under leverage=1
    time_frames=["1min", "5min", "15min", "60min"],
    window_sizes=[12, 8, 8, 24],
    execute_on="5Min"
)

# Results in observations:
# - market_data_1Minute_12: [12, num_features] - Last 12 one-minute bars
# - market_data_5Minute_8: [8, num_features] - Last 40 minutes
# - market_data_15Minute_8: [8, num_features] - Last 120 minutes
# - market_data_60Minute_24: [24, num_features] - Last 24 hours

Observation Structure

observation = {
    "market_data_1Minute_12": tensor([12, num_features]),
    "market_data_5Minute_8": tensor([8, num_features]),
    "account_state": tensor([6]),  # Universal 6-element state
}

# Account state (universal): [exposure_pct, position_direction, unrealized_pnl_pct,
#                             holding_time, leverage, distance_to_liquidation]
# Element definitions:
#   - exposure_pct: position_value / portfolio_value (0-1+ with leverage)
#   - position_direction: sign(position_size) (-1=short, 0=flat, +1=long)
#   - unrealized_pnl_pct: (current_price - entry_price) / entry_price * direction
#   - holding_time: bars the current position has been held (1 on the opening bar, 0 when flat)
#   - leverage: 1.0 for spot, 1-125 for futures
#   - distance_to_liquidation: normalized distance (1.0 for spot/no position)
#
# Spot mode: position_direction in {0, +1}, leverage=1.0, distance_to_liquidation=1.0
# Futures mode: position_direction in {-1, 0, +1}, leverage=1-125, calculated distance

Action Spaces

Standard (3 actions):

• Action 0: SELL/SHORT
• Action 1: HOLD
• Action 2: BUY/LONG

SLTP Combinatorial:

• Action 0: HOLD
• Actions 1..N: BUY/LONG with (SL, TP) combinations
• Actions N+1..2N: SHORT with (SL, TP) combinations (futures only)

See Advanced Customization for detailed explanations.

TorchTrade uses Hydra for configuration management with a defaults list pattern:

# examples/online_rl/ppo/config.yaml
defaults:
  - env: sequential_sltp  # Load environment config
  - _self_

collector:
  frames_per_batch: 100000
  total_frames: 100_000_000

optim:
  lr: 2.5e-4
  anneal_lr: true
  max_grad_norm: 0.5

loss:
  gamma: 0.9
  clip_epsilon: 0.1
  entropy_coef: 0.01

# examples/online_rl/env/sequential_sltp.yaml
env:
  name: SequentialTradingEnvSLTP
  leverage: 1
  symbol: "BTC/USD"
  time_frames: ["5Min", "15Min"]
  window_sizes: [10, 10]
  execute_on: "15Min"
  initial_cash: [1000, 5000]
  transaction_fee: 0.0025
  # ... more env config

Override from command line:

# Switch environment entirely
uv run python examples/online_rl/ppo/train.py env=sequential_futures

# Override specific parameters
uv run python examples/online_rl/ppo/train.py \
    env.symbol="ETH/USD" \
    env.leverage=10 \
    optim.lr=1e-4 \
    loss.gamma=0.95