PulseML

Pulse — a live ML training debugger for CLI and headless environments, built to work across major ML backends.

🔗 pulsedashb.netlify.app · GitHub · PyPI

Pulse is a live machine learning training debugger designed to monitor tensors, track metrics, detect numerical failures, verify mathematical relationships, and work with an integrated AI debugging agent.

The goal is simple:

TRACK → DIAGNOSE → VERIFY → PATCH

Key Features

• CLI / Headless First — Built for terminals, SSH sessions, Google Colab, containers, remote servers, and long-running training jobs. Pulse does not require a graphical interface.

• Live Training Monitoring — Track losses, metrics, tensors, gradients, activations, weights, and other numerical values while training is running. Inspect shapes, dtypes, devices, norms, statistics, NaN/Inf counts, and other diagnostics.

• CPU-First Tracking — Pulse is designed to stay off the GPU by default. Monitoring and diagnostics happen on the CPU whenever possible, allowing GPU training to continue independently.

• Opt-In GPU Tracking — GPU-resident variables are not automatically synchronized just because they exist. GPU tracking only happens when explicitly requested, because device-to-host transfers necessarily introduce overhead.

• Lightweight Tracking — Selective tracking, separate probe cadences, and cached statistics reduce unnecessary work inside the training loop. The objective is to make the debugger useful without turning it into the bottleneck.

• Smart Scalars — Loss-like variables such as loss, cost, nll, and cross_entropy are automatically recognized and prioritized. Scalar histories can be inspected directly from the CLI.

• Dynamic Variable Tracking — Add, remove, promote, or demote variables while training is running instead of restarting the entire job just to inspect another tensor.

• Deterministic Math Verification — Pulse can perform restricted numerical calculations to verify ratios, scaling factors, normalization values, update magnitudes, gradient relationships, and other mathematical claims instead of asking an LLM to perform exact arithmetic.

• Pulse AI Agent — The agent can reason over live training state, tensor statistics, scalar histories, gradients, activations, tracebacks, and relevant code to investigate failures and develop concrete fixes.

• Agentic Debugging — Pulse can move from observation to diagnosis to a proposed code change. When explicitly requested, the agent can generate or apply a structured patch for inspection.

• Automatic Intervention — With /autofix on, Pulse can pause training when configured detection logic identifies serious numerical or training problems, giving the debugging agent an opportunity to investigate before more compute is wasted.

• Universal Backend Support — Automatically detects and works with NumPy, PyTorch, TensorFlow, CuPy, and JAX through a shared backend abstraction layer.

• Cloud Workspace — Optional workspace synchronization can provide shared access to debugging sessions, incidents, tracebacks, agent conversations, and repository metadata.

Past Debugs

• Debugged a custom LLM after a 2.5x vocabulary increase by identifying a normalization bug where residual growth was divided by math.sqrt(num_layers) instead of num_layers. The resulting activation growth destabilized training and halted learning.

• Debugged a custom attention implementation producing NaN loss by tracing the failure to a missing infinity check before a division operation.

These are examples of the kind of numerical debugging Pulse is designed to support: observe the behavior, inspect the evidence, verify the math, and identify the actual failure rather than guessing from the final loss.

Install

pip install pulseml

Pulse is designed to run in standard Python terminal environments and does not require a GUI.

Quickstart

Import auto_track and call it immediately before your training loop.

Make sure your loop is wrapped in if __name__ == "__main__":, especially when using multiprocessing or process-spawning environments.

from pulse import auto_track

if __name__ == "__main__":
    auto_track()

    # Your training loop
    for epoch in range(num_epochs):
        # Training logic here
        pass

Pulse discovers numeric variables available to the training process and provides them through the CLI for monitoring.

You can then control tracking while the run is active:

/vars
/tracked
/add <variable>
/track <variable>
/lotrack <variable>
/gputrack <variable>
/gpuuntrack <variable>
/delete <variable>

CLI

Pulse is designed around a headless workflow.

Useful commands include:

/help
/vars
/tracked
/add <variable>
/track <variable>
/lotrack <variable>
/gputrack <variable>
/gpuuntrack <variable>
/delete <variable>
/autofix on|off
/code
/cloud

The CLI can pause training when intervention is required, allowing you to inspect the current state, add variables, ask the AI agent questions, or investigate a failure before continuing.

This makes Pulse suitable for:

• Local development
• SSH
• Google Colab
• Remote GPU machines
• Containers
• Cloud training
• Long-running experiments

CPU-First GPU Policy

Pulse is intentionally conservative around GPU access.

By default:

TRACKING_MODE = CPU_DEFAULT
GPU_TRACKING = OPT-IN
DEFAULT_GPU_OVERHEAD = 0

If a variable already exists on a GPU, Pulse does not automatically read it back to the CPU on every iteration.

If you explicitly request GPU tracking:

/gputrack <variable>

Pulse may perform device-to-host transfers to inspect the variable.

Those transfers can introduce overhead. That is expected and unavoidable when collecting GPU-resident data.

The design principle is:

If nobody asks Pulse to touch the GPU, Pulse doesn't touch the GPU.

AI Chat & API Keys

Pulse can use supported cloud or local AI providers for its debugging agent.

Common provider environment variables include:

ANTHROPIC_API_KEY
OPENAI_API_KEY
GEMINI_API_KEY
DEEPSEEK_API_KEY
MISTRAL_API_KEY
OPENROUTER_API_KEY

Provider configuration can be supplied through environment variables or Pulse configuration.

Local and self-hosted models can also be used where supported.

Do not place API keys directly in source code or commit them to a repository.

Deterministic Math

Pulse separates AI reasoning from exact numerical calculation.

For example, instead of allowing an AI agent to estimate whether an update is unusually large:

AI:
"That gradient seems pretty large."

Pulse can provide actual measurements and calculate the relevant quantities:

AI hypothesis
      ↓
Numerical calculation
      ↓
Exact result
      ↓
Evidence-backed diagnosis

This can be used for:

• Gradient/update ratios
• Scaling factors
• Normalization calculations
• Parameter changes
• Numerical thresholds
• Other restricted mathematical expressions

The AI remains responsible for reasoning about what the numbers mean; Pulse provides a deterministic path for checking the arithmetic.

Performance

Pulse is designed around the idea that a debugger should not become the training bottleneck.

The monitoring system uses:

• CPU-first inspection
• Opt-in GPU probing
• Selective variable tracking
• Lightweight tracking modes
• Separate probe cadences
• Cached statistics
• Host-side numerical processing where possible
• Minimal intervention in the training loop

The objective is:

MORE VISIBILITY
+
LESS OVERHEAD

Rather than collecting everything continuously, Pulse lets you decide what information is actually worth monitoring.

Cloud Workspace & Privacy

Pulse can optionally synchronize debugging information with a shared workspace.

Depending on configuration, this may include:

• Debug sessions
• Training incidents
• Tracebacks
• Agent conversations
• Repository metadata
• Team membership
• Telemetry

Cloud synchronization is best-effort and is not intended to block the training loop.

For sensitive projects, review your cloud and telemetry configuration carefully.

Telemetry can be disabled with:

PULSE_TELEMETRY=off

Using a local AI model keeps model requests local, but it does not automatically disable separately enabled Pulse workspace synchronization.

Why Pulse?

Traditional ML debugging often looks like:

Train
 ↓
Wait
 ↓
Loss becomes NaN
 ↓
Read thousands of log lines
 ↓
Add print statements
 ↓
Train again
 ↓
Still don't know

Pulse is built around:

Train
 ↓
Track
 ↓
Detect
 ↓
Measure
 ↓
Verify
 ↓
Diagnose
 ↓
Develop
 ↓
Patch

The important difference is that the debugging process starts before the final failure.

Pulse gives the agent and the engineer access to the numerical evidence surrounding the failure instead of forcing them to reconstruct what happened afterward.

Supported Backends

Project Direction

Pulse is being built toward a debugging workflow where live observation, deterministic computation, and AI reasoning work together:

TRAINING
   ↓
OBSERVATION
   ↓
NUMERICAL EVIDENCE
   ↓
AI REASONING
   ↓
VERIFICATION
   ↓
SOLUTION
   ↓
PATCH

The goal isn't to simply tell you:

"Your training is broken."

It's to help answer:

What broke?
Why did it break?
When did it start?
Can the numbers prove it?
What should change?
Can the fix be implemented?

Links

Dashboard

GitHub

PyPI

License

Proprietary. See LICENSE.

Use of this software is governed by the terms in that file — copying, redistribution, and reverse engineering are not permitted.