TrainLens

See what changed between training runs without leaving your notebook.

TrainLens is a lightweight Python library for comparing training runs, spotting common training problems, planning the next experiment, exporting results, and optionally asking an OpenAI-compatible model to explain the evidence.

Try it in a notebook in 20 seconds

First cell:

%pip install -q trainlens

Second cell:

from trainlens import compare_runs

compare_runs(
    {"loss": 0.52, "accuracy": 0.84},
    {"loss": 0.41, "accuracy": 0.89},
    baseline_name="baseline",
    experiment_name="new run",
)

That is enough. In Jupyter, TrainLens renders the result as Markdown automatically:

Summary

• Material improvement detected in accuracy, loss.

Metric	Baseline	Experiment	Delta	Direction
accuracy	0.84	0.89	+0.05	improved
loss	0.52	0.41	-0.11	improved

No model provider, API key, callback, or experiment tracker is required for local comparison.

A few useful snippets

Keep the structured result when you want to inspect it in code:

comparison = compare_runs(old_metrics, new_metrics)
comparison.improvements
comparison.regressions

Render the same comparison anywhere Markdown is useful:

markdown = comparison.to_markdown()
print(markdown)

Export a comparison to another format:

from trainlens import render_report

html = render_report(comparison, format="html")
json_text = render_report(comparison, format="json")

TrainLens also understands richer TrainingRun objects, including metric trajectories and configuration changes, when you need more than two dictionaries.

Ask an LLM for an improvement plan

The LLM is optional. TrainLens first performs deterministic local analysis, then sends a bounded, redacted evidence summary to the model. You can use a remote provider or a local OpenAI-compatible server.

1. Load the model provider

import os
from trainlens import OpenAICompatibleProvider

llm = OpenAICompatibleProvider.from_values(
    base_url="https://your-provider.example/v1",
    model="your-model",
    api_key=os.environ["YOUR_LLM_API_KEY"],
)

For a local server such as Ollama, LM Studio, vLLM, or llama.cpp, the same API works and an API key is usually unnecessary:

llm = OpenAICompatibleProvider.from_values(
    base_url="http://localhost:11434/v1",
    model="your-local-model",
)

If you prefer environment variables, configure TRAINLENS_LLM_BASE_URL, TRAINLENS_LLM_MODEL, and optionally TRAINLENS_LLM_API_KEY, then use:

llm = OpenAICompatibleProvider.from_env()

2. Define what you want the model to analyze

from trainlens import improvement_plan_prompt

prompt = improvement_plan_prompt(
    objective=(
        "Find the three highest-impact improvements for the next run. "
        "Prefer low-cost, controlled changes and explain the evidence for each one."
    ),
    model_family="vision-language model",
    focus_areas=(
        "generalization gap",
        "learning-rate schedule",
        "adapter capacity",
    ),
)

The default improvement recipe asks for evidence, the proposed change, rationale, expected effect, cost/risk, confidence, a measurable success criterion, and one concrete next run. It also tells the model not to invent missing hyperparameters.

3. Inspect the exact prompt before sending anything

from trainlens import preview_llm_request

preview = preview_llm_request(
    globals(),
    mode="improvement_ideas",
    provider=llm,
    prompt_options=prompt,
)

print(preview.system_prompt)  # trusted analysis instructions
print(preview.user_prompt)    # sanitized notebook evidence

preview_llm_request() performs no network request. The preview is built by the same code path used for the real OpenAI-compatible request, so what you inspect is what TrainLens will send, excluding the API key.

4. Generate the improvement report

from trainlens import build_improvement_ideas

report = build_improvement_ideas(
    globals(),
    provider=llm,
    prompt_options=prompt,
)

report  # renders as Markdown in Jupyter

TrainLens detects the metrics, model/trainer evidence, local signals, and deterministic recommendations already present in the notebook. The LLM explains and prioritizes that evidence rather than replacing the local analysis with an ungrounded interpretation.

What else can TrainLens do?

• Compare runs with metric-aware improvement/regression semantics.
• Inspect notebook training context without wiring a full MLOps stack.
• Monitor training with built-in and extensible alert detectors.
• Plan the next experiment with constraints, objectives, and Pareto-aware helpers.
• Save and load portable runs for repeatable comparisons.
• Export reports to Markdown, JSON, HTML, and optional PDF.
• Use an optional LLM to explain deterministic TrainLens findings through any OpenAI-compatible endpoint, including local servers.

The local analysis features are deterministic and do not make an LLM request.

Want the deeper API?

The README intentionally stays small. The versioned docs contain the advanced workflows:

• Getting started
• Notebook workflows
• Python API
• Monitoring
• LLM setup and privacy
• Exports and troubleshooting
• Documentación en español

Privacy

LLM support is optional. When enabled, TrainLens minimizes and bounds notebook evidence, redacts secrets, and keeps notebook-derived evidence separate from trusted system instructions. Treat generated explanations as assistance and verify them against the underlying training evidence.

Scope

TrainLens is a lightweight notebook reporting and experiment-understanding layer, not a full MLOps platform. It is designed for research and small-to-medium workflows where the important training context already lives in Python and Jupyter.

Contributing and license

See CONTRIBUTING.md to contribute and SECURITY.md to report vulnerabilities.

TrainLens is licensed under the Apache License 2.0.