expt-logger 0.1.0.dev5

Simple experiment logging library

expt_logger

Simple experiment tracking for RL training with a W&B-style API.

Quick Start

Install:

uv add expt-logger
# or
pip install expt-logger

Set your API key:

export EXPT_LOGGER_API_KEY=your_api_key

Start logging:

import expt_logger

# Initialize run with config
run = expt_logger.init(
    name="grpo-math",
    config={"lr": 3e-6, "batch_size": 8}
)

# Get experiment URLs
print(f"View experiment: {run.experiment_url}")
print(f"Base URL: {run.base_url}")

# Log RL rollouts with rewards
expt_logger.log_rollout(
    prompt="What is 2+2?",
    messages=[{"role": "assistant", "content": "The answer is 4."}],
    rewards={"correctness": 1.0, "format": 0.9},
    mode="train",
    commit=True  # Commit immediately and increment step
)

# Log scalar metrics
expt_logger.log({
    "train/loss": 0.45,
    "train/kl": 0.02,
    "train/reward": 0.85
}, commit=True)  # Commit and increment step

expt_logger.end()

Core Features

Scalar Metrics

Log training metrics with automatic step tracking:

# Batch multiple metrics at the same step (default behavior)
expt_logger.log({"loss": 0.5})
expt_logger.log({"accuracy": 0.9})
expt_logger.commit()  # Commits both at step 1, then increments to step 2

# Or commit immediately with commit=True
expt_logger.log({"loss": 0.4}, commit=True)  # Commits at step 2, increments to 3

# Use slash prefixes for train/eval modes
expt_logger.log({
    "train/loss": 0.5,
    "eval/loss": 0.6
}, step=10, commit=True)

# Or set mode explicitly
expt_logger.log({"loss": 0.5}, mode="eval", commit=True)

Note: Metrics default to "train" mode when no mode is specified and keys don't have slash prefixes.

Note: commit=False is now the default. Call expt_logger.commit() to batch multiple logs at the same step, or use commit=True for immediate commits.

Rollouts (RL-specific)

Log conversation rollouts with multiple reward functions:

# Batch multiple rollouts at the same step
expt_logger.log_rollout(
    prompt="Solve: x^2 - 5x + 6 = 0",
    messages=[
        {"role": "assistant", "content": "Let me factor this..."},
        {"role": "user", "content": "Can you verify?"},
        {"role": "assistant", "content": "Sure! (x-2)(x-3) = 0..."}
    ],
    rewards={
        "correctness": 1.0,
        "format": 0.9,
        "helpfulness": 0.85
    },
    mode="train"  # commit=False by default
)

expt_logger.log_rollout(
    prompt="Another problem...",
    messages=[{"role": "assistant", "content": "Solution..."}],
    rewards={"correctness": 0.8},
    mode="train"
)

expt_logger.commit()  # Commits both rollouts at the same step

# Or commit immediately with commit=True
expt_logger.log_rollout(
    prompt="Yet another...",
    messages=[{"role": "assistant", "content": "Answer..."}],
    rewards={"correctness": 1.0},
    step=5,
    mode="train",
    commit=True
)

• Messages format: List of dicts with "role" and "content" keys
• Rewards format: Dict of reward names to float values
• Mode: "train" or "eval" (default: "train")
• Commit: False (default) to batch, True to commit immediately

Configuration

Track hyperparameters and update them dynamically:

run = expt_logger.init(config={"lr": 0.001, "batch_size": 32})

# Update config during training
run.config.lr = 0.0005              # attribute style
run.config["epochs"] = 100          # dict style
run.config.update({"model": "gpt2"}) # bulk update

API Key & Server Configuration

API Key (required):

export EXPT_LOGGER_API_KEY=your_api_key

Or pass directly:

expt_logger.init(api_key="your_key")

Custom server URL (optional, for self-hosting):

export EXPT_LOGGER_BASE_URL=https://your-server.com

Or:

expt_logger.init(base_url="https://your-server.com")

Accessing Experiment URLs

Get the experiment URL and base URL from the run object:

run = expt_logger.init(name="my-experiment")

# Get the full experiment URL to view in browser
print(run.experiment_url)
# https://expt-platform.vercel.app/experiments/ccf1f879-50a6-492b-9072-fed6effac731

# Get the base URL of the tracking server
print(run.base_url)
# https://expt-platform.vercel.app

API Reference

expt_logger.init()

init(
    name: str | None = None,
    config: dict[str, Any] | None = None,
    api_key: str | None = None,
    base_url: str | None = None,
    auto_flush_interval: float = 10.0,
    auto_flush_buffer_size: int = 100
) -> Run

• name: Experiment name (auto-generated if not provided)
• config: Initial hyperparameters
• api_key: API key (or set EXPT_LOGGER_API_KEY)
• base_url: Custom server URL (or set EXPT_LOGGER_BASE_URL)
• auto_flush_interval: Seconds to wait before auto-flushing (default: 10.0)
• auto_flush_buffer_size: Number of items that trigger immediate flush (default: 100)

expt_logger.log()

log(
    metrics: dict[str, float],
    step: int | None = None,
    mode: str | None = None,
    commit: bool = False
)

• metrics: Dict of metric names to values
• step: Step number (auto-increments if not provided)
• mode: Default mode for keys without slashes (default: "train")
• commit: If False (default), buffer metrics until commit() is called. If True, commit immediately and increment step.

expt_logger.log_rollout()

log_rollout(
    prompt: str,
    messages: list[dict[str, str]],
    rewards: dict[str, float],
    step: int | None = None,
    mode: str = "train",
    commit: bool = False
)

• prompt: The prompt text
• messages: List of {"role": ..., "content": ...} dicts
• rewards: Dict of reward names to values
• step: Step number (uses current step if not provided)
• mode: "train" or "eval"
• commit: If False (default), buffer rollout until commit() is called. If True, commit immediately and increment step.

expt_logger.commit()

commit()

Commit all pending metrics and rollouts to the buffer and increment the step counter. This allows you to batch multiple log() and log_rollout() calls at the same step.

expt_logger.flush() / expt_logger.end()

• flush(): Manually send buffered data to server
• end(): Finish the run (called automatically on exit)

Note: Data is automatically flushed to the server using a debouncing mechanism:

• Every auto_flush_interval seconds after new data is logged (default: 10 seconds)
• Immediately when the buffer reaches auto_flush_buffer_size items (default: 100 items)
• On program exit or when end() is called

Advanced

Context Manager

Ensures automatic cleanup:

with expt_logger.init(name="my-run") as run:
    expt_logger.log({"loss": 0.5})
    expt_logger.commit()
# end() called automatically

Graceful Shutdown

The library handles cleanup on:

• Normal exit (atexit)
• Ctrl+C (SIGINT)
• SIGTERM

All buffered data is flushed before exit.

Development

For local development, see DEVELOPMENT.md.

Run the demo:

python demo.py          # GRPO-style training simulation
python demo.py commit   # Batching demo
python demo.py messages # Structured messages demo