🛠️ LangSmith CLI

[!TIP] Install as Claude Code Plugin

Step 1: Install the CLI tool

# Quick install (Linux/macOS)
curl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | sh

# Or use uv
uv tool install langsmith-cli

Step 2: Add the plugin to Claude Code

Option A: Terminal Command

claude plugin marketplace add gigaverse-app/langsmith-cli
claude plugin install langsmith-cli@langsmith-cli

Option B: Interactive UI Inside Claude Code, run /plugin (without arguments) and use the UI to add the marketplace

This gives Claude Code instant access to all LangSmith commands via skills.

🎯 Why LangSmith CLI?

Traditional tools are slow, verbose, and waste tokens. LangSmith CLI is different:

100% Feature Parity + 10x Better QoL 🚀

✨ Features

🏎️ Performance First

• <100ms startup via lazy-loaded imports
• Streams large datasets without memory bloat
• Async-ready architecture

🧠 Agent Optimized

# Traditional: Returns 20KB trace object (1000+ tokens)
langsmith-cli runs get abc123

# Agent Mode: Returns only what you need (40 tokens)
langsmith-cli --json runs get abc123 --fields inputs,outputs,error

96% context savings on large traces!

🎨 Human Friendly

• Beautiful Rich tables with syntax highlighting
• Color-coded statuses (🟢 success, 🔴 error, 🟡 pending)
• Smart column truncation for readability
• Export to CSV/YAML for spreadsheets

🔍 Power User Features

# Regex filtering
langsmith-cli runs list --name-regex "^prod-.*-v[0-9]+"

# Wildcard patterns
langsmith-cli runs list --name-pattern "*auth*"

# Smart filters
langsmith-cli runs list --slow --failed --today

# Live dashboard
langsmith-cli runs watch

🗄️ S3 Trace Archive

Retain traces in private S3, backfill long historical windows through LangSmith Bulk Export, and query the canonical Parquet directly with DuckDB:

# One-time historical export; dates are a half-open UTC range.
langsmith-cli --json archive backfill --config archive.yaml --route production \
  --start-date 2025-08-01 --end-date 2026-08-01 \
  --bulk-export-destination-id <uuid> --import-workers 8

# Scan the retained archive without paging through the live Runs API.
langsmith-cli --json runs search "timeout" --archive \
  --project prd/my-agent --last 365d --fields id,name,status,error

The backfill is resumable: it adopts exact matching remote jobs and skips sealed project-days. See the archive operator reference for setup, safe scaling, progress checks, and recovery, and the archive design for storage and invariants.

📦 Complete Coverage

Every LangSmith resource at your fingertips:

• ✅ Projects - List, create, inspect
• ✅ Runs - Search, stats, watch, sample, analyze, field discovery
• ✅ Datasets - CRUD + bulk JSONL uploads
• ✅ Examples - Full lifecycle management
• ✅ Prompts - Version control your prompts
• ✅ Self - Installation detection + auto-update

🚀 Installation

Quick Install (Recommended)

Linux/macOS:

curl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | sh

Windows:

iwr -useb https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.ps1 | iex

This standalone installer:

• Creates an isolated environment (no conflicts)
• Automatically adds langsmith-cli to your PATH
• Works without manually installing Python packages
• Requires Python 3.12+

Using uv

uv tool install langsmith-cli

Using pip

pip install langsmith-cli

For Claude Code Users

After installing the CLI above, add the skill:

/plugin marketplace add gigaverse-app/langsmith-cli

From Source

git clone https://github.com/gigaverse-app/langsmith-cli.git
cd langsmith-cli
uv sync
uv run langsmith-cli --help

🔑 Quick Start

1️⃣ Authenticate

langsmith-cli auth login
# Creates .env with your LANGSMITH_API_KEY

2️⃣ Explore Your Projects

langsmith-cli projects list

┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Project       ┃ Run Count  ┃ Last Run     ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ production    │ 12,450     │ 2 mins ago   │
│ staging       │ 3,241      │ 15 mins ago  │
│ development   │ 892        │ 1 hour ago   │
└───────────────┴────────────┴──────────────┘

3️⃣ Inspect Recent Runs

langsmith-cli runs list --project production --limit 5

4️⃣ Debug Errors Fast

langsmith-cli runs list --failed --recent --fields error

💡 Examples

🔍 Advanced Filtering

Find authentication runs that failed in the last hour:

langsmith-cli runs list \
  --name-pattern "*auth*" \
  --failed \
  --recent

Search for specific versioned services:

langsmith-cli runs list \
  --name-regex "^prod-api-v[0-9]+" \
  --min-latency 5s \
  --today

Multi-tag filtering (AND logic):

langsmith-cli runs list \
  --tag production \
  --tag experimental \
  --slow

📊 Aggregated Insights

langsmith-cli runs stats --project production

{
  "run_count": 12450,
  "error_rate": 0.023,
  "latency_p50": 0.234,
  "latency_p99": 1.892,
  "total_cost": 45.67,
  "last_run_time": "2026-01-14T20:15:30Z"
}

🔴 Live Monitoring

langsmith-cli runs watch --project production

🔴 Live Dashboard (Ctrl+C to exit)

┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓
┃ Run              ┃ Status ┃ Latency  ┃ Time     ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩
│ ChatOpenAI       │ 🟢     │ 0.234s   │ Just now │
│ RetrievalChain   │ 🟢     │ 1.456s   │ 2s ago   │
│ AuthService      │ 🔴     │ 5.123s   │ 5s ago   │
│ WebScraper       │ 🟡     │ -        │ 8s ago   │
└──────────────────┴────────┴──────────┴──────────┘

💾 Bulk Dataset Uploads

# Export examples to JSONL (using --output for reliable file writing)
langsmith-cli examples list --dataset my-dataset --output examples.jsonl

# Upload to new dataset
langsmith-cli datasets push examples.jsonl --dataset production-eval

🌐 Open in Browser

langsmith-cli runs open <run-id>
# Opens trace in LangSmith UI

📤 Export for Analysis

# Export to CSV for Excel
langsmith-cli runs list --format csv > runs.csv

# Export to YAML for configs
langsmith-cli projects list --format yaml > projects.yml

🔊 Verbosity Control

Control diagnostic output with industry-standard flags (following pip, Black, etc.):

# Default: Progress messages + warnings
langsmith-cli runs list --project production

# Quiet: Warnings only, no progress
langsmith-cli -q runs list --project production

# Silent: Errors only (cleanest for scripts)
langsmith-cli -qq runs list --project production

# Debug: Show API calls and processing details
langsmith-cli -v runs list --project production

# Trace: Ultra-verbose with HTTP requests and timing
langsmith-cli -vv runs list --project production

Clean JSON piping with -qq:

# Diagnostics on stderr, JSON on stdout
langsmith-cli --json runs list | jq

# Silent mode for cleanest piping
langsmith-cli --json -qq runs list | jq '.[] | .name'

# Suppress diagnostics with stderr redirection
langsmith-cli --json runs list 2>/dev/null | jq

🤖 AI Agent Integration

As a Claude Code Skill

[!NOTE] Prerequisites: Install CLI first (uv tool install langsmith-cli), then add skill (/plugin marketplace add gigaverse-app/langsmith-cli)

The CLI is optimized for Claude Code agents:

# In Claude Code, agents can run:
langsmith-cli --json runs list --project default --limit 10 --fields inputs,outputs,error

# Returns clean JSON without Rich formatting
# Uses --fields to minimize context usage
# Filters with --failed, --slow, --recent for targeted debugging

Key Agent Patterns:

• Always use --json as first argument
• Use --fields to reduce tokens by 90%+
• Combine smart filters: --failed --recent --slow
• Keep --limit small (default: 10)

Example Agent Usage

import subprocess
import json

# Agent efficiently fetches only errored runs
result = subprocess.run(
    ["langsmith-cli", "--json", "runs", "list",
     "--failed", "--limit", "5", "--fields", "error,inputs"],
    capture_output=True,
    text=True
)

runs = json.loads(result.stdout)
# Analyze errors with minimal context usage

📚 Documentation

Command Reference

langsmith-cli --help

# Core Commands
auth login              # Authenticate with LangSmith
projects list           # List all projects
runs list              # Search and filter runs
runs get <id>          # Inspect a specific run
runs get-latest        # Get most recent run matching filters
runs stats             # Aggregate statistics
runs watch             # Live run dashboard
runs open <id>         # Open trace in browser
runs search            # Full-text search across runs
runs sample            # Stratified sampling by tags/metadata
runs analyze           # Group runs and compute metrics
runs tags              # Discover tag patterns
runs metadata-keys     # Discover metadata keys
runs fields            # Discover field paths and types
runs describe          # Detailed field statistics
runs view-file         # View runs from JSONL files
runs usage             # Token usage analysis with grouping
runs pricing           # Model pricing coverage check
runs cache download    # Download runs to local JSONL cache
runs cache list        # List cached projects
runs cache clear       # Clear cached data
datasets list          # List datasets
datasets create        # Create new dataset
datasets push          # Bulk upload from JSONL
examples list          # List dataset examples
examples create        # Add example to dataset
prompts list           # List prompt repositories
prompts get            # Pull a prompt template
prompts push           # Push local prompt to LangSmith
self detect            # Show installation details
self update            # Update to latest version

Global Flags

--json              # Machine-readable JSON output (no Rich formatting)
--format {json|csv|yaml}  # Export format
--help              # Show help
--version           # Show version

Runs Filtering

🏗️ Architecture

Design Principles

1. Lazy Loading Performance

# ❌ Don't import heavy libraries at top level
from langsmith import Client

# ✅ Import inside command functions
@runs.command("list")
def list_runs(...):
    import langsmith  # Only loads when command executes
    client = langsmith.Client()

2. Context Efficiency

• Default JSON output is "sparse" (only essential fields)
• Field pruning with --fields for targeted data extraction
• No full trace blobs unless explicitly requested

3. Dual UX Pattern

# Check JSON flag for output mode
if ctx.obj.get("json"):
    click.echo(json.dumps(data))  # Strict JSON for agents
else:
    console.print(rich_table)      # Beautiful tables for humans

4. DRY Utilities

• Shared helpers in utils.py (100% test coverage)
• Consistent error handling with field context
• Reusable filtering, sorting, and formatting functions

🧪 Development

Setup

# Clone and setup
git clone https://github.com/gigaverse-app/langsmith-cli.git
cd langsmith-cli
uv sync

# Install pre-commit hooks
uv run pre-commit install

Testing

# Run all tests
uv run pytest

# With coverage
uv run pytest --cov=src --cov-report=term-missing

# E2E tests (requires LANGSMITH_API_KEY)
export LANGSMITH_API_KEY="lsv2_..."
uv run pytest tests/test_e2e.py -v

Code Quality

# Linting and formatting
uv run ruff check --fix
uv run ruff format

# Type checking
uv run pyright

Project Stats

• 92% Test Coverage (589 tests)
• Zero Type Errors (Pyright clean)
• 100% MCP Parity (13/13 tools)

🤝 Contributing

Contributions welcome! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/awesome)
3. Make your changes with tests
4. Ensure 100% coverage for new code
5. Run uv run pre-commit run --all-files
6. Submit a pull request

📝 License

MIT License - see LICENSE for details.

🙏 Acknowledgments

Built with:

• Click - CLI framework
• Rich - Terminal formatting
• LangSmith SDK - Official Python client
• uv - Fast Python package installer