Langfuse Lens Agent

Korean version: README_ko.md

Langfuse Lens Agent is an open-source LLM operations agent that turns Langfuse traces into actionable insights through an Observe → Analyze → Experiment → Improve cycle.

Ask natural-language questions in the web console, and the AI agent queries traces, diagnoses issues, suggests prompt improvements, and compares model outputs side by side.

---

Features

AI Agent (LangGraph)

• Trace analytics — filter by date, user, session, tags, metadata; summarize cost/latency/error trends
• Observability diagnostics — error classification, latency & cost trends, bottleneck detection
• Prompt engineering — A/B/C prompt rewrites with automatic scoring and a recommended variant
• Model comparison — run the same input across multiple LLMs and compare outputs
• Trace replay — re-run historical traces with a different model or prompt
• A/B experiments — dataset-driven variant execution + LLM-as-Judge scoring
• Response ranking — automatically compare and rank multiple model responses

Web Console (FastAPI + Jinja2)

Page	Purpose
Dashboard	Server health, chat stats, audit events, LLM key status
Agent Chat	Multi-turn conversation for trace analysis, prompt tuning, evaluation
Prompts	Langfuse prompt CRUD, version control, label promotion
Datasets	Langfuse dataset management, trace-to-item import
API Console	Dynamic Langfuse OpenAPI explorer and executor
Reports	Daily / weekly / monthly activity reports with auto-scheduling
Experiments	Trace replay, A/B testing, evaluation, response comparison
Settings	LLM keys, Langfuse connections, multi-project, local inference
Admin	User & API token management, audit logs, system health

Additional

• i18n — Korean / English real-time switching
• Dark / Light theme
• RBAC — 6 roles: viewer, analyst, editor, owner, ops, super_admin
• Multi-project — per-user multiple Langfuse project connections (encrypted at rest)
• Report scheduler — asyncio-based auto-generation (daily/weekly/monthly, KST)
• JSON / CSV export — experiment result download
• Dynamic OpenAPI tools — agent can call Langfuse API endpoints directly
• Python 3.11–3.14 — on 3.14, automatically uses Langfuse REST fallback

---

Supported LLMs

Provider	Examples
OpenAI	gpt-4o, gpt-4o-mini, gpt-4.1, o1, o3-mini
Anthropic	claude-sonnet-4-20250514, claude-haiku
Google	gemini-2.0-flash, gemini-2.5-pro
OpenRouter	deepseek/deepseek-chat-v3, meta-llama, etc.
Ollama (local)	llama3, phi3, mistral, etc.
vLLM / LM Studio	Local inference servers

---

Quick Start

1. Install

pip install langfuse-lens-agent

Or from source:

git clone https://github.com/BAEM1N/langfuse-lens-agent.git
cd langfuse-lens-agent
pip install -e .

2. Configure

cp .env.example .env

Minimal .env:

# At least one LLM key
OPENAI_API_KEY=sk-...

# Bootstrap admin (first run)
AGENT_BOOTSTRAP_ADMIN_TOKEN=your-secret-token

Langfuse connections are best configured per-user in the web UI (Settings > Langfuse Projects).

3. Run

langfuse-lens-web

Open http://localhost:8010 and log in with the bootstrap token.

---

Configuration

.env key reference

# Langfuse (system-level fallback, optional)
LANGFUSE_BASE_URL=https://cloud.langfuse.com
LANGFUSE_PUBLIC_KEY=
LANGFUSE_SECRET_KEY=
LANGFUSE_ENVIRONMENT=production

# LLM provider keys
OPENAI_API_KEY=
ANTHROPIC_API_KEY=
GOOGLE_API_KEY=
OPENROUTER_API_KEY=
OPENROUTER_BASE_URL=https://openrouter.ai/api/v1

# Local inference (optional)
OLLAMA_BASE_URL=
VLLM_BASE_URL=
LMSTUDIO_BASE_URL=

# Agent behaviour
AGENT_LLM_TEMPERATURE=0.2
AGENT_LLM_MAX_WORKERS=3
AGENT_LLM_COMPARISON_MODELS=openai:gpt-4o,anthropic:claude-sonnet-4-20250514

# Security
AGENT_BOOTSTRAP_ADMIN_TOKEN=
AGENT_DATA_ENCRYPTION_KEY=          # Fernet key for credential encryption
AGENT_AUTH_ALLOW_SIGNUP=true
AGENT_LANGFUSE_CONTROL_ALLOW_MUTATION=false

# Report scheduler (KST)
REPORT_SCHEDULER_ENABLED=true
REPORT_SCHEDULER_DAILY_HOUR=0
REPORT_SCHEDULER_DAILY_MINUTE=0

# Admin token policy
AGENT_TOKEN_DEFAULT_EXPIRES_DAYS=30
AGENT_TOKEN_DEFAULT_ROTATE_BEFORE_DAYS=3

Config.yaml (optional)

Supplements missing .env keys. Search order:

./Config.yaml → ./config/Config.yaml → AGENT_CONFIG_YAML_PATH

---

Docker

cp .env.example .env
# edit .env
docker compose up -d --build
curl http://localhost:8091/health

The container runs as a non-root user with a built-in health check.

---

Project Structure

langfuse-lens-agent/
├── api/                        # FastAPI routers & middleware
│   ├── routers/                # auth, admin, langfuse, experiments, reports, settings, pages
│   ├── deps.py                 # Auth / authz dependency helpers
│   ├── models.py               # Pydantic request/response models
│   └── state.py                # RBAC role & permission definitions
├── src/                        # Core agent logic
│   ├── agent_core.py           # LangGraph agent builder
│   ├── report_generator.py     # Daily / weekly / monthly report generation
│   ├── report_scheduler.py     # Async report auto-scheduler
│   ├── user_store.py           # SQLite store (users, chats, reports, experiments)
│   ├── tools/                  # Agent tools
│   │   ├── analysis.py         # Trace analysis & model comparison
│   │   ├── experiment_tools.py # Replay, A/B, evaluation, ranking
│   │   ├── langfuse_fetch.py   # Langfuse data fetching
│   │   └── openapi_registry.py # Dynamic OpenAPI tool generation
│   └── web_entry.py            # CLI entry point
├── templates/                  # Jinja2 HTML templates (11 pages)
├── static/                     # CSS, JS, locales (en/ko)
├── config/                     # Config.yaml, tool presets, runtime profiles
├── skills/                     # Agent skill references
├── tests/                      # 88 unit tests across 9 files
├── Dockerfile
├── docker-compose.yml
└── pyproject.toml

---

Security

Feature	Detail
Authentication	Bearer token (session-based) + bootstrap admin token
RBAC	6 hierarchical roles with scope-based permissions
Encryption	Fernet encryption for user Langfuse credentials at rest
Token management	Expiry, auto-rotation, revocation via admin API
Audit logging	All admin and write operations logged to SQLite
Timing-safe comparison	secrets.compare_digest for token validation
Input validation	Pydantic max_length, date parsing, field validators
Docker	Non-root user, health check, .dockerignore for secrets

---

API Overview

Method	Path	Description
POST	/api/auth/login	Log in
POST	/api/auth/register	Register
GET	/api/auth/me	Current user info
POST	/api/chat	Agent chat message
GET	/api/chat/threads	Chat thread list
GET	/api/reports	Report list
POST	/api/reports/generate	Generate report (daily/weekly/monthly)
GET	/api/experiments	Experiment list
POST	/api/experiments	Create experiment
GET	/api/experiments/{id}/export	Export as JSON/CSV
GET	/api/langfuse/prompts	Prompt list
GET	/api/langfuse/datasets	Dataset list
POST	/api/langfuse/openapi/execute	Execute Langfuse API operation
GET	/api/langfuse/tools/catalog	Agent tool catalog
GET	/api/admin/users	User management
GET	/api/admin/tokens	API token management
GET	/api/admin/audit-logs	Audit log list
GET	/api/admin/system/health	System health

Full API is browsable in the API Console page of the web UI.

---

Development

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
python -m pytest tests/ -v

# Dev server (hot reload)
langfuse-lens-web --reload

---

License

MIT — LICENSE

---

Contributing

Issues and PRs are welcome: GitHub Issues