ghostfolio-ai-agent 0.2.0

AI-powered conversational portfolio assistant for Ghostfolio with tool calling, verification, and evaluation framework

Ghostfolio AI Agent

AI-powered conversational portfolio assistant for Ghostfolio. Ask natural-language questions about your portfolio and get verified, fact-grounded responses.

Architecture

graph TD
    A[Streamlit Chat UI] -->|HTTP| B[FastAPI Server]
    B --> C{Agent}
    C -->|Primary| D[LLM Mode<br/>OpenRouter / OpenAI / Anthropic]
    C -->|Fallback| E[Rule-Based Mode]
    D --> F[Tool Layer<br/>7 Tools]
    E --> F
    F --> G{Data Source}
    G -->|Testing| H[Mock Provider]
    G -->|Production| I[Ghostfolio API]
    F --> J[Verification Layer]
    J -->|Traces| K[Langfuse]

Dual-mode agent: LLM-powered tool calling with automatic rule-based fallback. Every response passes through fact-grounding, disclaimer enforcement, and confidence scoring.

Features

• 7 portfolio tools: Summary, performance, transactions, accounts, market data, allocation analysis, risk rules
• LLM integration: Configurable model via OpenRouter (GPT, Claude) with direct OpenAI/Anthropic fallback
• Verification layer: Fact grounding, financial disclaimer, trade advice refusal, prompt injection defense
• Observability: Langfuse tracing for tool calls, LLM invocations, and verification
• 50+ eval test cases: Deterministic checks + LLM-as-judge scoring
• Production-ready: FastAPI + Streamlit, Railway deployment, CI/CD with linting and evals

Quick Start

# 1. Setup
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

# 2. Configure (optional — works with mock data by default)
cp .env.example .env  # Add API keys if desired

# 3. Run API
uvicorn app.main:app --reload

# 4. Run Chat UI (new terminal)
streamlit run ui/streamlit_app.py

# 5. Test
pytest -v

# 6. Run evals
python evals/run_evals.py

Configuration

Environment variables (prefix: GHOSTFOLIO_):

Core

Variable	Default	Description
GHOSTFOLIO_DEFAULT_DATA_SOURCE	mock	mock or ghostfolio_api
GHOSTFOLIO_BASE_URL	https://ghostfol.io	Ghostfolio instance URL
GHOSTFOLIO_REQUEST_TIMEOUT_SECONDS	10	HTTP timeout

LLM (OpenRouter — recommended)

The easiest way to configure the agent's LLM is via OpenRouter, which provides a unified API for multiple providers. Set two env vars:

Variable	Default	Description
GHOSTFOLIO_OPENROUTER_API_KEY	—	OpenRouter API key
GHOSTFOLIO_AGENT_MODEL	—	Model to use (see table below)

Available models:

AGENT_MODEL value	Routed to
gpt-o	openai/gpt-5.2
gpt-mini	openai/gpt-5.1-chat
claude-sonnet	anthropic/claude-sonnet-4.6
claude-opus	anthropic/claude-opus-4.6

You can also pass a raw OpenRouter model ID (e.g. anthropic/claude-haiku-4-5-20251001) for any model available on OpenRouter.

LLM (direct API keys — fallback)

If OpenRouter is not configured, the agent falls back to direct provider keys:

Variable	Default	Description
GHOSTFOLIO_OPENAI_API_KEY	—	OpenAI API key
GHOSTFOLIO_OPENAI_MODEL	gpt-4.1	OpenAI model
GHOSTFOLIO_ANTHROPIC_API_KEY	—	Anthropic API key
GHOSTFOLIO_ANTHROPIC_MODEL	claude-sonnet-4-20250514	Anthropic model
GHOSTFOLIO_LLM_ENABLED	true	Enable/disable LLM mode

Priority: OpenRouter > direct OpenAI > direct Anthropic > rule-based fallback.

Observability

Variable	Default	Description
GHOSTFOLIO_LANGFUSE_PUBLIC_KEY	—	Langfuse public key
GHOSTFOLIO_LANGFUSE_SECRET_KEY	—	Langfuse secret key
GHOSTFOLIO_LANGFUSE_HOST	https://cloud.langfuse.com	Langfuse host

Logging

Variable	Default	Description
GHOSTFOLIO_LOG_LEVEL	INFO	DEBUG|INFO|WARNING|ERROR
GHOSTFOLIO_LOG_FORMAT	json	json or text

Tools

Tool	Description
get_portfolio_summary	Portfolio value, holdings, allocations
get_performance	Returns for time ranges (1d, ytd, 1y, 5y, max)
get_transactions	Buy/sell activity history
get_account_details	Linked brokerage accounts and balances
get_market_data	Current prices for stock/ETF symbols
analyze_allocation	Sector, region, asset class breakdown + risk flags
check_risk_rules	Concentration, diversification, asset class risk checks

Verification

Every response is verified before delivery:

• Fact grounding: Numerical claims traced to tool output
• Disclaimer: Financial disclaimer on every response
• Trade advice refusal: Buy/sell recommendations politely refused
• Prompt injection defense: Override attempts detected and blocked
• Data freshness: Stale data warnings (>6h old)
• Confidence scoring: 0.4 (low) — 0.95 (high)

Evaluation

# Deterministic evals (50+ test cases, >80% gate)
python evals/run_evals.py

# LLM-as-judge (requires OpenAI key, advisory)
python evals/llm_judge.py

Categories: happy path (21), edge cases (10), adversarial (12), multi-step (10)

Multi-Model Comparison

Compare agent configurations side-by-side across the full eval dataset. Runs each model against all 53 cases using MockFileDataProvider (48 holdings, 577 transactions, 5 accounts), scores with deterministic checks + LLM-as-judge, measures response time, and outputs a ranked comparison table.

Available models: rule-based, gpt-o, gpt-mini, claude-haiku, claude-sonnet, claude-opus.

# Quick smoke test (no API keys needed)
python evals/compare_models.py --models rule-based --no-judge

# Compare two LLM models with verbose per-case output
python evals/compare_models.py --models gpt-mini claude-sonnet -v

# Full run (all models except opus, with LLM judge)
python evals/compare_models.py

# Include opus (expensive)
python evals/compare_models.py --include-expensive

# Filter by eval category
python evals/compare_models.py --models gpt-o claude-sonnet --categories happy_path edge_cases

Output includes a ranked summary table, per-category breakdown, and a detailed JSON results file. Ranking composite score: 0.4 × det_pass_rate + 0.4 × (judge/5) + 0.2 × (1 − error_rate).

Deployment (Railway)

1. Create a Railway project from this repo
2. Set environment variables:
   • GHOSTFOLIO_DEFAULT_DATA_SOURCE=mock (or ghostfolio_api)
   • GHOSTFOLIO_OPENROUTER_API_KEY=sk-or-... and GHOSTFOLIO_AGENT_MODEL=claude-sonnet (recommended)
   • Or GHOSTFOLIO_OPENAI_API_KEY=sk-... (direct OpenAI, alternative)
   • GHOSTFOLIO_LANGFUSE_PUBLIC_KEY / GHOSTFOLIO_LANGFUSE_SECRET_KEY (optional)
3. Deploy — Railway uses Procfile: web: bash scripts/start.sh
4. The Streamlit UI is the public entrypoint on $PORT

Project Structure

ghostfolio-agent/
├── app/
│   ├── agent.py           # Dual-mode LLM + rule-based agent
│   ├── config.py          # Environment-based settings
│   ├── ghostfolio_client.py # HTTP client with retry
│   ├── llm.py             # LLM factory (OpenRouter/OpenAI/Anthropic)
│   ├── main.py            # FastAPI server
│   ├── observability.py   # Langfuse tracing
│   ├── schemas.py         # Pydantic models
│   ├── telemetry.py       # Structured logging
│   ├── tool_defs.py       # Tool schemas for LLM
│   ├── tools.py           # 7 tool implementations
│   └── data_sources/
│       ├── base.py        # Provider protocol
│       ├── mock_provider.py
│       ├── mock_file_provider.py  # Large dataset provider
│       └── ghostfolio_api_provider.py
├── evals/
│   ├── eval_dataset.json  # 50+ test cases
│   ├── run_evals.py       # Deterministic eval runner
│   ├── llm_judge.py       # LLM-as-judge scorer
│   └── compare_models.py  # Multi-model comparison
├── tests/                 # pytest test suite
├── ui/
│   └── streamlit_app.py   # Chat interface
├── docs/
│   ├── architecture.md    # Architecture documentation
│   └── cost_analysis.md   # Cost projections
└── scripts/
    └── start.sh           # Railway startup script

Development

# Run tests
pytest -v

# Lint
ruff check app/ tests/ evals/

# Format
ruff format app/ tests/ evals/

License

See LICENSE.