splunkguard-mcp

A small Gemini-driven agent that drives the official Splunk MCP Server (Splunkbase App #7931) to answer natural-language questions about your Splunk data.

Ask "What CI pipelines failed last night and why?" or "Are there auth anomalies in the last 24 hours?" and get back a typed SplunkInvestigationReport with root cause, failure category, time range, affected components, and paste-ready SPL recommendations.

Extracted from the SplunkGuard hackathon project and packaged so other agents can use the same Splunk-MCP-over-Streamable-HTTP plumbing without dragging in the rest of that codebase.

Why this library exists

The Splunk MCP Server App is great but two things tripped us up when integrating it from Python:

1. Transport mismatch. App #7931 v1.1.0 uses the modern MCP Streamable HTTP transport (per the MCP 2025-06-18 spec), not the older SSE transport. If you reach for mcp.client.sse.sse_client (the obvious one), you'll get HTTP 405 Method Not Allowed because Splunk's /services/mcp only accepts POST. You need mcp.client.streamable_http.streamablehttp_client.

2. Self-signed cert handling. Local Splunk dev instances use self-signed TLS. The MCP SDK doesn't auto-honor an ssl.SSLContext you hand it — you need to inject a custom httpx_client_factory into the transport. We wire that for you.

This library handles both correctly out of the box.

Install

pip install splunkguard-mcp        # once published; for now:
pip install git+https://github.com/64johnlee/splunkguard-mcp

Python ≥ 3.10. Pulls google-genai, mcp, and httpx as deps.

Quick start

1. Spin up Splunk + the MCP Server App

docker run -d --name splunk \
  -p 8000:8000 -p 8088:8088 -p 8089:8089 \
  -e SPLUNK_PASSWORD=changeme \
  -e SPLUNK_GENERAL_TERMS=--accept-sgt-current-at-splunk-com \
  -e SPLUNK_START_ARGS=--accept-license \
  splunk/splunk:latest

# Install Splunkbase App #7931 (download .tgz after logging in at
# https://splunkbase.splunk.com/app/7931, then copy into the container):
docker cp splunk_mcp_server.tgz splunk:/tmp/
docker exec -u splunk splunk /opt/splunk/bin/splunk install app \
  /tmp/splunk_mcp_server.tgz -auth admin:changeme

# IMPORTANT: restart the *container*, not splunkd. `splunk restart` in
# Docker kills splunkd but never re-spawns it; `docker restart` does.
docker restart splunk

# Generate the MCP token:
curl -sk -u admin:changeme -X POST \
  "https://localhost:8089/services/mcp_token?username=admin&action=rotate"
TOKEN=$(curl -sk -u admin:changeme \
  "https://localhost:8089/services/mcp_token?username=admin&action=get" \
  | python3 -c "import json,sys;print(json.load(sys.stdin)['token'])")
echo "$TOKEN"   # use this as SPLUNK_MCP_TOKEN below

2. Investigate via the CLI

export GEMINI_API_KEY=...                # https://aistudio.google.com
export SPLUNK_MCP_TOKEN="$TOKEN"

splunkguard-mcp investigate \
  "What CI pipelines failed in the last 30 days?" \
  --earliest -30d \
  --no-verify-ssl

3. Or use the Python API

import asyncio
from splunkguard_mcp import SplunkInvestigator

agent = SplunkInvestigator(
    gemini_api_key="...",
    splunk_token="...",
    splunk_url="https://localhost:8089/services/mcp",
    verify_ssl=False,                         # local dev with self-signed cert
)
report = asyncio.run(agent.investigate(
    "What CI pipelines failed last night and why?",
    earliest="-24h",
    latest="now",
))
print(report.root_cause)
for a in report.recommended_actions:
    print(f"  - {a.action} (confidence: {a.confidence})")
    print(f"    SPL: {a.spl_query}")

4. Or just the raw MCP client (BYO LLM)

import asyncio
from splunkguard_mcp import SplunkMCPClient

async def main():
    async with SplunkMCPClient(splunk_token="...", verify_ssl=False) as c:
        tools = await c.list_tools()
        for t in tools:
            print(t.name, "—", t.description[:80])
        out = await c.call_tool(
            "splunk_run_query",
            {"query": "search index=_internal | head 5"},
        )
        print(out)

asyncio.run(main())

Benchmark

End-to-end on a local Splunk Enterprise 10.4.0 (Docker splunk/splunk:latest) + Splunkbase App #7931 v1.1.0 + Gemini 2.5 Flash (Google AI Studio, free tier):

The MCP path is roughly 4× slower than direct REST because it gives Gemini real tool-call latency budget to iterate; the trade-off is deeper, more specific findings (specific failing-job names, is_ongoing inference, more SPL recommendations).

What's in the package

splunkguard_mcp/
├── __init__.py        — public API: SplunkInvestigator, SplunkMCPClient,
│                        SplunkInvestigationReport, RecommendedAction
├── __main__.py        — `python -m splunkguard_mcp investigate "<question>"`
├── client.py          — SplunkMCPClient: async context manager that
│                        connects via Streamable HTTP + custom httpx factory
├── agent.py           — SplunkInvestigator: Gemini tool-call loop +
│                        FunctionDeclaration translation + structured-output parsing
├── models.py          — dataclasses: SplunkInvestigationReport,
│                        RecommendedAction
└── prompts.py         — system prompt + user-turn template

Gotchas (verified the hard way)

License

MIT — see LICENSE.

Related

• The full SplunkGuard hackathon project (with the --direct REST fallback path + GitLab→Splunk HEC ingester): https://github.com/64johnlee/hackathon-pipeline-guard
• The official Splunk MCP Server App on Splunkbase: https://splunkbase.splunk.com/app/7931
• The Model Context Protocol spec: https://modelcontextprotocol.io