graphsight-langgraph

See exactly why your LangGraph agent picked the context it picked.

Retrieval in an agent is a black box: documents go in, an answer comes out, and when the answer is wrong you're left guessing which context misled it. This package opens the box. One callback handler records a LangGraph run — every node, every retriever call, per-document scores, and (for graph-aware retrievers) the relational paths between retrieved entities — and one command renders it as an interactive graph in your browser.

your LangGraph agent ──▶ LangGraphTracer ──▶ AgentTrace (v0.1) ──▶ graphsight viewer
                          (callbacks)         neutral JSON          interactive graph

Only dependency: langchain-core. No engine, no backend, no account — nothing leaves your machine.

Installation

pip install graphsight-langgraph            # the tracer
pip install "graphsight-langgraph[example]" # + langgraph, for the GitHub CLI
pip install graphsight                      # the local viewer (recommended)

Compatibility

Quickstart: trace a GitHub repo in 60 seconds

No setup, no API keys — public repositories don't need a token:

graphsight-github-trace langchain-ai/langgraph "who fixed the recent streaming bugs?"
graphsight graphsight_out/trace_state.json

Your browser opens on a live graph of that repository's recent activity: the PRs that matched the question, the people who authored them, the issues they resolve — every node clickable, scores shown, execution timeline included.

graphsight-github-trace reference

graphsight-github-trace REPO [QUESTION] [options]

Method note: the CLI builds a corpus with relational edges (person AUTHORED pr/commit, pr RESOLVES issue, pr TOUCHES repo) and ranks by lexical overlap with 1-hop graph expansion — deliberately simple, and reported as such. The scores you see are exactly the scores computed; nothing is presented as semantic similarity.

Ask it who / what / when questions ("who touched auth recently?", "which issue needed two attempts?") — that's what commit and PR history can answer. How-does-it-work questions need semantic retrieval over code and docs, which this deliberately simple demo does not pretend to do.

Tracing your own agent

The complete integration:

from graphsight_langgraph import LangGraphTracer, capture

tracer = LangGraphTracer()
result = graph.invoke(inputs, config={"callbacks": [tracer]})

capture(tracer, query="why is checkout failing?", answer=result["answer"])
# -> .graphsight/20260725T071558_why-is-checkout-failing.json

capture() finishes the trace and appends it to a local history directory (./.graphsight/ by default, $GRAPHSIGHT_DIR to override) — browse every run with graphsight .graphsight/. For manual control use tracer.finish() + to_tracestate() / save_trace(); trace.to_dict() gives the framework-neutral AgentTrace for your own tooling.

Retrieved vs. used

When you pass answer=, each retrieved item gets an answer_overlap score — the lexical overlap between the item's content and the final answer. In the viewer, items that surfaced in the answer render highlighted; items retrieved but unused render dimmed with the label "retrieved, unused." That splits the two classic retrieval failures at a glance:

• right doc retrieved, ignored by the model → dimmed node with a high retrieval score
• wrong doc trusted → highlighted node that shouldn't be

The overlap is a lexical heuristic (labeled as such, threshold 0.2) — it is never presented as a model-computed relevance judgment. No answer= → no usage claims; everything renders plain.

Configuration propagation (read this once)

LangChain only propagates callbacks into runnables that receive the run's config. Inside a LangGraph node, pass the node's config through to sub-runnables, or the tracer will record the node but not what happened inside it:

def retrieve(state, config):                               # 1. accept config
    docs = retriever.invoke(state["q"], config=config)     # 2. pass it through
    return {"docs": docs}

Symptom if you skip this: the trace shows node spans but zero retrievals.

API reference

What gets captured

Making your retriever graph-aware

The tracer reads two optional metadata conventions from the Document objects your retriever returns:

Document(
    page_content="PR #4821 'fix: idempotent refund path' merged by Priya N. ...",
    metadata={
        "id": "pr_4821",             # stable node id (falls back to a content hash)
        "label": "PR #4821",         # display name
        "kind": "pull_request",      # normalized to a viewer entity type, see below
        "score": 0.94,               # any recognized score key
        "source": "https://github.com/acme/platform/pull/4821",
        "edges": [                   # optional — enables the relational view
            {"source": "pr_4821", "target": "svc_checkout",
             "relation": "TOUCHES", "weight": 0.9},
        ],
    },
)

kind values are normalized (pull_request → PR, service → Service, person/author → Person, ticket/issue/jira → Ticket, repo → Repo, library → Library, team → Team, tool → Tool; anything else renders as Document).

Degradation behavior

• No edges in metadata → a flat scored-retrieval view instead of relational path highlighting. Still useful; not graph-aware.
• No recognized score keys → scores stay None; no score chips render. Nothing is ever fabricated.
• The emitted confidence.rationale states that scores came from your retriever and were not recomputed — an imported trace never masquerades as an engine-computed one.

Schema (v0.2)

AgentTrace is the stable contract. Future adapters (LlamaIndex, raw OpenTelemetry) emit the same shape and render in the same viewer. v0.2 adds RetrievedItem.answer_overlap (additive — v0.1 traces stay valid).

{
  "schema_version": "0.2",
  "framework": "langgraph",
  "query": "…",
  "spans": [
    { "id": "…", "name": "retrieve", "kind": "node",       // node | retriever | llm | tool
      "parent_id": null, "start_ms": 0.0, "end_ms": 0.3, "status": "ok" }
  ],
  "retrievals": [
    {
      "span_id": "…", "query": "…",
      "arm": "graph",                                       // "vector" | "graph", auto-detected
      "items": [
        { "id": "pr_4821", "label": "PR #4821", "kind": "pull_request",
          "score": 0.94, "vector_score": null, "graph_score": null,
          "answer_overlap": 0.41,                             // null when no answer given
          "content": "…", "source_uri": "https://…", "metadata": {} }
      ],
      "edges": [                                            // optional — the relational view
        { "source": "pr_4821", "target": "svc_checkout", "relation": "TOUCHES", "weight": 0.9 }
      ]
    }
  ],
  "answer": "…",
  "latency_ms": 3.9
}

Troubleshooting

Roadmap

In order: a LlamaIndex adapter emitting the same AgentTrace, then a raw OpenTelemetry span ingestor. The schema is the contract; adapters stay thin.

Links

• Source & issue tracker: github.com/Kcodess2807/graphsight
• The viewer: graphsight on PyPI
• Beta test script: BETA.md

License

MIT © Arush Karnatak