notilens-langchain 0.1.0

NotiLens callback handler for LangChain — automatic run tracking, token metrics, and error alerting

notilens-langchain

NotiLens callback handler for LangChain. Automatically tracks every chain, agent, LLM call, tool invocation, and retriever query — sending lifecycle events, token metrics, durations, and errors to NotiLens with zero manual instrumentation.

Installation

pip install notilens-langchain

Quick start

from notilens_langchain import NotiLensCallbackHandler

handler = NotiLensCallbackHandler(
    agent="my-agent",
    token="YOUR_TOKEN",    # or set NOTILENS_TOKEN env var
    secret="YOUR_SECRET",  # or set NOTILENS_SECRET env var
)

Attach to any chain or LLM — that's it:

# On a chain
result = chain.invoke({"input": "..."}, config={"callbacks": [handler]})

# On an LLM
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model="gpt-4o", callbacks=[handler])

# On an agent
agent_executor.invoke({"input": "..."}, config={"callbacks": [handler]})

What gets tracked automatically

Event	NotiLens action
Chain starts	run.start()
Chain completes	run.complete()
Chain errors	run.fail(error)
LLM / chat model called	run.progress("Calling gpt-4o…")
LLM response received	run.metric(prompt_tokens, completion_tokens, total_tokens) + duration
LLM timeout (> call_timeout)	run.timeout()
LLM error	run.error()
Agent uses a tool	run.progress("Step N: using tool 'web_search'")
Agent loops (> loop_threshold steps)	run.loop()
Agent finishes	run.complete(output)
Tool starts	run.progress("Running tool 'calculator'…")
Tool completes	run.progress("Tool completed")
Tool errors	run.error()
Retriever starts	run.progress("Retrieving context for: …")
Retriever returns docs	run.progress("Retrieved 5 documents")
Retriever errors	run.error()

Configuration

handler = NotiLensCallbackHandler(
    agent="my-agent",           # agent name in NotiLens
    token="YOUR_TOKEN",         # NotiLens topic token
    secret="YOUR_SECRET",       # NotiLens topic secret
    task="rag-pipeline",        # optional fixed task label (default: chain class name)
    loop_threshold=10,          # agent steps before loop alert fires (default: 10)
    call_timeout=30.0,          # LLM seconds before timeout event fires (default: 30)
    min_level="info",           # minimum event level: "info", "warning", "error"
    silent=False,               # suppress SDK log output
)

Custom metrics & events

You can set custom metrics or fire custom events on the active run directly from the handler:

# Numeric values accumulate across calls; strings are replaced
handler.metric('cost_usd', 0.004)
handler.metric('cache_hits', 1)
handler.metric('model', 'gpt-4o')

# Fire a custom event with an optional level and metadata
handler.track('cache.hit', 'Retrieved from vector cache', level='info')
handler.track('guardrail.triggered', 'Blocked unsafe output', level='warning')

Note: Calling metric() or track() when no chain is running logs a warning and does nothing:

WARNING NotiLens: handler.metric('cost_usd', ...) called with no active run — ignoring.

Reusing an existing NotiLens agent

import notilens
from notilens_langchain import NotiLensCallbackHandler

nl = notilens.init("my-agent", token="...", secret="...")
handler = NotiLensCallbackHandler(nl_agent=nl)

Environment variables

export NOTILENS_TOKEN="your_token"
export NOTILENS_SECRET="your_secret"

# Token and secret are picked up automatically
handler = NotiLensCallbackHandler(agent="my-agent")