notilens-langchain 0.2.0

NotiLens callback handler for LangChain — automatic run tracking, token metrics, and error alerting

notilens-langchain

NotiLens callback handler for LangChain. Automatically tracks every chain, agent, LLM call, tool invocation, and retriever query — sending lifecycle events, token metrics, durations, and errors to NotiLens with zero manual instrumentation.

Installation

pip install notilens-langchain

Quick start

from notilens_langchain import NotiLensCallbackHandler

handler = NotiLensCallbackHandler(
    name="my-app",
    token="YOUR_TOKEN",    # or set NOTILENS_TOKEN env var
    secret="YOUR_SECRET",  # or set NOTILENS_SECRET env var
)

Attach to any chain or LLM — that's it:

# On a chain
result = chain.invoke({"input": "..."}, config={"callbacks": [handler]})

# On an LLM
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model="gpt-4o", callbacks=[handler])

# On an agent
agent_executor.invoke({"input": "..."}, config={"callbacks": [handler]})

What gets tracked automatically

Event	NotiLens action
Chain starts	run.start()
Chain completes	run.complete()
Chain errors	run.fail(error)
LLM / chat model called	run.progress("Calling gpt-4o…")
LLM response received	run.metric(prompt_tokens, completion_tokens, total_tokens) + duration
LLM timeout (> call_timeout)	run.timeout()
LLM error	run.error()
Agent uses a tool	run.loop("Agent step #N: using tool 'X'") + run.progress("Step N: using tool 'X'")
Agent finishes	run.complete(output)
Tool starts	run.progress("Running tool 'calculator'…")
Tool completes	run.progress("Tool completed")
Tool errors	run.error()
Retriever starts	run.progress("Retrieving context for: …")
Retriever returns docs	run.progress("Retrieved 5 documents")
Retriever errors	run.error()

Configuration

handler = NotiLensCallbackHandler(
    name="my-app",              # name shown in NotiLens
    token="YOUR_TOKEN",         # NotiLens topic token
    secret="YOUR_SECRET",       # NotiLens topic secret
    task="rag-pipeline",        # optional fixed task label (default: chain class name)
    call_timeout=30.0,          # LLM seconds before timeout event fires (default: 30)
    send_output=True,           # send final agent output as output.generated event (default: False)
    max_output_length=2000,     # max characters of output to send (default: 2000)
    min_level="info",           # minimum event level: "info", "warning", "error"
)

Custom metrics & events

You can set custom metrics or fire custom events on the active run directly from the handler:

# Numeric values accumulate across calls; strings are replaced
handler.metric('cost_usd', 0.004)
handler.metric('cache_hits', 1)
handler.metric('model', 'gpt-4o')

# Fire a custom event with an optional level and metadata
handler.track('cache.hit', 'Retrieved from vector cache', level='info')
handler.track('guardrail.triggered', 'Blocked unsafe output', level='warning')

# Send a rich notification with URLs and tags
handler.notify('report.ready', 'Summary complete',
    download_url='https://example.com/report.pdf',
    tags='langchain,rag',
)
handler.notify('result.image', 'Chart generated',
    image_url='https://example.com/chart.png',
    open_url='https://example.com/dashboard',
)

Note: Calling metric(), track(), or notify() when no chain is running logs a warning and does nothing:

WARNING NotiLens: handler.metric('cost_usd', ...) called with no active run — ignoring.

Reusing an existing NotiLens instance

import notilens
from notilens_langchain import NotiLensCallbackHandler

nl = notilens.init("my-app", token="...", secret="...")
handler = NotiLensCallbackHandler(nl_agent=nl)

Environment variables

export NOTILENS_TOKEN="your_token"
export NOTILENS_SECRET="your_secret"

# Token and secret are picked up automatically
handler = NotiLensCallbackHandler(name="my-app")