raindrop-crewai

Raindrop integration for CrewAI (Python). Automatically captures crew kickoff invocations, multi-agent collaboration, task execution, and token usage by monkey-patching Crew.kickoff*.

Installation

pip install raindrop-crewai crewai

Quick Start

from raindrop_crewai import RaindropCrewAI
from crewai import Agent, Crew, Task

raindrop = RaindropCrewAI(
    api_key="your-write-key",
    user_id="user-123",
)
raindrop.setup()  # auto-patch all Crew.kickoff* methods

agent = Agent(
    role="Senior Researcher",
    goal="Find one interesting fact about {topic}",
    backstory="You are an experienced researcher.",
)
task = Task(
    description="Identify one interesting fact about {topic}.",
    expected_output="A single sentence fact.",
    agent=agent,
)
crew = Crew(agents=[agent], tasks=[task])

result = crew.kickoff(inputs={"topic": "AI safety"})
print(result.raw)

raindrop.shutdown()

Factory function (alternative)

from raindrop_crewai import create_raindrop_crewai

raindrop = create_raindrop_crewai(api_key="your-write-key", user_id="user-123")
wrapped = raindrop.wrap(crew)        # per-instance wrap (no global monkey-patch)
result = wrapped.kickoff(inputs={"topic": "AI safety"})
raindrop.shutdown()

Projects

Route events to a specific project by passing its slug as project_id:

raindrop = RaindropCrewAI(
    api_key="your-write-key",
    project_id="support-prod",
)

project_id sets the X-Raindrop-Project-Id header on every event. Omit it (or pass "default") to use your org's default Production project, which is the existing behavior. The same option is accepted by the create_raindrop_crewai(...) and setup_crewai(...) helpers. Invalid slugs are ignored with a warning and no header is sent.

What Gets Captured

• Crew kickoff invocations — input variables, crew name, process type (sequential/hierarchical)
• Agent metadata — agent roles and count
• Task metadata — task descriptions, count, and per-task summaries
• Token usage — ai.usage.prompt_tokens, ai.usage.completion_tokens, ai.usage.cached_tokens, ai.usage.total_tokens
• Model name — extracted from the first agent's llm.model (when set)
• Errors — error.type / error.message properties; the original exception is always re-raised
• Async support — kickoff(), kickoff_async(), kickoff_for_each(), kickoff_for_each_async() are all instrumented
• Nested OTel trace spans — crew workflow → agent → task → LLM, via the opentelemetry-instrumentation-crewai instrumentor (see Tracing; per-tool spans are not produced — see Known Limitations)

Configuration patterns

Auto-patch all crews (recommended)

Monkey-patches the Crew class so every kickoff* call is traced:

from raindrop_crewai import setup_crewai

raindrop = setup_crewai(api_key="your-write-key", user_id="user-123")
# Every Crew.kickoff(...) is now traced automatically.

Wrap a specific crew

Per-instance wrap with no global side effects:

raindrop = RaindropCrewAI(api_key="your-write-key", user_id="user-123")
wrapped = raindrop.wrap(crew)
result = wrapped.kickoff(inputs={"topic": "..."})

Tool catalog capture (ai.prompt.tools)

Every model span of an instrumented kickoff (kickoff, kickoff_async, kickoff_for_each, kickoff_for_each_async, via setup() or wrap()) records the crew's tool catalog as ai.prompt.tools: the union of every agent's tools and every task's tools, each described the way CrewAI describes it to the model (convert_tools_to_openai_schema, i.e. the tool's args_schema as a strict-mode JSON Schema). Model calls happen per agent with that agent's (and the current task's) tools, while the wrapper hooks the crew, so this crew-wide union is a catalog that may overstate what any one model call received — not the exact per-request list — and the OTel crewai instrumentor emits no tool definitions of its own. The inferred catalog is therefore marked with ai.prompt.tools.source = "crewai.agent.tools" (an explicit tools override carries no source). Each tool is one JSON document in an OTLP string array: {"type":"function","name","description","inputSchema"} for function tools (the JSON Schema is recorded as given — properties, enums, required and per-field descriptions intact) or {"type":"provider-defined","id","name","args"} for provider-owned tools, which get no invented schema (CrewAI has no provider-defined tools of its own; pass them through tools if your model receives any). A crew whose agents expose no tools list records no attribute; a crew whose agents have no tools records [].

Pass tools to record a different catalog instead (it replaces the inferred list; tools=[] records an empty catalog). The constructor value is the default for every kickoff the wrapper instruments — including setup()'s process-global patch; wrap(crew, tools=[...]) replaces it for that crew:

raindrop = RaindropCrewAI(api_key="your-write-key", tools=[{"type": "function", "name": "get_weather", "inputSchema": {...}}])
wrapped = raindrop.wrap(crew)                  # constructor default
other = raindrop.wrap(other_crew, tools=[])    # this crew: "no tools"

Tool definitions are prompt content: they follow the same switch as prompts, so TRACELOOP_TRACE_CONTENT=false never records ai.prompt.tools or ai.prompt.tools.source. The catalog itself needs raindrop-ai>=0.0.70; the ai.prompt.tools.source marker needs raindrop-ai>=0.0.71 (on 0.0.70 the catalog is recorded without the marker). The attribute lands on the LLM.call spans the OTel CrewAI instrumentor produces; on crewai>=1.x, LLM(model=...) returns a native provider class whose completions bypass LLM.call, so no model span (and no ai.prompt.tools) is emitted for those providers until the instrumentor covers them.

Debug Mode

Enable verbose logging with debug=True:

raindrop = RaindropCrewAI(api_key="your-write-key", debug=True)

This sets the raindrop_crewai logger to DEBUG, surfacing telemetry-side failures that are otherwise swallowed (so the user's pipeline never crashes due to instrumentation).

Identify Users

Associate events with a user identity after initialization:

raindrop.identify("user-123", {"name": "Alice", "plan": "pro"})

Track Signals

Attach feedback, edits, or other custom signals to a previously-shipped event by its event_id. The Python SDK does not currently expose a lastEventId accessor like the TypeScript client does, so to attach a signal you need either (a) the event_id returned to your application out-of-band (e.g. logged by debug=True), or (b) the public ID surfaced on the dashboard's event detail page.

raindrop.track_signal(
    event_id="<event-id-from-dashboard-or-debug-log>",
    name="thumbs_up",
    signal_type="feedback",
    sentiment="POSITIVE",
    comment="Great answer!",
)

Tracing

When tracing_enabled=True (the default), the integration activates the opentelemetry-instrumentation-crewai instrumentor via traceloop-sdk, producing nested OTel spans for every crew execution:

• Crew workflow — root span covering the entire kickoff() call (crewai.workflow)
• Agent execution — child span per agent ({role}.agent)
• Task execution — child span per task ({name}.task)
• LLM calls — leaf spans for each underlying LLM call, with model name and token usage

Trace spans land in the dashboard's Traces tab after async ingestion (typically 30–120s).

To ship flat events only without OTel spans:

raindrop = RaindropCrewAI(api_key="your-write-key", tracing_enabled=False)

Library instrumentation is scoped to CrewAI

As of 0.0.2, the integration pins the Raindrop SDK's auto_instrument=False and activates exactly one instrumentor (Instruments.CREWAI). It never lets Traceloop monkey-patch other LLM/tool client libraries it detects in your process (OpenAI client, MCP transports, etc.) — including on the fallback path taken when the installed traceloop-sdk is too old to know about the CrewAI instrumentor, which previously enabled blanket auto-instrumentation of every detected library.

Payload size bounds

Kickoff inputs are serialized with a hard 1,000,000-character budget and a ...[truncated by raindrop] marker. The bound is enforced during serialization (cost proportional to the cap, not the payload), so a multi-MB template variable can't stall your event loop or produce an oversized event that gets dropped at the ingest limit. Crew output text is capped by the Raindrop SDK's own per-field limit (max_text_field_chars, raindrop-ai >= 0.0.51).

Async Usage

import asyncio

async def main():
    result = await crew.kickoff_async(inputs={"topic": "AI safety"})
    print(result.raw)

    results = await crew.kickoff_for_each_async(
        inputs=[{"topic": "AI"}, {"topic": "ML"}]
    )

asyncio.run(main())

Flushing and Shutdown

raindrop.flush()     # flush pending data
raindrop.shutdown()  # flush + release resources (call before process exit)

API Reference

RaindropCrewAI

Parameter	Type	Default	Description
api_key	Optional[str]	None	Raindrop API key. If None, telemetry shipping is disabled (a UserWarning is issued).
user_id	Optional[str]	None	Default user identifier for all events.
convo_id	Optional[str]	None	Conversation/thread ID to group related events.
project_id	Optional[str]	None	Route events to a specific project (slug); omit for the default Production project.
tracing_enabled	bool	True	Enable distributed tracing via the OTel CrewAI instrumentor.
bypass_otel_for_tools	bool	True	Forwarded to raindrop.init(). Controls whether tool spans on the underlying SDK skip the OTLP path. The CrewAI integration itself does not call interaction.track_tool(), so this flag has no observable effect on CrewAI events — exposed for SDK API parity.
tools	Optional[Sequence[Mapping[str, Any]]]	None	Default ai.prompt.tools override for every instrumented kickoff, replacing the inferred crew tool catalog ([] = "no tools"). See Tool catalog capture.
debug	bool	False	Enable debug logging.

Methods

Method	Description
setup()	Monkey-patch Crew.kickoff* so every kickoff is traced (one-time, idempotent).
wrap(crew, *, tools=None)	Per-instance wrap of a single Crew (no global side effects). tools overrides ai.prompt.tools for this crew. Returns the wrapped crew.
flush()	Flush all pending events to the Raindrop API.
shutdown()	Flush remaining events and release resources.
identify(user_id, traits)	Identify a user with optional traits (Dict[str, str | int | bool | float]).
track_signal(event_id, name, ...)	Track a signal event. See track_signal signature in wrapper.py for the full keyword-only parameters.
last_event_id	event_id of the most recent event this wrapper captured, or None
resume_interaction(event_id=None)	Reopen an event (default: last_event_id) as a core Interaction to update it; None when there is nothing to resume

Per-call properties and identity

Crew.kickoff() has no channel for your own event data, so wrap the call in event_context. Its properties merge into every event captured inside the block, and user_id / convo_id replace the wrapper's own values for those events:

from raindrop_crewai import event_context

with event_context(user_id="user-123", convo_id="conv-9", properties={"plan": "pro"}):
    crew.kickoff(inputs={"topic": "AI"})

Updating the last event

last_event_id is the event_id of the most recent event this wrapper captured. Use it to attach a signal, or reopen the event with resume_interaction() (defaults to last_event_id) and update it through the core SDK's Interaction:

interaction = raindrop.resume_interaction()
if interaction is not None:
    interaction.set_properties({"reviewed": True})
    interaction.finish()

Key clashes. If one of your properties has the same key as one the wrapper derives itself, your value wins, whichever update carries it (a debug-level log line names the key). The wrapper's own keys are ai.* (for example ai.usage.prompt_tokens), error/error.* and crewai.*; avoid them unless you mean to replace the wrapper's value.

Property-only updates need raindrop-ai 0.0.76 or newer; older core versions drop an update that carries no input or output.

last_event_id is per wrapper instance; under concurrent calls it is whichever event was captured last, so pass event_id= when you need to correlate concurrent calls.

Module-level helpers

• create_raindrop_crewai(...) — construct a RaindropCrewAI and return a no-op instance on failure (never crashes).
• setup_crewai(...) — convenience: construct a RaindropCrewAI and call setup() in one step.

User id

Every event needs a real user id: set user_id= on the wrapper or per call with event_context(user_id=...). Without one (missing, blank, or a placeholder such as "unknown" or "anonymous"), the event is not sent and the wrapper logs one warning; it never raises. Updating an event you resumed with resume_interaction(event_id=...) needs no user id, since ingest keeps the one it stored.

Model spans

With tracing on (the default), the wrapper records each model call a kickoff makes (CrewAI's per-call LLM events) as a crewai.llm_call span on the Raindrop client's tracing, with that call's own provider, model and usage. CrewAI reports no usage on the completion of a model response that executed a tool, so those calls share one crewai.llm_calls span carrying the rest of the kickoff's total usage (total minus the calls that reported theirs); it names their provider only when they all share one provider and one routing, and is otherwise left unpriced rather than mispriced or double-priced. A failed model call that reported no usage still gets its own span with its error (and no usage). On a CrewAI without per-call events it records one crewai.kickoff span with the crew's total usage instead. Each span carries:

• the model: gen_ai.provider.name (canonical name, such as openai or amazon-bedrock), gen_ai.request.model and gen_ai.response.model;
• the token usage the provider reports: gen_ai.usage.input_tokens, output_tokens, cache_read_input_tokens, cache_write_input_tokens and reasoning_tokens;
• the event it belongs to: ai.telemetry.metadata.raindrop.eventId and the traceloop.association.properties.* keys.

Token usage also stays on the event as ai.usage.* properties. When an OpenTelemetry auto-instrumentation already records the model calls (for example with auto_instrument=True on the client), the wrapper adds no span for that call, so a call is never counted twice. Only an instrumentation that traces the call counts: the CrewAI instrumentor, the provider's client instrumentation for a native LLM, or LiteLLM's for an LLM routed through LiteLLM. In a crew mixing native and LiteLLM-routed agents with only LiteLLM instrumented, the native calls keep their spans.

Known Limitations

• No per-tool spans / empty event.toolCalls — the underlying opentelemetry-instrumentation-crewai instrumentor only wraps Crew.kickoff (workflow), Agent.execute_task (.agent), Task.execute_sync (.task), and LLM.call (LLM generation). It does NOT produce a span per tool invocation, so event.toolCalls on the dashboard will be empty for CrewAI runs. Tool usage surfaces only in the agent's final output text and indirectly in the agent/task span durations.
• Streaming — CrewAI's CrewStreamingOutput is returned to the caller as-is; the flat Raindrop event is shipped after the stream completes.
• finish_reason is per-LLM, not per-crew — CrewOutput does not expose finish_reason. Per-LLM finish reasons appear on the LLM trace spans inside the workflow trace, not on the flat event.
• Python SDK feature surface — the Python SDK is module-level and does not expose EventShipper / TraceShipper classes. The identify() and track_signal() methods are pass-through wrappers around the raindrop.analytics.* module functions.
• wrapt < 2 required for tracing — the OTel CrewAI instrumentor uses wrapt.wrap_function_wrapper(..., module=...) which was removed in wrapt 2.0. The package's dev dependencies pin wrapt<2; production environments using wrapt 2.x will see no trace spans (the flat event still ships).

Application Git metadata

RaindropCrewAI(...), create_raindrop_crewai(...), and setup_crewai(...) accept the keyword-only app_git option. It defaults to True: explicit Raindrop Git environment or deployment context is applied immediately, and the base SDK may perform one bounded background local-Git lookup from the process working directory. Event capture, flush, and shutdown never wait for that lookup. Pass False to disable enrichment, or pass an AppGitOptions mapping with commit_sha, commit_dirty, branch, source_directory, detect_branch, and/or auto_detect. Automatic branch discovery remains opt-in through detect_branch=True (or RAINDROP_GIT_DETECT_BRANCH=true).

For an ordinary in-process application, the process working directory is treated as the application-under-test checkout. A remote, coding, workflow, or observer process must not rely on its own checkout: pass app_git=False, provide explicit revision values, or set source_directory to the actual application checkout. Canonical per-operation properties remain authoritative. When supplying client=, configure app_git while constructing that Raindrop client; the supplied client is authoritative and the wrapper's app_git argument does not reconfigure it.

Release order is deliberate: first publish the base SDK feature, then publish the wrapper feature release with its minimum dependency coordinated to that base release. The existing raindrop-ai lower bound remains compatible, but application Git metadata is unavailable on an older core and must not be claimed complete until the base is upgraded. Until coordination assigns a released version, the wrapper checks for an explicit base app_git parameter and omits the option when unsupported. Explicit non-default configuration is debug-logged and omitted. Unsupported app_git is determined by signature inspection before construction, not by retrying initialization after a TypeError; Git configuration adds no initialization attempts and does not change any existing framework-specific initialization fallback.

Standard options and flush results

When the wrapper builds its own client, it accepts the core SDK's standard options by the same names and passes them on:

raindrop = RaindropCrewAI(
    api_key="rk_...",
    user_id="user-123",
    redact_pii=True,        # local PII redaction of event input/output
    flush_timeout=5,        # seconds; deadline for flush() and shutdown()
    debug=True,             # also raises the core SDK's log verbosity
    max_queue_size=5_000,   # any other option `Raindrop(...)` declares
)

result = raindrop.flush()   # the core's FlushResult; safe to ignore

• An option the installed raindrop-ai does not support is ignored with a debug log, never an error. flush_timeout and the FlushResult return value need raindrop-ai 0.0.76 or later; on older releases flush() and shutdown() return None.
• With client= (a Raindrop you built), configure these options on that client; the wrapper ignores them (debug log).
• flush() and shutdown() never raise. The result is truthy when everything was delivered, and reports pending, dropped and timed_out.

Testing

cd packages/crewai-python
pip install -e ".[crewai,dev]"
python -m pytest tests/ -v   # unit tests (no external services)

End-to-end behavior is verified by the cross-SDK conformance harness. This package ships a thin conformance driver at conformance/driver.py that maps the shared scenario corpus onto the wrapper's public API; known gaps are tracked as ticket-linked entries in conformance/failures.txt. The fault lane runs on every PR touching packages/*-python/** (.github/workflows/conformance-wrappers-python.yml) against a local capture server; the prod lane verifies delivery by reading back through the public Query API. The harness is pinned by commit SHA (HARNESS_REF). See the harness docs: HOW-IT-WORKS · AGENTS · README.

Documentation

Full documentation: Raindrop CrewAI Integration.