raindrop-crewai

Raindrop integration for CrewAI (Python). Automatically captures crew kickoff invocations, multi-agent collaboration, task execution, and token usage by monkey-patching Crew.kickoff*.

Installation

pip install raindrop-crewai crewai

Quick Start

from raindrop_crewai import RaindropCrewAI
from crewai import Agent, Crew, Task

raindrop = RaindropCrewAI(
    api_key="your-write-key",
    user_id="user-123",
)
raindrop.setup()  # auto-patch all Crew.kickoff* methods

agent = Agent(
    role="Senior Researcher",
    goal="Find one interesting fact about {topic}",
    backstory="You are an experienced researcher.",
)
task = Task(
    description="Identify one interesting fact about {topic}.",
    expected_output="A single sentence fact.",
    agent=agent,
)
crew = Crew(agents=[agent], tasks=[task])

result = crew.kickoff(inputs={"topic": "AI safety"})
print(result.raw)

raindrop.shutdown()

Factory function (alternative)

from raindrop_crewai import create_raindrop_crewai

raindrop = create_raindrop_crewai(api_key="your-write-key", user_id="user-123")
wrapped = raindrop.wrap(crew)        # per-instance wrap (no global monkey-patch)
result = wrapped.kickoff(inputs={"topic": "AI safety"})
raindrop.shutdown()

Projects

Route events to a specific project by passing its slug as project_id:

raindrop = RaindropCrewAI(
    api_key="your-write-key",
    project_id="support-prod",
)

project_id sets the X-Raindrop-Project-Id header on every event. Omit it (or pass "default") to use your org's default Production project, which is the existing behavior. The same option is accepted by the create_raindrop_crewai(...) and setup_crewai(...) helpers. Invalid slugs are ignored with a warning and no header is sent.

What Gets Captured

• Crew kickoff invocations — input variables, crew name, process type (sequential/hierarchical)
• Agent metadata — agent roles and count
• Task metadata — task descriptions, count, and per-task summaries
• Token usage — ai.usage.prompt_tokens, ai.usage.completion_tokens, ai.usage.cached_tokens, ai.usage.total_tokens
• Model name — extracted from the first agent's llm.model (when set)
• Errors — error.type / error.message properties; the original exception is always re-raised
• Async support — kickoff(), kickoff_async(), kickoff_for_each(), kickoff_for_each_async() are all instrumented
• Nested OTel trace spans — crew workflow → agent → task → LLM, via the opentelemetry-instrumentation-crewai instrumentor (see Tracing; per-tool spans are not produced — see Known Limitations)

Configuration patterns

Auto-patch all crews (recommended)

Monkey-patches the Crew class so every kickoff* call is traced:

from raindrop_crewai import setup_crewai

raindrop = setup_crewai(api_key="your-write-key", user_id="user-123")
# Every Crew.kickoff(...) is now traced automatically.

Wrap a specific crew

Per-instance wrap with no global side effects:

raindrop = RaindropCrewAI(api_key="your-write-key", user_id="user-123")
wrapped = raindrop.wrap(crew)
result = wrapped.kickoff(inputs={"topic": "..."})

Tool catalog capture (ai.prompt.tools)

Every model span of an instrumented kickoff (kickoff, kickoff_async, kickoff_for_each, kickoff_for_each_async, via setup() or wrap()) records the crew's tool catalog as ai.prompt.tools: the union of every agent's tools and every task's tools, each described the way CrewAI describes it to the model (convert_tools_to_openai_schema, i.e. the tool's args_schema as a strict-mode JSON Schema). Model calls happen per agent with that agent's (and the current task's) tools, while the wrapper hooks the crew, so this crew-wide union is a catalog that may overstate what any one model call received — not the exact per-request list — and the OTel crewai instrumentor emits no tool definitions of its own. The inferred catalog is therefore marked with ai.prompt.tools.source = "crewai.agent.tools" (an explicit tools override carries no source). Each tool is one JSON document in an OTLP string array: {"type":"function","name","description","inputSchema"} for function tools (the JSON Schema is recorded as given — properties, enums, required and per-field descriptions intact) or {"type":"provider-defined","id","name","args"} for provider-owned tools, which get no invented schema (CrewAI has no provider-defined tools of its own; pass them through tools if your model receives any). A crew whose agents expose no tools list records no attribute; a crew whose agents have no tools records [].

Pass tools to record a different catalog instead (it replaces the inferred list; tools=[] records an empty catalog). The constructor value is the default for every kickoff the wrapper instruments — including setup()'s process-global patch; wrap(crew, tools=[...]) replaces it for that crew:

raindrop = RaindropCrewAI(api_key="your-write-key", tools=[{"type": "function", "name": "get_weather", "inputSchema": {...}}])
wrapped = raindrop.wrap(crew)                  # constructor default
other = raindrop.wrap(other_crew, tools=[])    # this crew: "no tools"

Tool definitions are prompt content: they follow the same switch as prompts, so TRACELOOP_TRACE_CONTENT=false never records ai.prompt.tools or ai.prompt.tools.source. The catalog itself needs raindrop-ai>=0.0.70; the ai.prompt.tools.source marker needs raindrop-ai>=0.0.71 (on 0.0.70 the catalog is recorded without the marker). The attribute lands on the LLM.call spans the OTel CrewAI instrumentor produces; on crewai>=1.x, LLM(model=...) returns a native provider class whose completions bypass LLM.call, so no model span (and no ai.prompt.tools) is emitted for those providers until the instrumentor covers them.

Debug Mode

Enable verbose logging with debug=True:

raindrop = RaindropCrewAI(api_key="your-write-key", debug=True)

This sets the raindrop_crewai logger to DEBUG, surfacing telemetry-side failures that are otherwise swallowed (so the user's pipeline never crashes due to instrumentation).

Identify Users

Associate events with a user identity after initialization:

raindrop.identify("user-123", {"name": "Alice", "plan": "pro"})

Track Signals

Attach feedback, edits, or other custom signals to a previously-shipped event by its event_id. The Python SDK does not currently expose a lastEventId accessor like the TypeScript client does, so to attach a signal you need either (a) the event_id returned to your application out-of-band (e.g. logged by debug=True), or (b) the public ID surfaced on the dashboard's event detail page.

raindrop.track_signal(
    event_id="<event-id-from-dashboard-or-debug-log>",
    name="thumbs_up",
    signal_type="feedback",
    sentiment="POSITIVE",
    comment="Great answer!",
)

Tracing

When tracing_enabled=True (the default), the integration activates the opentelemetry-instrumentation-crewai instrumentor via traceloop-sdk, producing nested OTel spans for every crew execution:

• Crew workflow — root span covering the entire kickoff() call (crewai.workflow)
• Agent execution — child span per agent ({role}.agent)
• Task execution — child span per task ({name}.task)
• LLM calls — leaf spans for each underlying LLM call, with model name and token usage

Trace spans land in the dashboard's Traces tab after async ingestion (typically 30–120s).

To ship flat events only without OTel spans:

raindrop = RaindropCrewAI(api_key="your-write-key", tracing_enabled=False)

Library instrumentation is scoped to CrewAI

As of 0.0.2, the integration pins the Raindrop SDK's auto_instrument=False and activates exactly one instrumentor (Instruments.CREWAI). It never lets Traceloop monkey-patch other LLM/tool client libraries it detects in your process (OpenAI client, MCP transports, etc.) — including on the fallback path taken when the installed traceloop-sdk is too old to know about the CrewAI instrumentor, which previously enabled blanket auto-instrumentation of every detected library.

Payload size bounds

Kickoff inputs are serialized with a hard 1,000,000-character budget and a ...[truncated by raindrop] marker. The bound is enforced during serialization (cost proportional to the cap, not the payload), so a multi-MB template variable can't stall your event loop or produce an oversized event that gets dropped at the ingest limit. Crew output text is capped by the Raindrop SDK's own per-field limit (max_text_field_chars, raindrop-ai >= 0.0.51).

Async Usage

import asyncio

async def main():
    result = await crew.kickoff_async(inputs={"topic": "AI safety"})
    print(result.raw)

    results = await crew.kickoff_for_each_async(
        inputs=[{"topic": "AI"}, {"topic": "ML"}]
    )

asyncio.run(main())

Flushing and Shutdown

raindrop.flush()     # flush pending data
raindrop.shutdown()  # flush + release resources (call before process exit)

API Reference

RaindropCrewAI

Methods

Module-level helpers

• create_raindrop_crewai(...) — construct a RaindropCrewAI and return a no-op instance on failure (never crashes).
• setup_crewai(...) — convenience: construct a RaindropCrewAI and call setup() in one step.

Known Limitations

• No per-tool spans / empty event.toolCalls — the underlying opentelemetry-instrumentation-crewai instrumentor only wraps Crew.kickoff (workflow), Agent.execute_task (.agent), Task.execute_sync (.task), and LLM.call (LLM generation). It does NOT produce a span per tool invocation, so event.toolCalls on the dashboard will be empty for CrewAI runs. Tool usage surfaces only in the agent's final output text and indirectly in the agent/task span durations.
• Streaming — CrewAI's CrewStreamingOutput is returned to the caller as-is; the flat Raindrop event is shipped after the stream completes.
• finish_reason is per-LLM, not per-crew — CrewOutput does not expose finish_reason. Per-LLM finish reasons appear on the LLM trace spans inside the workflow trace, not on the flat event.
• Python SDK feature surface — the Python SDK is module-level and does not expose EventShipper / TraceShipper classes. The identify() and track_signal() methods are pass-through wrappers around the raindrop.analytics.* module functions.
• wrapt < 2 required for tracing — the OTel CrewAI instrumentor uses wrapt.wrap_function_wrapper(..., module=...) which was removed in wrapt 2.0. The package's dev dependencies pin wrapt<2; production environments using wrapt 2.x will see no trace spans (the flat event still ships).

Application Git metadata

RaindropCrewAI(...), create_raindrop_crewai(...), and setup_crewai(...) accept the keyword-only app_git option. It defaults to True: explicit Raindrop Git environment or deployment context is applied immediately, and the base SDK may perform one bounded background local-Git lookup from the process working directory. Event capture, flush, and shutdown never wait for that lookup. Pass False to disable enrichment, or pass an AppGitOptions mapping with commit_sha, commit_dirty, branch, source_directory, detect_branch, and/or auto_detect. Automatic branch discovery remains opt-in through detect_branch=True (or RAINDROP_GIT_DETECT_BRANCH=true).

For an ordinary in-process application, the process working directory is treated as the application-under-test checkout. A remote, coding, workflow, or observer process must not rely on its own checkout: pass app_git=False, provide explicit revision values, or set source_directory to the actual application checkout. Canonical per-operation properties remain authoritative. When supplying client=, configure app_git while constructing that Raindrop client; the supplied client is authoritative and the wrapper's app_git argument does not reconfigure it.

Release order is deliberate: first publish the base SDK feature, then publish the wrapper feature release with its minimum dependency coordinated to that base release. The existing raindrop-ai lower bound remains compatible, but application Git metadata is unavailable on an older core and must not be claimed complete until the base is upgraded. Until coordination assigns a released version, the wrapper checks for an explicit base app_git parameter and omits the option when unsupported. Explicit non-default configuration is debug-logged and omitted. Unsupported app_git is determined by signature inspection before construction, not by retrying initialization after a TypeError; Git configuration adds no initialization attempts and does not change any existing framework-specific initialization fallback.

Testing

cd packages/crewai-python
pip install -e ".[crewai,dev]"
python -m pytest tests/ -v   # unit tests (no external services)

End-to-end behavior is verified by the cross-SDK conformance harness. This package ships a thin conformance driver at conformance/driver.py that maps the shared scenario corpus onto the wrapper's public API; known gaps are tracked as ticket-linked entries in conformance/failures.txt. The fault lane runs on every PR touching packages/*-python/** (.github/workflows/conformance-wrappers-python.yml) against a local capture server; the prod lane verifies delivery by reading back through the public Query API. The harness is pinned by commit SHA (HARNESS_REF). See the harness docs: HOW-IT-WORKS · AGENTS · README.

Documentation

Full documentation: Raindrop CrewAI Integration.