LoongSuite BFCL v4 Instrumentation

LoongSuite Python instrumentation for the Berkeley Function Call Leaderboard v4 (bfcl-eval, package bfcl_eval).

Span Topology

ENTRY  enter_ai_application_system          gen_ai.span.kind=ENTRY,  op=enter
└─ AGENT  invoke_agent {test_entry_id}      gen_ai.span.kind=AGENT,  op=invoke_agent
   ├─ STEP  react step                      gen_ai.span.kind=STEP,   op=react
   │   ├─ LLM   chat {model}                (created by downstream vendor SDK probe)
   │   └─ TOOL  execute_tool {fn}           gen_ai.span.kind=TOOL,   op=execute_tool
   └─ STEP  react step
       └─ ...

This instrumentation deliberately does not create LLM spans. They are emitted by the downstream vendor SDK probe (OpenAI / Anthropic / Google / DashScope / LiteLLM / etc.) so that token usage and request payloads stay in sync with the SDK that actually performed the request.

Installation

pip install loongsuite-instrumentation-bfclv4

Usage

opentelemetry-instrument bfcl generate \
    --model gpt-4o-2024-11-20-FC \
    --test-category simple_python \
    --num-threads 2

Or programmatically:

from opentelemetry.instrumentation.bfclv4 import BFCLv4Instrumentor

BFCLv4Instrumentor().instrument()
# ... run BFCL ...
BFCLv4Instrumentor().uninstrument()

Compatibility With Downstream LLM SDK Probes

OSS Provider Notes

For OSS handlers (vLLM / SGLang served via the OpenAI-compatible API), the BFCL probe sets gen_ai.provider.name to vllm / sglang / oss and adds bfcl.oss.backend for disambiguation. Downstream OpenAI probes will still report gen_ai.provider.name=openai on the LLM span; this is expected.

Custom Attributes