OpenInference LiteLLM Instrumentation

LiteLLM allows developers to call all LLM APIs using the openAI format. LiteLLM Proxy is a proxy server to call 100+ LLMs in OpenAI format. Both are supported by this auto-instrumentation.

This package implements OpenInference tracing for the following LiteLLM functions:

  • completion()
  • acompletion()
  • completion_with_retries()
  • embedding()
  • aembedding()
  • image_generation()
  • aimage_generation()

These traces are fully OpenTelemetry compatible and can be sent to an OpenTelemetry collector for viewing, such as Arize Phoenix.

Installation

pip install openinference-instrumentation-litellm

Quickstart

In a notebook environment (jupyter, colab, etc.) install openinference-instrumentation-litellm if you haven't already as well as arize-phoenix and litellm.

pip install openinference-instrumentation-litellm arize-phoenix litellm

First, import dependencies required to autoinstrument liteLLM and set up phoenix as an collector for OpenInference traces.

import litellm
import phoenix as px

from openinference.instrumentation.litellm import LiteLLMInstrumentor

from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor

Next, we'll start a phoenix server and set it as a collector.

session = px.launch_app()
endpoint = "http://127.0.0.1:6006/v1/traces"
tracer_provider = TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter(endpoint)))

Set up any API keys needed in you API calls. For example:

import os
os.environ["OPENAI_API_KEY"] = "PASTE_YOUR_API_KEY_HERE"

Instrumenting LiteLLM is simple:

LiteLLMInstrumentor().instrument(tracer_provider=tracer_provider)

Now, all calls to LiteLLM functions are instrumented and can be viewed in the phoenix UI.

completion_response = litellm.completion(model="gpt-3.5-turbo", 
                   messages=[{"content": "What's the capital of China?", "role": "user"}])
print(completion_response)

acompletion_response = await litellm.acompletion(
            model="gpt-3.5-turbo",
            messages=[{ "content": "Hello, I want to bake a cake","role": "user"},
                      { "content": "Hello, I can pull up some recipes for cakes.","role": "assistant"},
                      { "content": "No actually I want to make a pie","role": "user"},],
            temperature=0.7,
            max_tokens=20
        )
print(acompletion_response)

embedding_response = litellm.embedding(model='text-embedding-ada-002', input=["good morning!"])
print(embedding_response)

image_gen_response = litellm.image_generation(model='dall-e-2', prompt="cute baby otter")
print(image_gen_response)

You can also uninstrument the functions as follows

LiteLLMInstrumentor().uninstrument(tracer_provider=tracer_provider)

Now any liteLLM function calls you make will not send traces to Phoenix until instrumented again

More Info

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.1.41)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for arizeai from gravatar.com arizeai

Credits

Author: OpenInference Authors

License expression

Apache-2.0
View SPDX License List

Requires

Python <3.14, >=3.9

Provides Extra

test

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

openinference_instrumentation_litellm-0.1.19.tar.gz (16.7 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

openinference_instrumentation_litellm-0.1.19-py3-none-any.whl (12.4 kB view details)

Uploaded Python 3

File details

Details for the file openinference_instrumentation_litellm-0.1.19.tar.gz.

File metadata

File hashes

Hashes for openinference_instrumentation_litellm-0.1.19.tar.gz
Algorithm Hash digest
SHA256 23afa2dc0760d6c1d6f16e65609318e93564594e5062e3b9634f7b9db10d612d
MD5 3eaf59e888797f7a2967a12e17977791
BLAKE2b-256 bac53740ea7a357ff0cd837780089a7b6a3f71ce187da19807821d60c370c75f

See more details on using hashes here.

File details

Details for the file openinference_instrumentation_litellm-0.1.19-py3-none-any.whl.

File metadata

File hashes

Hashes for openinference_instrumentation_litellm-0.1.19-py3-none-any.whl
Algorithm Hash digest
SHA256 de357f22960a8b3798be2560d3388405a9229d863ace77018313d4eb44f76ed8
MD5 d54ff26ec1ee6fbc9d5766737b767e9f
BLAKE2b-256 f2c77cb069455b4ae45f4ae179fe5242198be19b05905aaf9e20536606f0423b

See more details on using hashes here.

Release history Release notifications | RSS feed

0.1.41

2 files

0.1.40

2 files

0.1.39

2 files

0.1.38

2 files

0.1.37

2 files

0.1.36

2 files

0.1.35

2 files

0.1.34

2 files

0.1.33

2 files

0.1.32

2 files

0.1.31

2 files

0.1.30

2 files

0.1.29

2 files

0.1.28

2 files

0.1.27

2 files

0.1.26

2 files

0.1.25

2 files

0.1.24

2 files

0.1.23

2 files

0.1.22

2 files

0.1.21

2 files

0.1.20

2 files

This release

0.1.19 This release

2 files

0.1.18

2 files

0.1.17

2 files

0.1.16

2 files

0.1.15

2 files

0.1.14

2 files

0.1.13

2 files

0.1.12

2 files

0.1.11

2 files

0.1.10

2 files

0.1.9

2 files

0.1.8

2 files

0.1.7

2 files

0.1.6

2 files

0.1.5

2 files

0.1.4

2 files

0.1.3

2 files

0.1.2

2 files

0.1.1

2 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page