💡 Features
🚪 AI Gateway:
- Unified API Signature: If you've used OpenAI, you already know how to use Portkey with any other provider.
- Interoperability: Write once, run with any provider. Switch between any model from any provider seamlessly.
- Automated Fallbacks & Retries: Ensure your application remains functional even if a primary service fails.
- Load Balancing: Efficiently distribute incoming requests among multiple models.
- Semantic Caching: Reduce costs and latency by intelligently caching results.
🔬 Observability:
- Logging: Keep track of all requests for monitoring and debugging.
- Requests Tracing: Understand the journey of each request for optimization.
- Custom Tags: Segment and categorize requests for better insights.
🚀 Quick Start
4️ Steps to Integrate the SDK
- Get your Portkey API key and your virtual key for AI providers.
- Construct your LLM, add Portkey features, provider features, and prompt.
- Construct the Portkey client and set your usage mode.
- Now call Portkey regularly like you would call your OpenAI constructor.
Let's dive in! If you are an advanced user and want to directly jump to various full-fledged examples, click here.
Step 1️⃣ : Get your Portkey API Key and your Virtual Keys for AI providers
Portkey API Key: Log into Portkey here, then click on the profile icon on top left and “Copy API Key”.
import os
os.environ["PORTKEY_API_KEY"] = "PORTKEY_API_KEY"
Virtual Keys: Navigate to the "Virtual Keys" page on Portkey and hit the "Add Key" button. Choose your AI provider and assign a unique name to your key. Your virtual key is ready!
Step 2️⃣ : Construct your LLM, add Portkey features, provider features, and prompt
Portkey Features: You can find a comprehensive list of Portkey features here. This includes settings for caching, retries, metadata, and more.
Provider Features:
Portkey is designed to be flexible. All the features you're familiar with from your LLM provider, like top_p, top_k, and temperature, can be used seamlessly. Check out the complete list of provider features here.
Setting the Prompt Input:
This param lets you override any prompt that is passed during the completion call - set a model-specific prompt here to optimise the model performance. You can set the input in two ways. For models like Claude and GPT3, use prompt = (str), and for models like GPT3.5 & GPT4, use messages = [array].
Here's how you can combine everything:
from portkey import LLMOptions
# Portkey Config
provider = "openai"
virtual_key = "key_a"
trace_id = "portkey_sdk_test"
# Model Settings
model = "gpt-4"
temperature = 1
# User Prompt
messages = [{"role": "user", "content": "Who are you?"}]
# Construct LLM
llm = LLMOptions(provider=provider, virtual_key=virtual_key, trace_id=trace_id, model=model, temperature=temperature)
Steo 3️⃣ : Construct the Portkey Client
Portkey client's config takes 3 params: api_key, mode, llms.
api_key: You can set your Portkey API key here or withos.ennvironas done above.mode: There are 3 modes - Single, Fallback, Loadbalance.- Single - This is the standard mode. Use it if you do not want Fallback OR Loadbalance features.
- Fallback - Set this mode if you want to enable the Fallback feature.
- Loadbalance - Set this mode if you want to enable the Loadbalance feature.
llms: This is an array where we pass our LLMs constructed using the LLMOptions constructor.
import portkey
from portkey import Config
portkey.config = Config(mode="single",llms=[llm])
Step 4️⃣ : Let's Call the Portkey Client!
The Portkey client can do ChatCompletions and Completions.
Since our LLM is GPT4, we will use ChatCompletions:
response = portkey.ChatCompletions.create(
messages=[{
"role": "user",
"content": "Who are you ?"
}]
)
print(response.choices[0].message)
You have integrated Portkey's Python SDK in just 4 steps!
🔁 Demo: Implementing GPT4 to GPT3.5 Fallback Using the Portkey SDK
import os
os.environ["PORTKEY_API_KEY"] = "PORTKEY_API_KEY" # Setting the Portkey API Key
import portkey
from portkey import Config, LLMOptions
# Let's construct our LLMs.
llm1 = LLMOptions(provider="openai", model="gpt-4", virtual_key="key_a"),
llm2 = LLMOptions(provider="openai", model="gpt-3.5-turbo", virtual_key="key_a")
# Now let's construct the Portkey client where we will set the fallback logic
portkey.config = Config(mode="fallback",llms=[llm1,llm2])
# And, that's it!
response = portkey.ChatCompletions.create()
print(response.choices[0].message)
📔 Full List of Portkey Config
| Feature | Config Key | Value(Type) | Required |
|---|---|---|---|
| Provider Name | provider |
string |
✅ Required |
| Model Name | model |
string |
✅ Required |
| Virtual Key OR API Key | virtual_key or api_key |
string |
✅ Required (can be set externally) |
| Cache Type | cache_status |
simple, semantic |
❔ Optional |
| Force Cache Refresh | cache_force_refresh |
True, False (Boolean) |
❔ Optional |
| Cache Age | cache_age |
integer (in seconds) |
❔ Optional |
| Trace ID | trace_id |
string |
❔ Optional |
| Retries | retry |
integer [0,5] |
❔ Optional |
| Metadata | metadata |
json object More info |
❔ Optional |
🤝 Supported Providers
| Provider | Support Status | Supported Endpoints | |
|---|---|---|---|
 |
OpenAI | ✅ Supported | /completion, /embed |
 |
Azure OpenAI | ✅ Supported | /completion, /embed |
 |
Anthropic | ✅ Supported | /complete |
 |
Cohere | 🚧 Coming Soon | generate, embed |
📝 Full Documentation | 🛠️ Integration Requests |
1 maintainer
Release files for portkey-ai 0.1.53
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| portkey-ai-0.1.53.tar.gz | 19.2 kB | Details |
Release files / portkey-ai-0.1.53.tar.gz
| Download URL | portkey-ai-0.1.53.tar.gz |
|---|---|
| Size | 19.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
aa771b3fd30d6440b683a930ccff1be0ab8d51c70aec35b1dd4e923f57ceacaf
|
|
BLAKE2b-256 checksum How to use checksums |
45ef4d18bdbd5809f6582dd41a1514501e4998ff6f888488d30db96a35a9b9a6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.18
|
