NotDiamond Python library for easy OpenAI integration.
Project description
Getting started with Not Diamond
Not Diamond helps you leverage multiple LLMs dynamically, sending queries to the best-suited model in real-time.
Key features
- Not a proxy: When we receive a query, our backend service returns a recommendation of the model you should call and the actual call goes out client-side. So we never see your response outputs.
- Privacy preserving: To protect your privacy, we use fuzzy hashing to conceal your query strings from us, and our model recommendation engine works entirely using the fuzzy hashed queries. This means we also never see the raw query strings you send to an LLM.
- Maximize performance: By dynamically routing to the best-suited LLM for each query, we improve your overall LLM output quality and enhance product quality. Our recommendations are based on millions of data points from rigorous evaluation benchmarks and real-world data.
- Reduce cost and latency: Most of the time, a small and specialized model can optimally handle a given query. We can help you determine this in real-time, cutting your LLM inference costs while reducing latency.
👍 Free to use!
Not Diamond is free up to 100K monthly requests. Beyond this pay just $10 per 10K requests.
Installation
Requires Python 3.9+
pip install notdiamond
API keys
Sign up and get a Not Diamond API key.
Create a .env
file with your Not Diamond API key, and the API keys of the models you want to route between.
OPENAI_API_KEY="YOUR_OPENAI_API_KEY"
GOOGLE_API_KEY="YOUR_GOOGLE_API_KEY"
ANTHROPIC_API_KEY="YOUR_ANTHROPIC_API_KEY"
NOTDIAMOND_API_KEY="YOUR_NOTDIAMOND_API_KEY"
Alternatively, you can also set API keys programmatically as described further below.
📘 API keys
The
notdiamond
library uses your API keys client-side to call the LLM we recommend. We never pass your keys to our servers.
🚧 Model access
Since
notdiamond
calls the LLMs client-side using your keys, we will only call models you have access to. You can also use our router to determine the best model to call regardless of whether you have access or not (see example). Our router supports most of the popular open and proprietary models (see full list).Drop me a line if you have a specific model requirement and we're happy to work with you to support it.
Example
If you already have existing projects in either OpenAI SDK or LangChain, check out our OpenAI and Langchain integration guides. Otherwise, continue reading.
Create a main.py
file in the same folder as the .env
file you created earlier, or try it in Colab.
from notdiamond.llms.llm import NDLLM
from notdiamond.prompts.prompt import NDPrompt, NDContext, NDQuery, NDPromptTemplate
# Define your prompt and query
prompt = NDPrompt("You are a world class software developer.") # The system prompt, defines the LLM's role
query = NDQuery("Write a merge sort in Python.") # The specific query written by an end-user
# Define the prompt template to combine prompt and query into a single string
prompt_template = NDPromptTemplate("About you: {prompt}\n{query}",
partial_variables={"prompt": prompt, "query": query})
# Define the available LLMs you'd like to route between
llm_providers = ['openai/gpt-3.5-turbo', 'openai/gpt-4','openai/gpt-4-1106-preview', 'openai/gpt-4-turbo-preview',
'anthropic/claude-2.1', 'anthropic/claude-3-sonnet-20240229', 'anthropic/claude-3-opus-20240229',
'google/gemini-pro']
# Create the NDLLM object -> like a 'meta-LLM' combining all of the specified models
nd_llm = NDLLM(llm_providers=llm_providers)
# After fuzzy hashing the inputs, the best LLM is determined by the ND API and the LLM is called client-side
result, session_id, provider = nd_llm.invoke(prompt_template=prompt_template)
print("ND session ID: ", session_id) # A unique ID of the invoke. Important for future references back to ND API
print("LLM called: ", provider.model) # The LLM routed to
print("LLM output: ", result.content) # The LLM response
👍 Run it!
python main.py
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for notdiamond_openai-0.1.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | be21395fe0fdb0220e9812bbcd9bd631bd397d4b6f45ced6a48861c7055753fa |
|
MD5 | faadb2c6ec1e988833cd2a39995aa228 |
|
BLAKE2b-256 | ae5ac8418409166608d6360c9869bf4d3c677c555d666c2e901af681e4561880 |