LiteLLM Router Integration for Kamiwaza AI
This package provides a custom router for LiteLLM that integrates with Kamiwaza AI model deployments. The KamiwazaRouter extends LiteLLM's Router class to enable efficient routing of requests to Kamiwaza-deployed models.
Features
- Dynamic Model Discovery: Automatically discovers available models from Kamiwaza deployments
- Multi-Instance Support: Connect to multiple Kamiwaza instances simultaneously
- Caching: Efficient caching of model lists with configurable TTL
- Model Pattern Filtering: Filter models based on name patterns (e.g., only use "qwen" or "gemma" models)
- Static Model Configuration: Support for static model configurations alongside Kamiwaza models
- Fallback Routing: Automatic fallback between models in case of failures
Installation
pip install litellm-kamiwaza
For running the examples, you'll also need:
pip install python-dotenv
Requirements
- Python 3.7+
- litellm>=1.0.0
- kamiwaza-client>=0.1.0
Usage
Basic Usage
from litellm_kamiwaza import KamiwazaRouter
# Initialize router with automatic Kamiwaza discovery
router = KamiwazaRouter()
# Use the router like a standard litellm Router
response = router.completion(
model="deployed-model-name",
messages=[{"role": "user", "content": "Hello, world!"}]
)
Configuration Options
Environment Variables
KAMIWAZA_API_URL: URL for the Kamiwaza API (e.g., "https://localhost/api")KAMIWAZA_URL_LIST: Comma-separated list of Kamiwaza URLs (e.g., "https://instance1/api,https://instance2/api")KAMIWAZA_VERIFY_SSL: Set to "true" to enable SSL verification (default: "false")
Router Configuration
# Initialize with specific Kamiwaza URL
router = KamiwazaRouter(
kamiwaza_api_url="https://my-kamiwaza-server.com/api",
cache_ttl_seconds=600, # Cache model list for 10 minutes
model_pattern="72b", # Only use models with "72b" in their name
)
# Initialize with multiple Kamiwaza instances
router = KamiwazaRouter(
kamiwaza_uri_list="https://instance1.com/api,https://instance2.com/api",
cache_ttl_seconds=300
)
# Initialize with static model list alongside Kamiwaza models
router = KamiwazaRouter(
kamiwaza_api_url="https://my-kamiwaza-server.com/api",
model_list=[
{
"model_name": "my-static-model",
"litellm_params": {
"model": "openai/gpt-4",
"api_key": "sk-your-api-key",
"api_base": "https://api.openai.com/v1"
},
"model_info": {
"id": "my-static-model",
"provider": "static",
"description": "Static model configuration"
}
}
]
)
Pattern Matching Examples
You can filter models by name patterns:
# Only use models with "qwen" in their name
router = KamiwazaRouter(
kamiwaza_api_url="https://my-kamiwaza-server.com/api",
model_pattern="qwen"
)
# Only use gemma models
router = KamiwazaRouter(
kamiwaza_uri_list="https://instance1.com/api,https://instance2.com/api",
model_pattern="gemma"
)
# Only use static models
router = KamiwazaRouter(
model_pattern="static"
)
Static Models Configuration
For more organized static model configurations, you can create a static_models_conf.py file in your project root:
# static_models_conf.py
from typing import List, Dict, Any, Optional
def get_static_model_configs() -> List[Dict[str, Any]]:
"""Returns a list of statically defined model configurations."""
return [
{
"model_name": "static-custom-model",
"litellm_params": {
"model": "openai/model",
"api_key": "your-api-key",
"api_base": "https://your-endpoint.com/v1"
},
"model_info": {
"id": "static-custom-model",
"provider": "static",
"description": "Static model configuration"
}
}
]
The KamiwazaRouter will automatically detect and use these static models.
Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
License
This project is licensed under the MIT License - see the LICENSE file for details.
Metadata
Release files for litellm-kamiwaza 0.1.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| litellm_kamiwaza-0.1.4.tar.gz | 21.6 kB | Without provenance | Details for litellm_kamiwaza-0.1.4.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| litellm_kamiwaza-0.1.4-py3-none-any.whl | Python 3 | none | any | Details for litellm_kamiwaza-0.1.4-py3-none-any.whl |
Total release size: 41.4 kB
Release files / litellm_kamiwaza-0.1.4.tar.gz
| Download URL | litellm_kamiwaza-0.1.4.tar.gz |
|---|---|
| Size | 21.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a00a7e836f7b6b91bf6af0c2190335e4d75fe994acb8d18c7bdc69a77634f827
|
|
BLAKE2b-256 checksum How to use checksums |
4e80fcf26d92694ab22536f72388e6733cb96f2c4cdd0510f57a4d669a1613a8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Release files / litellm_kamiwaza-0.1.4-py3-none-any.whl
| Download URL | litellm_kamiwaza-0.1.4-py3-none-any.whl |
|---|---|
| Size | 19.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1bbf83046ca1e2abbaf816efd01461d48432127af6dbab02dac3edd80da6e271
|
|
BLAKE2b-256 checksum How to use checksums |
4a2b34394ba24b532ff5c91b1dffb6d0c47a39e229d3977d0e1c3265b40606e2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Security for litellm-kamiwaza 0.1.4
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.