RAGA AI CATALYST
Project description
RagaAI Catalyst
RagaAI Catalyst is a powerful tool for managing and optimizing LLM projects. It provides functionalities for project management, trace recording, and experiment management, allowing you to fine-tune and evaluate your LLM applications effectively.
Table of Contents
Installation
To install RagaAI Catalyst, you can use pip:
pip install ragaai-catalyst
Configuration
Before using RagaAI Catalyst, you need to set up your credentials. You can do this by setting environment variables or passing them directly to the RagaAICatalyst
class:
from ragaai_catalyst import RagaAICatalyst
catalyst = RagaAICatalyst(
access_key="YOUR_ACCESS_KEY",
secret_key="YOUR_SECRET_KEY",
api_keys={"OPENAI_API_KEY": "YOUR_OPENAI_API_KEY"}
)
Usage
Project Management
Create and manage projects using RagaAI Catalyst:
# Create a project
project = catalyst.create_project(
project_name="Test-RAG-App-1",
description="Description of the project"
)
# List projects
projects = catalyst.list_projects()
print(projects)
Trace Management
Record and analyze traces of your RAG application:
from ragaai_catalyst import Tracer
# Start a trace recording
tracer = Tracer(
project_name="Test-RAG-App-1",
metadata={"key1": "value1", "key2": "value2"},
tracer_type="langchain",
pipeline={
"llm_model": "gpt-3.5-turbo",
"vector_store": "faiss",
"embed_model": "text-embedding-ada-002",
}
).start_trace()
# Your code here
# Stop the trace recording
tracer.stop_trace()
# Alternatively, use a context manager
with tracer.trace():
# Your code here
Experiment Management
Create and manage experiments to evaluate your RAG application:
from ragaai_catalyst import Experiment
# Create an experiment
experiment_manager = Experiment(
project_name="Test-RAG-App-1",
experiment_name="Exp-01",
experiment_description="Experiment Description",
dataset_name="Dataset Created from UI",
)
# Add metrics to the experiment
experiment_manager.add_metrics(
metrics=[
{"name": "hallucination", "config": {"model": "gpt-4o", "provider":"OpenAI"}}
]
)
# Add multiple metrics
experiment_manager.add_metrics(
metrics=[
{"name": "hallucination", "config": {"model": "gpt-4o", "provider":"OpenAI"}},
{"name": "hallucination", "config": {"model": "gpt-4", "provider":"OpenAI"}},
{"name": "hallucination", "config": {"model": "gpt-3.5-turbo", "provider":"OpenAI"}}
]
)
# Get the status of the experiment
status = experiment_manager.get_status()
print("Experiment Status:", status)
# Get the results of the experiment
results = experiment_manager.get_results()
print("Experiment Results:", results)
Dataset Management
Create and manage trace datasets for your projects.
from ragaai_catalyst import Dataset
# Initialize Dataset management for a specific project
dataset_manager = Dataset(project_name="Test-RAG-App-1")
# List existing datasets
datasets = dataset_manager.list_datasets()
print("Exisiting Datasets:", datasets)
# Create a new dataset with filters
dataset_manager.create_dataset(
dataset_name='Test-dataset-1',
filter_list=[
{
"name": "llm_model",
"values": ["gpt-3.5-turbo", "gpt-4"]
},
{
"name": "prompt_length",
"lte": 27,
"gte": 23
}
]
)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ragaai_catalyst-1.0.6.1b7.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8250b4368c6a1a85001a2a8dc8a17500ad3ea427af3b9fbed3c74dbff3450112 |
|
MD5 | 04320b740b0228ad4f445685a458886a |
|
BLAKE2b-256 | 70ddbdbfc2ea59a2891599c22bac1230946007f8a9bfe820908fb2bf8e787592 |
Hashes for ragaai_catalyst-1.0.6.1b7-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 199adb81b3632463a87b8e75fc5f4e9ecd0b4850aef563f77e0d03aef17d27e9 |
|
MD5 | 94182719a70e623baed80e01df2c1a11 |
|
BLAKE2b-256 | d0d087df93d8b81e6781ed45790e71871a2b591858c334c646540ed39f5602d7 |