RAGA AI CATALYST
Project description
RagaAI Catalyst
RagaAI Catalyst is a powerful tool for managing and optimizing LLM projects. It provides functionalities for project management, trace recording, and experiment management, allowing you to fine-tune and evaluate your LLM applications effectively.
Table of Contents
Installation
To install RagaAI Catalyst, you can use pip:
pip install ragaai-catalyst
Configuration
Before using RagaAI Catalyst, you need to set up your credentials. You can do this by setting environment variables or passing them directly to the RagaAICatalyst
class:
from ragaai_catalyst import RagaAICatalyst
catalyst = RagaAICatalyst(
access_key="YOUR_ACCESS_KEY",
secret_key="YOUR_SECRET_KEY",
api_keys={"OPENAI_API_KEY": "YOUR_OPENAI_API_KEY"}
)
Usage
Project Management
Create and manage projects using RagaAI Catalyst:
# Create a project
project = catalyst.create_project(
project_name="Test-RAG-App-1",
description="Description of the project"
)
# List projects
projects = catalyst.list_projects()
print(projects)
Trace Management
Record and analyze traces of your RAG application:
from ragaai_catalyst import Tracer
# Start a trace recording
tracer = Tracer(
project_name="Test-RAG-App-1",
metadata={"key1": "value1", "key2": "value2"},
tracer_type="langchain",
pipeline={
"llm_model": "gpt-3.5-turbo",
"vector_store": "faiss",
"embed_model": "text-embedding-ada-002",
}
).start_trace()
# Your code here
# Stop the trace recording
tracer.stop_trace()
# Alternatively, use a context manager
with tracer.trace():
# Your code here
Experiment Management
Create and manage experiments to evaluate your RAG application:
from ragaai_catalyst import Experiment
# Create an experiment
experiment_manager = Experiment(
project_name="Test-RAG-App-1",
experiment_name="Exp-01",
experiment_description="Experiment Description",
dataset_name="Dataset Created from UI",
)
# Add metrics to the experiment
experiment_manager.add_metrics(
metrics=[
{"name": "hallucination", "config": {"model": "gpt-4o", "provider":"OpenAI"}}
]
)
# Add multiple metrics
experiment_manager.add_metrics(
metrics=[
{"name": "hallucination", "config": {"model": "gpt-4o", "provider":"OpenAI"}},
{"name": "hallucination", "config": {"model": "gpt-4", "provider":"OpenAI"}},
{"name": "hallucination", "config": {"model": "gpt-3.5-turbo", "provider":"OpenAI"}}
]
)
# Get the status of the experiment
status = experiment_manager.get_status()
print("Experiment Status:", status)
# Get the results of the experiment
results = experiment_manager.get_results()
print("Experiment Results:", results)
Dataset Management
Create and manage trace datasets for your projects.
from ragaai_catalyst import Dataset
# Initialize Dataset management for a specific project
dataset_manager = Dataset(project_name="Test-RAG-App-1")
# List existing datasets
datasets = dataset_manager.list_datasets()
print("Exisiting Datasets:", datasets)
# Create a new dataset with filters
dataset_manager.create_dataset(
dataset_name='Test-dataset-1',
filter_list=[
{
"name": "llm_model",
"values": ["gpt-3.5-turbo", "gpt-4"]
},
{
"name": "prompt_length",
"lte": 27,
"gte": 23
}
]
)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ragaai_catalyst-1.0.6.1b3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 469702d7db615a926b6a73297fd8aace929611c831248d4a9de51e4954936d9d |
|
MD5 | 4d994ee6483fc5b66ca6d4f8aee29ddb |
|
BLAKE2b-256 | 277235b3184d6a9c305b09c30c6dbdd27ece870d38cd0521795682280a06312d |
Hashes for ragaai_catalyst-1.0.6.1b3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 009132889650730f247d078ab70423d4cbb0aea8e83b5fe6f5318a0cf540ed5e |
|
MD5 | a4407d5e0653819bc3cfb49c3b49340d |
|
BLAKE2b-256 | 45a9ec42daaeb1a609950fcbff5290ed1965d1f51125bcf6d9b0b772d307e298 |