RAGA AI CATALYST
Project description
RagaAI Catalyst
RagaAI Catalyst is a powerful tool for managing and optimizing LLM projects. It provides functionalities for project management, trace recording, and experiment management, allowing you to fine-tune and evaluate your LLM applications effectively.
Table of Contents
Installation
To install RagaAI Catalyst, you can use pip:
pip install ragaai-catalyst
Configuration
Before using RagaAI Catalyst, you need to set up your credentials. You can do this by setting environment variables or passing them directly to the RagaAICatalyst
class:
from ragaai_catalyst import RagaAICatalyst
catalyst = RagaAICatalyst(
access_key="YOUR_ACCESS_KEY",
secret_key="YOUR_SECRET_KEY",
api_keys={"OPENAI_API_KEY": "YOUR_OPENAI_API_KEY"}
)
Usage
Project Management
Create and manage projects using RagaAI Catalyst:
# Create a project
project = catalyst.create_project(
project_name="Test-RAG-App-1",
description="Description of the project"
)
# List projects
projects = catalyst.list_projects()
print(projects)
Trace Management
Record and analyze traces of your RAG application:
from ragaai_catalyst import Tracer
# Start a trace recording
tracer = Tracer(
project_name="Test-RAG-App-1",
metadata={"key1": "value1", "key2": "value2"},
tracer_type="langchain",
pipeline={
"llm_model": "gpt-3.5-turbo",
"vector_store": "faiss",
"embed_model": "text-embedding-ada-002",
}
).start_trace()
# Your code here
# Stop the trace recording
tracer.stop_trace()
# Alternatively, use a context manager
with tracer.trace():
# Your code here
Experiment Management
Create and manage experiments to evaluate your RAG application:
from ragaai_catalyst import Experiment
# Create an experiment
experiment_manager = Experiment(
project_name="Test-RAG-App-1",
experiment_name="Exp-01",
experiment_description="Experiment Description",
dataset_name="Dataset Created from UI",
)
# Add metrics to the experiment
experiment_manager.add_metrics(
metrics=[
{"name": "hallucination", "config": {"model": "gpt-4o", "provider":"OpenAI"}}
]
)
# Add multiple metrics
experiment_manager.add_metrics(
metrics=[
{"name": "hallucination", "config": {"model": "gpt-4o", "provider":"OpenAI"}},
{"name": "hallucination", "config": {"model": "gpt-4", "provider":"OpenAI"}},
{"name": "hallucination", "config": {"model": "gpt-3.5-turbo", "provider":"OpenAI"}}
]
)
# Get the status of the experiment
status = experiment_manager.get_status()
print("Experiment Status:", status)
# Get the results of the experiment
results = experiment_manager.get_results()
print("Experiment Results:", results)
Dataset Management
Create and manage trace datasets for your projects.
from ragaai_catalyst import Dataset
# Initialize Dataset management for a specific project
dataset_manager = Dataset(project_name="Test-RAG-App-1")
# List existing datasets
datasets = dataset_manager.list_datasets()
print("Exisiting Datasets:", datasets)
# Create a new dataset with filters
dataset_manager.create_dataset(
dataset_name='Test-dataset-1',
filter_list=[
{
"name": "llm_model",
"values": ["gpt-3.5-turbo", "gpt-4"]
},
{
"name": "prompt_length",
"lte": 27,
"gte": 23
}
]
)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ragaai_catalyst-1.0.6.1b5.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3e6451b8754e9d02c7516aafeae4b1ef904119d074656a885c6c488b34b6def4 |
|
MD5 | 9c96bd5909b634fbbcc9e20ddcdd5e4f |
|
BLAKE2b-256 | 71611389f9b966bf85bb4bb339c0a174ef21694c7a5ebba976f1f7a3a218a069 |
Hashes for ragaai_catalyst-1.0.6.1b5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6d2cb8e6dff2300ee9b3ff4c434899fe81d94727fdf6bba3b4a25de254b65048 |
|
MD5 | e8d3ed331c4274ca3556177d9bc6cd49 |
|
BLAKE2b-256 | 046ae2d2cbdf4bcf3ade4537e6182ba54368d634760edc8d3f13988e85883f0e |