uptrain 0.3.3

UpTrain - ML Observability and Retraining Framework

An open-source framework to evaluate, test and monitor LLM applications

Docs - Slack Community - Bug Report - Feature Request

UpTrain is a Python framework that ensures your LLM applications are performing reliably by allowing users to check aspects such as correctness, structural integrity, bias, hallucination, etc. UpTrain can be used to:

1. Validate model's response and safeguard your users against hallucinations, bias, incorrect output formats, etc.
2. Experiment across multiple model providers, prompt templates, and quantify model's performance.
3. Monitor your model's performance in production and protect yourself against unwanted drifts

Key Features 💡

• ChatGPT Grading - Utilize LLMs to grade your model outputs.
• Custom Grading Checks - Write your custom grading prompts.
• Embeddings Similarity Check - Compute cosine similarity between prompt and response embeddings
• Output Validation - Safeguard your users against inappropriate responses
• Prompt A/B Testing - Experiment across multiple prompts and compare them quantatively.
• UMAP Visualization and Clustering - Visualize your embedding space using tools like UMAP and t-SNE.
• Hallucination Checks - Use metrics like custom grading, text similarity, and embedding similarity to check for hallucinations.
• Toxic Keywords Checks - Make sure your model outputs are not biased or contain toxic keywords.
• Feature Slicing - Built-in pivoting functionalities for data dice and slice to pinpoint low-performing cohorts.
• Realtime Dashboards - Monitor your model's performance in realtime.

Get started 🙌

To run it on your machine, checkout the Quickstart tutorial:

Install the package through pip:

pip install uptrain

Note: Uptrain uses commonly used python libraries like openai-evals and sentence-transformers. To make sure, all the functionalities work, use the uptrain-add command to install the full version of the package.

uptrain-add --feature full

How to define checks:

Say we want to check whether our model's responses contain any grammatical mistakes or not.

# Define your checkset - list of simple checks, dataset file, 
# and api_keys

checkset = CheckSet(
    checks = Check(
        name = "grammar_score",
        operators = [
            GrammarScore(
                col_in_text = "model_response",
                col_out = "grammar_score"
            ),
        ],
        plots = PlotlyChart.Table(title="Grammar scores"),
    ),
    source = JsonReader(fpath = '...')
)
settings = Settings(openai_api_key = '...')

checkset.setup(settings)
checkset.run()

Integrations

Eval Frameworks	LLM Providers	LLM Packages	Serving frameworks
OpenAI Evals ✅	GPT-3.5-turbo ✅	Langchain 🔜	HuggingFace 🔜
EleutherAI LM Eval 🔜	GPT-4 ✅	Llama Index 🔜	Replicate 🔜
BIG-Bench 🔜	Claude 🔜	AutoGPT 🔜
	Cohere 🔜

UpTrain in Action

Experimentation

You can use the UpTrain framework to run and compare LLM responses for different prompts, models, LLM chains, etc. Check out the experimentation tutorial to learn more.

Validation

You can use the UpTrain Validation Manager to define checks, retry logic and validate your LLM responses before showing it to your users. Check out the tutorial here.

Monitoring

You can use the UpTrain framework to continuously monitor your model's performance and get real-time insights on how well it is doing on a variety of evaluation metrics. Check out the monitoring tutorial to learn more.

Why UpTrain 🤔?

Large language models are trained over billions of data points and perform really well over a wide variety of tasks. But one thing these models are not good at is being deterministic. Even with the most well-crafted prompts, the model can misbehave for certain inputs, be it hallucinations, wrong output structure, toxic or biased response, irrelevant response, and error modes can be immense.

To ensure your LLM applications work reliably and correctly, UpTrain makes it easy for developers to evaluate the responses of their applications on multiple criteria. UpTrain's evaluation framework can be used to:

1. Validate (and correct) the response of the model before showing it to the user
2. Get quantitative measures to experiment across multiple prompts, model providers, etc.
3. Do unit testing to ensure no buggy prompt or code gets pushed into your production
4. Monitor your LLM applications in real time and understand when they are going wrong in order to fix them before users complain.

We are constantly working to make UpTrain better. Want a new feature or need any integrations? Feel free to create an issue or contribute directly to the repository.

License 💻

This repo is published under Apache 2.0 license. We are also working towards adding a hosted offering to make setting off eval runs easier - please fill this form to get a waitlist slot.

Stay Updated ☎️

We are continuously adding tons of features and use cases. Please support us by giving the project a star ⭐!

Provide feedback (Harsher the better 😉)

We are building UpTrain in public. Help us improve by giving your feedback here.

Contributors 🖥️

We welcome contributions to UpTrain. Please see our contribution guide for details.