uptrain 0.3.7

UpTrain - ML Observability and Retraining Framework

An open-source framework to evaluate and monitor LLM applications

Try out Evaluations - UpTrain in Action - Slack Community - Feature Request

UpTrain is a Python framework that ensures your LLM applications are performing reliably by allowing users to check aspects such as correctness, structural integrity, bias, hallucination, etc. UpTrain can be used to:

Experimentation

UpTrain framework can be used to experiment across multiple prompts, model providers, chain configurations, etc. and get quantitative scores to compare them. Check out the experimentation tutorial to learn more.

Validation

You can use the UpTrain Validation Manager to define checks, retry logic and validate your LLM responses before showing it to your users. Check out the tutorial here.

Monitoring

You can use the UpTrain framework to continuously monitor your model's performance and get real-time insights on how well it is doing on a variety of evaluation metrics. Check out the monitoring tutorial to learn more.

Get started 🙌

To run it on your machine, checkout the Quickstart tutorial:

Install the package through pip:

pip install uptrain

Note: Uptrain uses commonly used python libraries like openai-evals and sentence-transformers. To make sure, all the functionalities work, use the uptrain-add command to install the full version of the package.

uptrain-add --feature full

How to use UpTrain:

Using UpTrain's builtin evaluation sets:

UpTrain provides a variety of checks like response relevance, response completeness, factual accuracy, retrieved-context quality, etc. which can be accessed using UpTrain's API key. Learn more about these evaluations.

Get your free UpTrain API Key here.

data = pl.DataFrame({
  "question": ["What is the meaning of life?"],
  "response": ["Who knows 🤔"]
})

check = CheckResponseCompleteness()
output = check.setup(Settings(uptrain_access_token="up-9g....")).run(data)

Configuring your own evaluation sets:

Say we want to plot a line chart showing whether our model's responses contain any grammatical mistakes or not.

# Step 1: Choose and create the appropriate operator from UpTrain
grammar_score = GrammarScore(
  col_in_text = "model_response",       # input column name (from dataset)
  col_out = "grammar_score"             # desired output column name
)

# Step 2: Create a check with the operators and the required plots as arguments 
grammar_check = Check(
  operators = [grammar_score],
  plots = LineChart(y = "grammar_score")
)

# Step 3: Create a CheckSet with the checks and data source as arguments
checkset = CheckSet(
    checks = [grammar_check]
    source = JsonReader(fpath = '...')
)

# Step 4: Set up and run the CheckSet
checkset.setup(Settings(openai_api_key = '...'))
checkset.run(dataset)

Key Features 💡

• ChatGPT Grading - Utilize LLMs to grade your model outputs.
• Custom Grading Checks - Write your custom grading prompts.
• Embeddings Similarity Check - Compute cosine similarity between prompt and response embeddings
• Output Validation - Safeguard your users against inappropriate responses
• Prompt A/B Testing - Experiment across multiple prompts and compare them quantatively.
• UMAP Visualization and Clustering - Visualize your embedding space using tools like UMAP and t-SNE.
• Hallucination Checks - Use metrics like custom grading, text similarity, and embedding similarity to check for hallucinations.
• Toxic Keywords Checks - Make sure your model outputs are not biased or contain toxic keywords.
• Feature Slicing - Built-in pivoting functionalities for data dice and slice to pinpoint low-performing cohorts.
• Realtime Dashboards - Monitor your model's performance in realtime.

Integrations

Eval Frameworks	LLM Providers	LLM Packages	Serving frameworks
OpenAI Evals ✅	GPT-3.5-turbo ✅	Langchain 🔜	HuggingFace 🔜
EleutherAI LM Eval 🔜	GPT-4 ✅	Llama Index 🔜	Replicate 🔜
BIG-Bench 🔜	Claude 🔜	AutoGPT 🔜
	Cohere 🔜

Why UpTrain 🤔?

Large language models are trained over billions of data points and perform really well over a wide variety of tasks. But one thing these models are not good at is being deterministic. Even with the most well-crafted prompts, the model can misbehave for certain inputs, be it hallucinations, wrong output structure, toxic or biased response, irrelevant response, and error modes can be immense.

To ensure your LLM applications work reliably and correctly, UpTrain makes it easy for developers to evaluate the responses of their applications on multiple criteria. UpTrain's evaluation framework can be used to:

1. Validate (and correct) the response of the model before showing it to the user
2. Get quantitative measures to experiment across multiple prompts, model providers, etc.
3. Do unit testing to ensure no buggy prompt or code gets pushed into your production
4. Monitor your LLM applications in real time and understand when they are going wrong in order to fix them before users complain.

We are constantly working to make UpTrain better. Want a new feature or need any integrations? Feel free to create an issue or contribute directly to the repository.

License 💻

This repo is published under Apache 2.0 license. We are also working towards adding a hosted offering to make setting off eval runs easier - please fill this form to get a waitlist slot.

Stay Updated ☎️

We are continuously adding tons of features and use cases. Please support us by giving the project a star ⭐!

Provide feedback (Harsher the better 😉)

We are building UpTrain in public. Help us improve by giving your feedback here.

Contributors 🖥️

We welcome contributions to UpTrain. Please see our contribution guide for details.