aimon 0.1.0

The AIMon SDK that is used to interact with the AIMon API and the product.

🎉Welcome to AIMon Rely

AIMon Rely is a state-of-the-art system consisting of multiple models for detecting LLM quality issues during offline evaluations and continuous production monitoring. We offer hallucination metrics that is fast, reliable and cost-effective. We also support additional metrics such as completeness, conciseness and toxicity.

Read our blog post for more details.

✨ Join our community on Slack or reach out to us at [e-mail] to get your API key.

Metrics Supported

These are the quality metrics that are currently available via the API. Some of them are in progress and will be available in a future release.

Metric	Status
Model Hallucination (Passage and Sentence Level)	✓
Completeness	✓
Conciseness	✓
Toxicity	✓
Semantic Similarity	⌛
Sentiment	⌛
Coherence	⌛
Sensitive Data (PII/PHI/PCI)	⌛

Product

Follow these steps to use the product:

• Step 1: Get access to the beta product by joining the wait list on our website or by requesting it on Slack or sending an email to [e-mail]
• Step 2: Install the AIMon SDK by running pip install aimon in your terminal.
• Step 3: Refer to the sample notebook for an example of how to instrument an LLM application using our SDK.

API

Steps to use the API:

• Step 1: Get your API key by requesting it on our Slack or sending an email to [e-mail]
• Step 2: You can try the API using either of these methods
  • [OPTION 1] Try the simple langchain summarization application that is augmented with AIMon Rely to detect hallucinations at the sentence level.
    • Step 1: Run pip install -r examples/requirements.txt && pip install aimon
    • Step 2: Run streamlit run examples/langchain_summarization_app.py
  • [OPTION 2] Download the Postman collection specified below to access the API
    • Model Hallucination (Passage and Sentence Level): Postman Collection

Sandbox

You can play with a Sandbox that is available on our website.

Benchmarks

Hallucination Detection

To demonstrate the effectiveness of our system, we benchmarked it against popular industry benchmarks for the hallucination detection task. The table below shows our results.

A few key takeaways:

✅ AIMon Rely is 10x cheaper than GPT-4 Turbo.

✅ AIMon Rely is 4x faster than GPT-4 Turbo.

✅ AIMon Rely provides the convenience of a fully hosted API that includes baked-in explainability.

✅ Support for a context length of up to 32,000 tokens (with plans to further expand this in the near future).

Overall, AIMon Rely is 10 times cheaper, 4 times faster and close to or even better than GPT-4 on the benchmarks making it a suitable choice for both offline and online detection of hallucinations.

Completeness, Conciseness Detection

There is a lack of industry standard benchmark datasets for these metrics. We will be publishing an evaluation dataset soon. Stay Tuned! ⌛

Pricing

Please reach out to [e-mail] for pricing details related to the product and the API.

Future Work

• We are working on additional metrics as detailed in the table above.
• In addition, we are working on something awesome to make the offline evaluation and continuous model quality monitoring experience more seamless.

Join our Slack for the latest updates and discussions on generative AI reliability.