eLLMental SDK

eLLMental is a Python library for interacting with a custom evaluation API. This library allows you to set an evaluation context, execute evaluations, push test results, and manage search pipelines and test cases.

Installation

Since eLLMental is a private repository, you need to install it from a local path. First clone the repo and then you can install the library using pip from the local path:

pip install /path/to/your/eLLMental
Replace /path/to/your/eLLMental with the actual path to the directory containing setup.py.

Usage

Below is a step-by-step guide on how to use the eLLMental library.

Initialization

First, import the library and create an instance of the eLLMental class with the base URL of your API:

from Ellmental import Ellmental

base_url = "http://your-api-url.com"
authorization_token= "your bearer toke generated by eLLMental"
eLLMental = Ellmental(base_url, authorization_token)

Search pipeline

Listing Search Pipelines

List all available search pipelines:

pipelines = eLLMental.list_search_pipelines()
print(pipelines)

Searching with a Pipeline

Perform a search using a specific pipeline and context:

pipeline_id = "your_pipeline_id"
context_id = "your_context_id"
query = "your_query"
results = pipelines = eLLMental.list_search_pipelines()
.search_pipeline_query(pipeline_id, context_id, query)
print(results)

Evaluations

Setting the Evaluation Context

Set the evaluation context by specifying the test suite ID:

test_suite_id = "your_test_suite_id"
eLLMental.set_evaluation_context(test_suite_id)

Listing Evaluation Cases

List all test cases in the evaluation context:

test_cases = eLLMental.list_evaluation_cases()
print(test_cases)

Starting an Evaluation Execution

Start an evaluation execution:

eLLMental.start_evaluation_execution()

Pushing Evaluation Results

Push the results of a test case:

test_case_id = "your_test_case_id"
test_result = {"some": "result"}
status = "Pass"  # or "Failure"
metadata = {"additional": "info"}
response = eLLMental.push_evaluation_result(test_case_id, test_result, status, metadata)
print(response)

Finishing the Evaluation Execution

Finish the evaluation execution:

response = eLLMental.finish_evaluation_execution()
print(response)

Complete evaluation example:

from Ellmental import Ellmental
import json
import time

def main():
    eLLMental = eLLMental("http://localhost:4000", "API KEY TOKEN") # init the SDK with your instance URL
    eLLMental.set_evaluation_context(2) # init the evaluation context with your evaluation id
    test_cases = eLLMental.list_evaluation_cases() # fetch all your evaluation cases
    eLLMental.start_evaluation_execution() # start a new evaluation execution
    for test_case in test_cases:

        test_input = json.loads(test_case["test_query"])
        expected_result = int(test_case["expected_results"])
        test_result = function_under_test(test_input[0], test_input[1])
        status = "Pass" if expected_result == test_result else "Fail"

        eLLMental.push_evaluation_result(test_case["id"], test_result, status) # notify the evaluation result

    eLLMental.finish_evaluation_execution() # finish the current evaluation execution

def function_under_test(a, b):
    time.sleep(2)
    return a + b

if __name__ == "__main__":
    main()

Analytics

🚧 TO DO 🚧

Data

🚧 TO DO 🚧

Development

If you are contributing to the SDK after perfom your changes in order to create a new pip package to be installed / released you have to run the following command:

python setup.py sdist bdist_wheel