evalyn-sdk-harishatmecs 0.0.6

Python SDK to send evaluation data to an API with support for batch processing

EvalynSDK - EvaluatorSDK

EvaluatorSDK is a lightweight Python SDK that helps developers collect and send evaluation data to a centralized API endpoint for analysis. It supports both individual and batch submissions from datasets (Excel or JSON).

🚀 Features

• Read config from environment variables
• Automatically generate and track request IDs
• Add and manage parameters per request
• Send evaluation data as POST requests to API
• Support for batch processing from .xlsx or .json datasets
• Exception handling and validation

📦 Installation

Make sure pip is installed on your system. Then, you can install the SDK using pip:

To install the SDK, simply run:

pip install evalyn-sdk-harishatmecs==0.0.6

Environment Setup

Create a .env file in your project root and add the following:

Currently it works in ATMECS network only, please contact your architect (Harish Nankani) for access. This IP will change when it is deployed in ATMECS servers.

#EVAL_API_URL=http://172.29.208.1:8001/api/proj_eval
EVAL_API_URL=http://10.10.10.154:8002/api/proj_eval
EVAL_API_KEY=your_project_code_here

DOCS_PATH=your_docs_path_here_optional
VECTORDB_PATH=your_vectordb_path_here_optional
EVAL_DATASET_FILE=your_excel_file_path_here_optional

The SDK uses python-dotenv to load these values automatically.

Usage

To use the SDK, import it in your Python script and create an instance of the EvaluatorSDK class. Then, you can call its methods to interact with the API.

from evalyn_sdk_harishatmecs.sdk.evaluator import EvaluatorSDK
import traceback

# Initialize SDK
eval_obj = EvaluatorSDK()

# Start a new request (creates unique request_id internally)
eval_obj.start_request()

# Add all relevant parameters
eval_obj.add_project_code("demo")
eval_obj.add_prompt(request.query)
eval_obj.add_no_of_docs_in_context(no_of_docs_in_faiss)
eval_obj.add_top_k(top_k)
eval_obj.add_no_of_docs_retrieved(no_of_docs_retrieved)
eval_obj.add_retrieved_ans(retrieved_ans)
eval_obj.add_final_ans(retrieved_ans)
eval_obj.add_ground_truth(ground_truth)
eval_obj.add_retrieval_time(retrieval_time)
eval_obj.add_generation_time(generation_time)
eval_obj.add_total_query_time(total_query_time)
eval_obj.add_throughput(throughput)
eval_obj.add_retrieved_docs_list(retrieved_docs_list)

# Send the collected data
try:
    # Recommended: send in background thread
    eval_obj.send_in_background()
except Exception as ex:
    print(f"Error sending evaluation data: {ex}")
    traceback.print_exc()

Coming Soon

• Retry logic on failed sends
• Improved error handling and logging
• Support for different types of datasets (CSV, JSON, etc.)
• Built-in validation and schema support
• Enhanced security features