dimensional monitoring cli
Project description
dimon: Your Canonical Answer Management Tool
dimon is a command-line interface (CLI) tool for evaluating and managing golden dataset for evaluation your retrival mechnism.
Installation
You can install dimon directly from PyPI:
pip install dimon
Prepare env file
Create a file named .env
, or copy it from .env.example
.
This file should contain the default configuration values for dimon. Here is an example of what the .env
file might look like:
DATABASE_URL=http://localhost:9200
DATABASE_INDEX=your_index
USERNAME=elastic
PASSWORD=your_password_here
DATASET_PATH=/path/to/your/dataset
Replace the values with the actual ones.
Note: Currently, the script only supports ElasticSearch as the vector database. Ensure that your ElasticSearch instance is correctly set up and accessible via the URL provided in the DATABASE_URL parameter. The DATABASE_INDEX should point to the specific index within ElasticSearch that you want to use for storing and querying vectors.
The CLI tool provides three main commands:
evaluate
This command evaluates a specified model on a given dataset. It computes embeddings for each document in the corpus, retrieves the top-k most similar documents for each query, and calculates the Mean Reciprocal Rank (MRR) for evaluation.
dimon evaluate --database_url 'http://localhost:9200' --dataset 'golden_dataset.csv'
diff
This command reads a report file, filters out entries with MRR below a certain threshold
dimon diff --file_name 'mrr_report.csv' --threshold 0.2
merge
This command reads a report file, filters out entries marked as 'Accept', and simulates a merging process.
dimon merge --report_name 'to_confirmed_change.csv'
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dimon_cli-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fc4dc6ff892c1f849737b5ddb900e1cca91636174651b091ed35f2ddd81ab904 |
|
MD5 | b1c28044f71a2b0378b8d90e1937fd70 |
|
BLAKE2b-256 | c7d7426da609e0c1724ab439ed4d2f2dc869a02eef290dc39b180d43bb1dd987 |