Python package to compute NLU metrics
Project description
Python package to compute metrics on a NLU/ASR parsing pipeline
Install
pip install snips_nlu_metrics
Pure NLU Metrics API
from snips_nlu_metrics import (
compute_train_test_nlu_metrics, compute_cross_val_nlu_metrics)
from snips_nlu import SnipsNLUEngine as NLUTrainingEngine
from snips_nlu_rust import NLUEngine as NLUInferenceEngine
tt_metrics = compute_train_test_nlu_metrics(train_dataset="path/to/train_dataset.json",
test_dataset="path/to/test_dataset.json",
training_engine_class=NLUTrainingEngine,
inference_engine_class=NLUInferenceEngine)
cv_metrics = compute_cross_val_nlu_metrics(dataset="path/to/dataset.json",
training_engine_class=NLUTrainingEngine,
inference_engine_class=NLUInferenceEngine,
nb_folds=5)
End-to-End Metrics API
The metrics API lets you compute metrics on a full end-to-end ASR + NLU pipeline. To do that, you will need to implement an engine class that inherits or satisfy the API of the following Engine abstract class:
from abc import ABCMeta, abstractmethod
class Engine(object):
"""
Abstract class which represents an engine that can be used in the metrics
API. All engine classes must inherit from `Engine` or satisfy its API.
"""
__metaclass__ = ABCMeta
@abstractmethod
def fit(self, dataset):
pass
@abstractmethod
def parse(self, text):
pass
Here is how you can use the end-to-end metrics API, if you have a EndToEndEngine that inherits from Engine:
from snips_nlu_metrics import compute_train_test_metrics, compute_cross_val_metrics
tt_metrics = compute_train_test_metrics(train_dataset="path/to/train_dataset.json",
test_dataset="path/to/test_dataset.json",
engine_class=EndToEndEngine)
cv_metrics = compute_cross_val_metrics(dataset="path/to/dataset.json",
engine_class=EndToEndEngine,
nb_folds=5)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for snips_nlu_metrics-0.11.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 60b9e5fcecf317806cfe153a69f02c2074ef834676e1a09d15f642e52151922b |
|
MD5 | 109ba182406b37d1e20ff318707c1b66 |
|
BLAKE2b-256 | 323208b263cd892f33e5a43e59f8f1e9241c7508fe3fb14c1a2e8b92acab6899 |