Snips NLU Engine

This library can be used to benchmark NLU solutions such as Snips NLU. To install the snips-nlu python library, and fetch the language resources for english, run the following commands:

$ pip install snips-nlu
$ snips-nlu download en

Then, you can compute metrics for the snips-nlu pipeline using the metrics API as follows:

from snips_nlu import SnipsNLUEngine
from snips_nlu_metrics import compute_train_test_metrics, compute_cross_val_metrics

tt_metrics = compute_train_test_metrics(train_dataset="samples/train_dataset.json",
                                        test_dataset="samples/test_dataset.json",
                                        engine_class=SnipsNLUEngine)

cv_metrics = compute_cross_val_metrics(dataset="samples/cross_val_dataset.json",
                                       engine_class=SnipsNLUEngine,
                                       nb_folds=5)

Custom NLU Engine

You can also compute metrics on a custom NLU engine, here is a simple example:

import random

from snips_nlu_metrics import compute_train_test_metrics

class MyNLUEngine:
    def fit(self, dataset):
        self.intent_list = list(dataset["intents"])
        return self

    def parse(self, text):
        return {
            "input": text,
            "intent": {
                "intentName": random.choice(self.intent_list),
                "probability": 0.5
            },
            "slots": []
        }

compute_train_test_metrics(train_dataset="samples/train_dataset.json",
                           test_dataset="samples/test_dataset.json",
                           engine_class=MyNLUEngine)