deeppavlov 0.4.0

An open source library for building end-to-end dialog systems and training chatbots.

DeepPavlov is an open-source conversational AI library built on TensorFlow and Keras. It is designed for

• development of production ready chat-bots and complex conversational systems,
• NLP and dialog systems research.

Demo

Demo of selected features is available at demo.ipavlov.ai

Breaking changes in version 0.4.0!

• default target variable name for neural evolution was changed from MODELS_PATH to MODEL_PATH.

Breaking changes in version 0.3.0!

• component option fit_on_batch in configuration files was removed and replaced with adaptive usage of the fit_on parameter.

Breaking changes in version 0.2.0!

• utils module was moved from repository root in to deeppavlov module
• ms_bot_framework_utils,server_utils, telegram utils modules was renamed to ms_bot_framework, server and telegram correspondingly
• rename metric functions exact_match to squad_v2_em and squad_f1 to squad_v2_f1
• replace dashes in configs name with underscores

Breaking changes in version 0.1.0!

• As of version 0.1.0 all models, embeddings and other downloaded data for provided configurations are by default downloaded to the .deeppavlov directory in current user's home directory. This can be changed on per-model basis by modifying a ROOT_PATH variable or related fields one by one in model's configuration file.

• In configuration files, for all components, dataset readers and iterators "name" and "class" fields are combined into the "class_name" field.

• deeppavlov.core.commands.infer.build_model_from_config() was renamed to build_model and can be imported from the deeppavlov module directly.

• The way arguments are passed to metrics functions during training and evaluation was changed and documented.

Hello Bot in DeepPavlov

Import key components to build HelloBot.

from deeppavlov.skills.pattern_matching_skill import PatternMatchingSkill
from deeppavlov.agents.default_agent.default_agent import DefaultAgent 
from deeppavlov.agents.processors.highest_confidence_selector import HighestConfidenceSelector

Create skills as pre-defined responses for a user's input containing specific keywords or matching regexps. Every skill returns response and confidence.

hello = PatternMatchingSkill(responses=['Hello world!'], patterns=["hi", "hello", "good day"])
bye = PatternMatchingSkill(['Goodbye world!', 'See you around'], patterns=["bye", "chao", "see you"])
fallback = PatternMatchingSkill(["I don't understand, sorry", 'I can say "Hello world!"'])

Agent executes skills and then takes response from the skill with the highest confidence.

HelloBot = DefaultAgent([hello, bye, fallback], skills_selector=HighestConfidenceSelector())

Give the floor to the HelloBot!

print(HelloBot(['Hello!', 'Boo...', 'Bye.']))

Jupyter notebook with HelloBot example.

Features

Components

Named Entity Recognition | Slot filling

Intent/Sentence Classification | Question Answering over Text (SQuAD)

Sentence Similarity/Ranking | TF-IDF Ranking

Morphological tagging | Automatic Spelling Correction

ELMo training and fine-tuning

Skills

Goal(Task)-oriented Bot | Seq2seq Goal-Oriented bot

Open Domain Questions Answering | eCommerce Bot

Frequently Asked Questions Answering | Pattern Matching

Embeddings

ELMo embeddings for the Russian language

FastText embeddings for the Russian language

Auto ML

Tuning Models with Evolutionary Algorithm

Installation

0. Currently we support Linux and Windows platforms and Python 3.6

   • Python 3.5 is not supported!
   • Windows platform requires Git for Windows (for example, git), Visual Studio 2015/2017 with C++ build tools installed!

1. Create a virtual environment with Python 3.6:

   virtualenv env
   

2. Activate the environment:

   • Linux

   source ./env/bin/activate
   

   • Windows

   .\env\Scripts\activate.bat
   

3. Install the package inside this virtual environment:

   pip install deeppavlov
   

Quick start

To use our pre-trained models, you should first install their requirements:

python -m deeppavlov install <path_to_config>

Then download the models and data for them:

python -m deeppavlov download <path_to_config>

or you can use additional key -d to automatically download all required models and data with any command like interact, riseapi, etc.

Then you can interact with the models or train them with the following command:

python -m deeppavlov <mode> <path_to_config> [-d]

• <mode> can be train, predict, interact, interactbot, interactmsbot or riseapi
• <path_to_config> should be a path to an NLP pipeline json config (e.g. deeppavlov/configs/ner/slotfill_dstc2.json) or a name without the .json extension of one of the config files provided in this repository (e.g. slotfill_dstc2)

For the interactbot mode you should specify Telegram bot token in -t parameter or in TELEGRAM_TOKEN environment variable. Also you should use --no-default-skill optional flag if your component implements an interface of DeepPavlov Skill to skip its wrapping with DeepPavlov DefaultStatelessSkill. If you want to get custom /start and /help Telegram messages for the running model you should:

• Add section to deeppavlov/utils/settings/models_info.json with your custom Telegram messages
• In model config file specify metadata.labels.telegram_utils parameter with name which refers to the added section of deeppavlov/utils/settings/models_info.json

You can also serve DeepPavlov models for:

• Microsoft Bot Framework (see developer guide for the detailed instructions)
• Amazon Alexa (see developer guide for the detailed instructions)

For riseapi mode you should specify api settings (host, port, etc.) in deeppavlov/utils/settings/server_config.json configuration file. If provided, values from model_defaults section override values for the same parameters from common_defaults section. Model names in model_defaults section should be similar to the class names of the models main component. Here is detailed info on the DeepPavlov REST API

All DeepPavlov settings files are stored in deeppavlov/utils/settings by default. You can get full path to it with python -m deeppavlov.settings settings. Also you can move it with with python -m deeppavlov.settings settings -p <new/configs/dir/path> (all your configuration settings will be preserved) or move it to default location with python -m deeppavlov.settings settings -d (all your configuration settings will be RESET to default ones).

For predict you can specify path to input file with -f or --input-file parameter, otherwise, data will be taken from stdin.
Every line of input text will be used as a pipeline input parameter, so one example will consist of as many lines, as many input parameters your pipeline expects.
You can also specify batch size with -b or --batch-size parameter.

Documentation

docs.deeppavlov.ai

Docker images

We have built several DeepPavlov based Docker images, which include:

• DeepPavlov based Jupyter notebook Docker image;
• Docker images which serve some of our models and allow to access them via REST API (riseapi mode).

Here is our DockerHub repository with images and deployment instructions.

Tutorials

Jupyter notebooks explaining how to use DeepPalov for different tasks can be found in /examples/

License

DeepPavlov is Apache 2.0 - licensed.

Support and collaboration

If you have any questions, bug reports or feature requests, please feel free to post on our Github Issues page. Please tag your issue with bug, feature request, or question. Also we’ll be glad to see your pull requests to add new datasets, models, embeddings, etc. In addition, we would like to invite everyone to join our community forum, where you can ask the DeepPavlov community any questions, share ideas, and find like-minded people.

The Team

DeepPavlov is built and maintained by Neural Networks and Deep Learning Lab at MIPT within iPavlov project (part of National Technology Initiative) and in partnership with Sberbank.