Communication Robot Framework
Project description
CLTL Leolani Combot
CLTL Leolani Combot provides the framework for applications that implement human-robot interaction with conversation.
About the Project
This is the successor of the Leolani platform with an improved modular architecture.
Applications
Clone one of the application parents from this project space and follow the instructions there to run them:
Components
Currently, the following components are implemented for the framework:
- Application Repository
Repository to share artifacts and external dependencies between components. - EMISSOR
Representation of interaction data. - Backend
Hardware integration and signal generation. - Automatic Speech Recognition (ASR)
Transcription of audio signals to text. - Voice Activity Detection (VAD)
Detection of speech in audio signals. - Face Recognition
Currently includes face detection, age-gender detection, face recognition - Chat UI
Simple Chat client to display and interact with the conversation - Eliza chat
Eliza based chat. - Get To Know You (G2KY) chat
Establish name based on face recognition. - About-agent Answer questions about the agent itself.
- Knowledge extraction Extracts factoid triples and perspectives from statements and gives back responses or it extracts SPARQL queries from questions and generates answers.
- Knowledge linking Resolves IRIs for mentions and perceptions of things and people such that triples are augmented with IRIs.
- Knowledge representation The models and functions to support the episodic Knowledge Graph.
- Language generation Generates texts from triples.
- Mention detection Detects mentions of entities and visual objects in text.
- Object recognition Detects people and objects in images.
- Question processor Answers questions about the world through the Internet.
- Visual responder Answers questions to the visual context.
To create a new component follow the instructions in the template component.
Getting Started
Running an application
Clone one of the above applications and follow the instructions there to run them.
Prerequisites
Python
Most of the repositories require at least Python 3.8.
One option to manage Python versions is to use
pyenv. Note, however, that pyenv doesn't
work well together with anaconda. To detect if you are using anaconda, run
which python. If you use anaconda, you can use that to manage your Python
version.
make
To build the application, make is used.
On OS X it is recommeded to upgrade make. Since OS X doesn't use standard GNU
utils due their restrictive licence, default make on OS X is way outdated.
One option is to use homebrew:
brew install make
and add the installed gmake command by adding
PATH="$(brew --prefix)/opt/make/libexec/gnubin:$PATH"
to your ~/.zshrc
Docker
To be added.
Development
To work on the development of a specific application, start from the parent repository and follow the steps described below. The description uses the Eliza app as example.
Check-out
To check out all code needed for the Eliza App, follow the instructions in the Eliza app.
Build and run the application
The application is structured into separate components which have their own git repositories and can be run as separate Python applications. The parent repository of the application contains all those component repositories as git submodules.
There is a central application (cltl-eliza-app) that configures and runs all the necessary components it needs, either inside a Python application or as containerized services in a Kubernetes cluster or using docker compose. To run the application, first all components need to be packaged and made available to the application. For this purpose there are makefiles available in the components and the application parent that automate this process. To build the application run
make build
from the parent repository. This command will download external dependencies to cltl-requirements, setup virtual environments for all components, package them and publish the packages to cltl-requirements to make them available to the application and other components.
To run the application follow the instructions in the Eliza parent.
Make changes to the code
Individual components in the parent repository are edited and committed separately, and, after a stable version is
reached, the state of the components is commited in the parent repository, for the workflow see
Working with git submodules.
Modularization allows developing components in isolation. The application and other components depend on a packaged
version of a component only, therefore changes will become available outside of the component only after rebuilding
the application, see above.
To use PyCharm for development see the instructions in Workflow using PyCharm.
To commit changes made to the application see the instructions in Working with git submodules.
Adding a new component
To add a new component to an application follow the instruction in the template component.
Create a new application
HOWTOs
- Workflow using PyCharm
- Working with git submodules
- Setup a new component
- Add a component to a Python app
Content of this repository
This repo provides infrastructre and general code for the platform:
Infrastructure
The cltl.combot.infra module contains library code for infrastructre used in
the application.
Event bus
Components of the application can communicate via an event bus. The
cltl.combot.infra.event module provides the interface and different implementations
of the event bus.
The cltl.combot.infra.topic_worker module provides a convenience class to
implement the subscription to one or multiple topics in the event bus.
Configuration manager
Configuration is made available in the application via a configuration manager.
The cltl.combot.infra.config module provides the interface and different
implementations of the configuration manager.
Resource manager
Access to resources in the application is made available via a resource manager.
This includes providing resources and waiting for resources to become available
as well as managing access to shared resources. The cltl.combot.infra.resource
module provides the interface and different implementations of the resource
manager.
Time util
The cltl.combot.infra.time_util module provides time related utilities to
ease the usage of a consistent time format throughout the application.
Dependency injection
The cltl.combot.infra.di_container module provides a simple utility to use
dependency inject in the application.
Common libraries
To be added.
Events based on EMISSOR
The cltl.combot.event module contains common event payloads.
Contributing
Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature) - Commit your Changes (
git commit -m 'Add some AmazingFeature') - Push to the Branch (
git push origin feature/AmazingFeature) - Open a Pull Request
License
Distributed under the MIT License. See LICENSE for more information.
Authors
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
