VISION-ENABLED DIALOGUE
A dialogue system that combines speech and images in a conversation history with summarisation strategies to improve contextual understanding. It uses a configurable (for example, text-only or multimodal) language model to generate responses. It can be used as a Python library or as a ROS 2 node.
This repository contains the implementation described in the paper "I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robots" by Giulio Antonio Abbo and Tony Belpaeme (March 2025) available here. Link to the HRI'25 LBR paper. For the version of the code used in the paper, see the v0.2.0.
Contents
- Features
- Installation
- Usage - Get started
- Usage - Generating responses
- Usage - Rendering outputs and handling interruptions
- Usage - Manage prompt length
- ROS 2 compatibility
- Furhat robot compatibility
- Contributing
- Reference
Features
- Separates conversation management from response generation, so the dialogue flow, model choice, etc., can be changed independently.
- Supports both text-only generation and vision-language generation through a shared response-generator interface.
- Can choose between the text model and the vision model with a lightweight chooser model, or force the next turn to use the VLM.
- In addition to the usual prompt instructions, the system allows for steering instructions to be sent per turn or persistently; these are put at the end of the prompt to further guide the model's output.
- Applies background history-compaction strategies so older frames (images) and mixed content can be summarised instead of keeping every raw item in the prompt.
- Streams generated text in sentence-sized chunks for faster interactions.
- Stores camera frames, spoken turns, thoughts, and generated turns in a typed conversation history.
- Preserves generation metadata in the history for later inspection.
Installation
For the standalone Python entrypoint, install the main dependencies:
pip install vision-enabled-dialogue
# Or, if you want to install from source, clone the repository and run:
pip install -r requirements.txt
# To run the examples, you will also need OpenCV:
pip install opencv-python
The default GPT client in this repository is configured in code and points to http://localhost:11434/v1 with the API key ollama. If you want to use a different backend, set the environment variables OPENAI_API_BASE_URL and OPENAI_API_KEY before running the code, or pass a custom ResponseGenerator to the DialogueManager constructor. To use OpenAI models, set OPENAI_API_BASE_URL to an empty string.
Usage - Get started
The script in main.py shows a simple example of using the DialogueManager with a webcam and a vision-enabled LLM. Through the default SummariseHeavy strategy with a `HidingSummariser, it always keeps four images in memory and hides the rest. It also contains a more in-depth example of how to customise the dialogue manager.
Ensure you have Ollama running with the gemma3:27b model on port 11434, then run the script:
python main.py
Usage - Generating responses
Responses are generated by the DialogueManager class, which takes a ResponseGenerator, which represent a set of instructions for the LLM. It uses a LLM implementation to produce the actual responses. Currently only the GPT implementation is available, which uses the OpenAI conventions, also compatible with Ollama.
The response is streamed in sentence-sized chunks, so the first part of the response is available before the entire generation is complete. The metadata about the generation is stored in the conversation history, so you can inspect it later.
Usage - Rendering outputs and handling interruptions
Provide a function to handle the generated text chunks to the DialogueManager constructor, for example to print them to the console or send them to a TTS engine. If the speech is interrupted, this function should return the text that was actually spoken, so the dialogue manager can store it in the conversation history. The full generated text is also stored in the conversation history.
The ConversationHistory class uses two callback functions to notify when a new part is added or when a part is summarised. You can use these callbacks to update the UI or trigger other actions.
Usage - Manage prompt length
Conversation length is controlled by the history layer in vision_enabled_dialogue/conversation_history.
The main building blocks are:
ConversationHistory, which stores the orderedConversationPartobjects and applies growth strategies in the background.GrowthStrategy, which defines how to summarise parts of the conversation using aSummariser.SummariseHeavy, summarises older heavy parts (e.g., frames) once their count exceeds a limit.KeepLastNStrategy, which keeps the last N important parts (e.g., text) and summarises the rest.
Summariser, which defines how to summarise conversation parts, when a growth strategy requires it.FrameSummariser, which produces a summary of images.MixedSummariser, which produces a text summary for mixed conversation parts.HidingSummariser, which hides parts from the transcript (but does not remove them from memory).
For instance, to summarise the initial part of the conversation when it becomes too long, you can use a KeepLastNStrategy with a MixedSummariser.
If you want to reduce the prompt size, substituting consecutive frames with a text summary, you can use SummariseHeavy strategy with a FrameSummariser, defining the minimum and maximum number of frames to keep in the prompt. If instead you want to hide older parts from the prompt, while keeping them in memory, you can use HidingSummariser instead.
ROS 2 compatibility
This package is compatible with ROS 2 (tested on Humble). An example node is provided in vision_enabled_dialogue/main_ros.py. It declares the persistent_steering parameter and uses these interfaces:
- Subscribed topics:
/humans/voices/anonymous_speaker/speech,dialogue/steering,/camera/image_raw,/camera/image_raw/compressed - Published topics:
/conversation,/transcript - Action client:
/tts_engine/tts
To test the node, you can install this repo as a ROS 2 Python package (in your-workspace/src/vision_enabled_dialogue), make sure the dependencies are installed, then build the workspace, source and run it with:
colcon build --symlink-install
source ./install/setup.bash
ros2 run vision_enabled_dialogue main
Furhat robot compatibility
Following the deprecation of Furhat's RemoteAPIs, this repository no longer provides a Furhat-specific entrypoint. You can use this library in your existing Furhat project.
Contributing
Contributions are welcome! Please open an issue or submit a pull request. For major changes, please open an issue first to discuss what you would like to change.
Reference
If you use this code in your work, please cite the paper:
@inproceedings{10.5555/3721488.3721641,
author = {Abbo, Giulio Antonio and Belpaeme, Tony},
title = {I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robots},
year = {2025},
publisher = {IEEE Press},
booktitle = {Proceedings of the 2025 ACM/IEEE International Conference on Human-Robot Interaction},
pages = {1176–1180},
numpages = {5},
keywords = {conversation, dialogue, hri, large language model, prompt engineering, ros, vision language model},
location = {Melbourne, Australia},
series = {HRI '25}
}
Metadata
Release files for vision-enabled-dialogue 0.3.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| vision_enabled_dialogue-0.3.3.tar.gz | 27.6 kB | With provenance | Details for vision_enabled_dialogue-0.3.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| vision_enabled_dialogue-0.3.3-py3-none-any.whl | Python 3 | none | any | Details for vision_enabled_dialogue-0.3.3-py3-none-any.whl |
Total release size: 59.8 kB
Release files / vision_enabled_dialogue-0.3.3.tar.gz
| Download URL | vision_enabled_dialogue-0.3.3.tar.gz |
|---|---|
| Size | 27.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
991aa734131b25f10a94d886228475bce54edb982ac6381a7bb85e2870d2e2f4
|
|
BLAKE2b-256 checksum How to use checksums |
b1b965bb249b3186e5e89f9984124153cb566ca77f56b1b814245c1f31b03dd0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 13, 2026.
Transparency logRelease files / vision_enabled_dialogue-0.3.3-py3-none-any.whl
| Download URL | vision_enabled_dialogue-0.3.3-py3-none-any.whl |
|---|---|
| Size | 32.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
a3a03e8647720c3f5342f5eeef3a320c0cc7a7272bb4eaa29f50560aa85a5e1f
|
|
BLAKE2b-256 checksum How to use checksums |
69b981f39148ebb095b2cda2537667798ef3a7fe1a530d5094db0fa62c431e53
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 13, 2026.
Transparency logSecurity for vision-enabled-dialogue 0.3.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for vision_enabled_dialogue-0.3.3.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 13, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.