Skip to main content

BALanced Execution through Natural Activation : a human-computer interaction methodology for code running.

Project description

BALanced Execution through Natural Activation : a human-computer interaction methodology for code running.

Python License Maintainer

BALENA is a voice interaction framework utilizing state-of-the-art natural language processing and audio processing models to create a system that can interpret voice commands and associate them with predefined actions. The framework leverages the power of transformers and signal processing to understand user intent via spoken language and correlates them with a series of predefined actionable responses.

Plot

Features

  • Real-time audio streaming and recording: Record audio from the microphone in real time for processing.
  • Speech recognition with Wav2Vec 2.0: Use a pre-trained Wav2Vec 2.0 model to convert speech to text.
  • Text similarity and action triggering: Encode the transcribed text to a vector space and find the closest action using sentence similarity techniques.
  • High-pass filtering: Process the audio signal with a high-pass filter to enhance signal quality.
  • Auto-correction: Utilize the Jaccard distance to correct words in the transcribed text auto-magically.
  • Framework flexibility: Support for different device execution contexts, allowing for usage on both CPU and CUDA devices.

Dependencies

Below is a list of the main dependencies for Speech2Interact:

  • ssl: To handle SSL/TLS encryption for establishing secure connections.
  • time: For timing operations, such as streaming duration.
  • warnings: To manage runtime warnings produced during execution.
  • nltk: Utilized for natural language processing tasks like Jaccard distance calculations.
  • numpy: Provides support for large, multi-dimensional arrays and matrices, along with a collection of mathematical functions to operate on these arrays.
  • pyaudio: Allows for real-time audio recording and playback.
  • scipy: Used for signal processing, including applying the high-pass filter.
  • torch: An open-source machine learning framework that accelerates the path from research prototyping to production deployment.
  • transformers: Provides thousands of pre-trained models to perform tasks on texts such as classification, information extraction, question answering, and more.
  • sentence_transformers: A Python framework for state-of-the-art sentence and text embeddings.

Pre-trained Models

  • Wav2Vec 2.0: facebook/wav2vec2-large-960h
  • Sentence Similarity: sentence-transformers/all-mpnet-base-v2

Usage

Here's how you can use apple-ocr:

  1. Installation: Install the required libraries, including Torch, NumPy...
  2. Initialization: Create an instance of the OCR class, providing an image to be processed.
from balena.hci import Speech2Interact

actions = {
		"validate": [
        "confirm", "approve", "verify", "validate", 
        "authenticate", "ratify", "endorse", "certify", 
        "pass", "authorize", "accredit", "yes"
    ],
    "invalidate": [
        "reject", "deny", "invalidate", "disapprove", 
        "refuse", "void", "nullify", "revoke", 
        "discredit", "disqualify", "abrogate", "annul", "no"
    ]
}

instance = Speech2Interact(
    actions=actions,
    wav2vec_model="facebook/wav2vec2-large-960h", 
    sentence_similarity_model="sentence-transformers/all-mpnet-base-v2"
)

action = instance.recognize_speech(
		duration=3
)

The recognize_speech function captures audio for a set duration, processes the audio, and attempts to match the spoken words with a predefined action.

Citing this project

If you use this code in your research, please use the following BibTeX entry.

@misc{louisbrulenaudet2023,
	author = {Louis Brulé Naudet},
	title = {BALanced Execution through Natural Activation : a human-computer interaction methodology for code running},
	howpublished = {\url{https://github.com/louisbrulenaudet/balena}},
	year = {2023}
}

Feedback

If you have any feedback, please reach out at louisbrulenaudet@icloud.com.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

balena-cpu-1.0.0.tar.gz (15.2 kB view details)

Uploaded Source

Built Distribution

balena_cpu-1.0.0-py3-none-any.whl (14.7 kB view details)

Uploaded Python 3

File details

Details for the file balena-cpu-1.0.0.tar.gz.

File metadata

  • Download URL: balena-cpu-1.0.0.tar.gz
  • Upload date:
  • Size: 15.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.7

File hashes

Hashes for balena-cpu-1.0.0.tar.gz
Algorithm Hash digest
SHA256 786243cc4e35863907941a2a2bf08bd63349c59cfb9b3ecc42300411d0b9122e
MD5 442b64d77366e8dce30bf7c09bba6d52
BLAKE2b-256 3e6bd672d2f3db8c81c201d80b73140fe46f862ff8519d327975735ed0890f67

See more details on using hashes here.

File details

Details for the file balena_cpu-1.0.0-py3-none-any.whl.

File metadata

  • Download URL: balena_cpu-1.0.0-py3-none-any.whl
  • Upload date:
  • Size: 14.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.7

File hashes

Hashes for balena_cpu-1.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 80cf487623d5edb671dbddfeb73a89b9a4a658adeb8c2ca296cd959659408649
MD5 0cbda165371f9e839ddf0a9d578d50ca
BLAKE2b-256 a03105cc16fb8a9b3441061794070f8171e5932ae83002ed93f147565f3170fc

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page