jarvis-ironman 3.2

Fully Functional Voice Based Natural Language UI

Fully Functional Voice Based Natural Language UI

Platform Supported

Language Stats

Repo Stats

Code Stats

Deployments

Activity

Development and Maintenance

Reach Out

Kick off

Install

python -m pip install jarvis-ironman

Initiate

import jarvis


if __name__ == '__main__':
    jarvis.start()

Prerequisites

• MacOS
  Tested on macOS High Sierra, Mojave, Catalina, Big Sur, Monterey and Ventura*

  • System Preferences → Security & Privacy → Privacy
  • Click + sign and add the preferred IDE and Terminal in the following sections in left pane.
    • Microphone - Required to listen and respond.
    • Accessibility - Required to use key combinations for brightness and volume controls.
    • Camera - [Optional] Required only during face recognition/detection.
    • Automation - Required to control System Events and other apps like Outlook and Calendar.
    • Files and Folders [OR] Full Disk Access - Required for all FileIO operations.

• Linux
  Tested on Ubuntu 22.04 LTS

  • Store the host machine's password as the env var: ROOT_PASSWORD
  • Unlike macOS and Windows, Ubuntu does not have app specific permissions.

• Windows
  Tested on Windows 10

  • Settings → Privacy
    • Microphone - Required to listen and respond.
    • Camera - [Optional] Required only during face recognition/detection.
    • Unlike macOS, Windows pops a confirmation window to Allow or Deny access to files and folders.
  • Install Anaconda or Miniconda, VisualStudio C++ BuildTools, and Git
  • Make sure C++ build tools are installed completely and restart
  • Add anaconda/miniconda scripts location to PATH in Environment Variables

Test peripherals

Camera

from jarvis.modules.camera import camera


if __name__ == '__main__':
    cam_object = camera.Camera()
    print(cam_object.get_index())  # Get index id of each camera
    print(cam_object.list_cameras())  # List all connected cameras

Text to Speech

from jarvis.modules.speaker import speak


if __name__ == '__main__':
    speak_object = speak.Speaker()
    speak_object.run()
    print(list(speak_object.get_all_voices()))

Speech to Text

import asyncio

from jarvis.modules.microphone import recognizer


if __name__ == '__main__':
    asyncio.run(recognizer.main())

ENV Variables

Environment variables are loaded from a .env file and validated using pydantic

More on Environment variables

• ROOT_PASSWORD - System password to get the system vitals and run other sudo commands. (mandatory for Linux)
• NAME - Name which Jarvis should address the user by. Defaults to Vignesh
• TITLE - Title which Jarvis should address the user by. Defaults to sir
• LIMITED - Boolean flag to run only the main version of Jarvis skipping background processes. Defaults to False (enforced based on the number of CPU cores)
• PLOT_MIC - Boolean value to show microphone usage in realtime. Defaults to True

Audio

• WAKE_WORDS - List of wake words to initiate Jarvis. Defaults to ['jarvis'] (Defaults to ['alexa'] in legacy macOS)
  :warning: Jarvis has limitations on the wake words as it relies on ML libraries for wake word detection.

• VOICE_NAME - Name of the voice supported by the OperatingSystem. Defaults to the author's favorite.

• SPEECH_RATE - Speed/rate at which the text should be spoken. Defaults to the value from pyttsx3 module. Typically 200

  To add more voices

  macOS:

  • System Preferences → Accessibility → Spoken Content → System voice → Manage Voices...

  Windows:

  • Settings → Time & Language → Speech → Manage voices → Add voices

• SENSITIVITY - Hot word detection sensitivity. Allowed range: [0-1] Defaults to 0.5

• LISTENER_TIMEOUT - Timeout in seconds until which the listener should wait for speech. Defaults to 3

• LISTENER_PHRASE_LIMIT - Timeout in seconds until which the listener will remain active. Defaults to None

• RECOGNIZER_SETTINGS - A JSON object with customized speech recognition settings.

  Custom settings for speech recognition

  These are customized according to the author's voice pitch. Please use recognizer.py to figure out the suitable values on a trial and error basis.

  Sample settings (formatted as JSON object)

  • RECOGNIZER_SETTINGS: '{"energy_threshold": 1100, "dynamic_energy_threshold": false, "pause_threshold": 2, "phrase_threshold": 0.1, "non_speaking_duration": 2}'

  Description

  • energy_threshold: Minimum audio energy to consider for recording. Greater the value, louder the voice should be.
  • dynamic_energy_threshold: Change considerable audio energy threshold dynamically.
  • pause_threshold: Seconds of non-speaking audio before a phrase is considered complete.
  • phrase_threshold: Minimum seconds of speaking audio before it can be considered a phrase - values below this are ignored. This helps to filter out clicks and pops.
  • non_speaking_duration: Seconds of non-speaking audio to keep on both sides of the recording.

---

Peripherals

• CAMERA_INDEX - Camera index that has to be used. Run camera.py to get the index value of each camera.
• SPEAKER_INDEX - Speaker index that has to be used. Run peripherals.py to get the index value of each speaker.
• MICROPHONE_INDEX - Microphone index that has to be used. Run peripherals.py to get the index value of each microphone.

Logging

• DEBUG - Boolean flag to enable debug level for logging. Defaults to False
• LOG_RETENTION - Number of days to store the logs. Defaults to 10

Units

• DISTANCE_UNIT - Unit in which speed/distance should be measured. Defaults to miles
• TEMPERATURE_UNIT - Unit in which temperature should be measured. Defaults to fahrenheit

Features

• GIT_USER - GitHub Username
• GIT_PASS - GitHub Token
• BIRTHDAY - Birth date in the format DD-MM - Example: 24-April
• WEATHER_API - API Key from openweathermap
• NEWS_API - API Key from newsapi
• MAPS_API - API Key for maps from Google
• OPENAI_API - API Key from openai
• OPENAI_REUSE_THRESHOLD - This is the threshold for sequence matching repeated requests. Defaults to None

Setting the OPENAI_REUSE_THRESHOLD (range: 0.5 to 0.9) will try to avoid hitting OpenAI API more frequently for repeated identical requests.
This threshold is the ratio of acceptable result for SequenceMatching between a new request and historically identical request.

Email/SMS notifications

• GMAIL_USER - Gmail account username to send and read emails.
• GMAIL_PASS - Gmail account password to send and read emails.
• OPEN_GMAIL_USER - Alternate gmail account username to send an SMS. (Defaults to GMAIL_USER)
• OPEN_GMAIL_PASS - Alternate gmail account password to send an SMS. (Defaults to GMAIL_PASS)
• RECIPIENT - Email address to which the emails from jarvis have to be received.

iOS integrations

• ICLOUD_USER - iCloud account username/email.
• ICLOUD_PASS - iCloud account password.
• ICLOUD_RECOVERY - Recovery phone number to activate lost mode on a target device - Example: [phone]
• PHONE_NUMBER - To send SMS from Jarvis - Example: [phone]

Calendar/Meeting integrations

• ICS_URL - Shared calendar URL to get meetings information from. Should end with .ics
• MUTE_FOR_MEETINGS - Boolean value to disable the listener during meetings. Defaults to False
• EVENT_APP - To read events from outlook or calendar application in macOS. Defaults to calendar
  :bulb:   When calender is used, the name of the calendar within the Calendar.app should be Jarvis
  

Background scans [Defaults to 1 hour]

• SYNC_MEETINGS - Interval in seconds to generate meetings information using an ics URL.
• SYNC_EVENTS - Interval in seconds to generate events information using calendar or outlook application.

Scheduled

• WEATHER_ALERT - Time (in 12h %I:%M %p format) when the weather alert should be fired - Example: 09:00 AM

  • WEATHER_ALERT_MIN - Degrees below which alert has to be fired.
  • WEATHER_ALERT_MAX - Degrees above which alert has to be fired.

  Alerts in SMS and Email if temperature is higher than WEATHER_ALERT_MAX or lower than WEATHER_ALERT_MIN or severe weather warnings.
  This feature can also be enabled from automation.yaml by using the keyword weather in phrase for the task

• CRONTAB - Runs external tasks using cron expressions. Needs to be stored as env var.

  Sample value

  [
    "0 0 * * 1-5/2 find /var/log -delete",
    "0 5 * * 1 tar -zcf /var/backups/home.tgz /home/"
  ]
  

Wi-Fi Controls

• WIFI_SSID - SSID of the wireless connection.
• WIFI_PASSWORD - Password for the wireless connection.
• CONNECTION_RETRY - Frequency in seconds to check for an active internet connection. Defaults to 10 seconds.

VPNServer integration

• VPN_USERNAME - Username to create vpn-server. Defaults to profile username.
• VPN_PASSWORD - Password to authenticate vpn-server. Defaults to profile password.
• VPN_DOMAIN - Domain name for the hosted zone.
• VPN_RECORD_NAME - Alias record name to access VPN server.

Car Controls - Applies only for JLR vehicles subscribed to InControl application.

• CAR_EMAIL - Email address to log in to InControl API.
• CAR_PASS - Password to authenticate InControl API.
• CAR_PIN - InControl PIN.

Garage Controls - Applies only for garages using MyQ garage controller.

• MYQ_USERNAME - Email address to log in to MyQ API.
• MYQ_PASSWORD - Password to authenticate MyQ API.

Telegram Bot integration

• BOT_TOKEN - Telegram BOT token.
• BOT_CHAT_IDS - List of userID/chatID to authorize.
• BOT_USERS - List of usernames to authorize.

OS Agnostic Voice Model

• SPEECH_SYNTHESIS_TIMEOUT - Timeout to connect to the docker container that processes text to speech requests.

• SPEECH_SYNTHESIS_VOICE - Voice for the speech synthesis model. Defaults to author's favorite.

• SPEECH_SYNTHESIS_QUALITY - Quality of speech synthesis conversion. Defaults to medium.
  

  To enable independent speech-synthesis

  docker run \
      -it \
      -p 5002:5002 \
      -e "HOME=${HOME}" \
      -v "$HOME:${HOME}" \
      -v /usr/share/ca-certificates:/usr/share/ca-certificates \
      -v /etc/ssl/certs:/etc/ssl/certs \
      -w "${PWD}" \
      --user "$(id -u):$(id -g)" \
      thevickypedia/speech-synthesis
  

  :bulb:   Speech Synthesis can run on a docker container for better voices but, response might be negligibly slower. If you don't have docker installed or simply don't want to use it, set the SPEECH_SYNTHESIS_TIMEOUT env var to 0. This is also done automatically if failed to launch a docker container upon startup.

---

Offline communicator

• OFFLINE_PORT - Port number to initiate offline communicator. Defaults to 4483
• OFFLINE_PASS - Secure phrase to authenticate offline requests. Defaults to OfflineComm
• WORKERS - Number of uvicorn workers (processes) to spin up. Defaults to 1

Stock Portfolio

• ROBINHOOD_USER - Robinhood account username.
• ROBINHOOD_PASS - Robinhood account password.
• ROBINHOOD_QR - Robinhood login QR code

API Features

• ROBINHOOD_ENDPOINT_AUTH - Authentication token to access the robinhood portfolio which is generated every hour.
• SURVEILLANCE_ENDPOINT_AUTH - Token to access webcam live feed via Jarvis API.
• SURVEILLANCE_SESSION_TIMEOUT - Session time out for /surveillance. Defaults to 300 seconds.
• STOCK_MONITOR_API - API key(s) to skip OTP validation for /stock-monitor endpoint.

Entry should be a dictionary of with email as key and token as value. Eg: '{"[e-mail]": "@lfaROm$o"}'

Custom Feature Flag

• AUTHOR_MODE - Enables reverse proxy tunnel (to expose API endpoint), robinhood portfolio scan (every hour) and stock monitoring (every 15 minutes). Defaults to False

Contacts [Optional]

Jarvis can send on demand notifications using a contacts.yaml file stored in fileio directory. Uses gmail-connector for SMS and email notifications.

Setup Instructions

Note: Jarvis currently supports sending emails only when the contacts.yaml file is present, however phone numbers can be used directly.

phone:
  Tony: 0123456789
  Thor: 1234567890
email:
  Eddard: [e-mail]
  Aegon: [e-mail]

Smart Devices [Optional]

Jarvis supports MagicHome for lights, LGWebOS and Roku for TVs.

This feature requires a smart_devices.yaml file which should be stored within the fileio directory.

• TV hostnames should include the brand name [LG/Roku] to distinguish the modules accordingly.

  • This will be set by default, if yours doesn't include the brand name change it in the TV settings.

• To wake up Roku TVs using MAC address, make sure the Bandwidth saver feature is turned off under,

  Settings/Network/Bandwidth saver >> Off
  

• For first time users on LGWebOS TVs, there will be a prompt on the TV to accept the connection request.

  • Once the connection request is accepted a client key will be generated and logged.
  • Please make sure to store this in smart_devices.yaml file to avoid repeated connection prompt.

Setup Instructions

• The source file should have an initial key to distinguish between Lights and TV
• The name used in the keys (for both lights and tv) will be the identifier when an action is requested.
• Lights should be a dictionary of identifier and a list of hostnames.
• TVs should be a nested dictionary of multiple parameters.
• The source file (smart_devices.yaml) should be as following:

Lights:
  bedroom:
    - 'HOSTNAMES'
  hallway:
    - 'HOSTNAMES'
  hallway basement:
    - 'HOSTNAMES'
  kitchen:
    - 'HOSTNAMES'
  living room:
    - 'HOSTNAMES'
TV:
  living room tv:
    hostname: 'LGWebOSTV'
    client_key: 'CLIENT_KEY'  # Generated automatically if unavailable
    mac_address:  # Mac addresses can either be a string or a list
    - 'WIRED_MAC_ADDRESS'
    - 'WIRELESS_MAC_ADDRESS'
  bedroom tv:
    hostname: 'RokuTV'
    mac_address: 'MAC_ADDRESS'

Automation [Optional]

Jarvis can execute offline compatible tasks at pre-defined times without any user interaction.

This feature requires an automation.yaml file which should be stored within the fileio directory.

Setup Instructions

The YAML file should be a dictionary within a dictionary that looks like the below.

OPTIONAL: The key, day can be a list of days, or a str of a specific day or simply a str saying weekday or weekend when the particular automation should be executed.

Not having the key day will run the automation daily. Date format should match exactly as described below.

06:00 AM:
  - task: set my bedroom lights to 50%
    day: weekday  # Runs only between Monday and Friday
06:30 AM:
  - task: set my bedroom lights to 100%
    day:  # Runs only on Monday, Wednesday and Friday
    - Monday
    - wednesday
    - FRIDAY
08:00 AM:  # Runs only on Saturday and Sunday
  - task: set my bedroom lights to 100%
    day: weekend
09:00 PM:  # Runs daily (can take both list of dict and dict as argument)
  task: set my bedroom lights to 5%
12:00 AM:  # Even performs tasks that are not supported via voice commands
  - task: restart all background processes
  - task: turn off all lights
    day: weekday

Background Tasks [Optional]

Jarvis supports both internal and external background tasks to be scheduled.

• Jarvis can run internal tasks at certain intervals.

This feature requires a background_tasks.yaml file which should be stored within the fileio directory.

Setup Instructions

- seconds: 1_800
  task: just turn off all lights  # Runs every 30 minutes - 'just' flag retains the lights' last setting (eg: brightness or color)
  ignore_hours:  # Ignore the schedule at 5 AM and 10 PM
    - 5
    - 22
- seconds: 10_800
  task: remind me to drink water  # Runs every 3 hours ignoring the hours specified
  ignore_hours: "21-6"  # Ignore the schedule between 9 PM and 6 AM

Simulation [Optional]

Jarvis can execute tasks as a simulation to test the required functions and send an email with the results.

This feature requires a simulation.yaml file which should be stored within the fileio directory.

Setup Instructions

The YAML file should be a list of phrases within a dictionary that looks like the below.

meeting_event:
- get me the events from my calendar
- what meetings do I have today

Custom Conditions [Optional]

Jarvis can execute function(s) directly based on a custom condition map, via both voice and offline communicators.

• The condition mapping may contain one or more functions to be executed for a single keyword/phrase.
• These functions are executed as ordered in the mapping file and responses will be delivered as each task is done.
• For offline communicators, in case of multi-function mapping, the responses are gathered and then delivered at once.

This feature requires a conditions.yaml file which should be stored within the fileio directory.

Setup Instructions

The YAML file should be a dictionary within a dictionary that looks like the below.

lumos:  # custom keyword
  lights: turn on all lights  # function_name: phrase passed as argument
knox:
  lights: turn off all lights
  television: turn off bedroom tv
fire up the chopper:  # custom phrase
  garage: open garage
  car: start car

:warning: This is a direct mapping to the functions, so the input phrases are not validated by Jarvis. This will raise a UserWarning if invalid function name or phrase is entered in the mapping file.

Guide

Please refer to the wiki page for API usage, access controls, env variables, features' overview and demo videos.

FAQs

Please refer to the FAQs section of the wiki.

Coding Standards

Docstring format: Google
Styling conventions: PEP 8
Clean code with pre-commit hooks: flake8 and isort

Release Notes

Requirement

python -m pip install changelog-generator

Usage

changelog reverse -f release_notes.rst -t 'Release Notes'

Linting

PreCommit will ensure linting, and the doc creation are run on every commit.

Requirement

pip install sphinx==5.1.1 pre-commit recommonmark

Usage

pre-commit run --all-files

Pypi Package

https://pypi.org/project/jarvis-ironman/

Runbook

https://thevickypedia.github.io/Jarvis/

License & copyright

© Vignesh Sivanandha Rao

Licensed under the MIT License