uniteai 0.3.0

AI, Inside your Editor.

Your AI Stack in your Editor: Voice-to-text, Local LLM, and GPT, +more.

---

Requirements: Python 3

Editor: VSCode(ium) or Emacs or Any Editor with LSP capabilities (most).

The Mission

The future is about Humans Augmented with AIs.

We need our AI Stack (Online, or local models)

Inside a convenient interface (Text Editors > Web UIs)

Friendly with any editor (The project is an LSP and therefore highly portable)

And close to the code (It's easy to tweak and add features. All the logic happens in friendly python code, not bespoke one-off editor code).

Screencast Demo

Some Core Features

screencast.webm

Document Chat (NEW)

aka Retrieval Augmented Generation

screencast_document_chat.webm

Quickstart, installing Everything

You can install more granularly than everything, but we'll demo everything first.

1.) Make sure Python 3 + Pip is installed.

python --version
pip --version

# or

python3 --version
pip3 --version

2.) The only platform-dependent dependency right now is portaudio, and that is only needed if you want speech-to-text/transcription.

# Mac
brew install portaudio

# Ubuntu/Debian
sudo apt install portaudio19-dev

3.) Install UniteAI.

pip3 install --user "uniteai[all]"  # install deps for all features

Your editor will make use of the installed binary, it so it needs to be on your PATH. Add your python installations bin/ to your system PATH environment variable.

# Linux and Mac
~/.local/bin

# Windows
C:\Users\USERNAME\AppData\Roaming\Python\PythonXX\Scripts\

uniteai_lsp should now be callable from a fresh terminal (but you will never need to call it yourself).

4.) Optional: Start the longlived LLM server which offers your editor a connection to your local large language model.

The editor will not start up the LLM itself, so you can just run this from anywhere, and the editor will connect to it.

uniteai_llm

5.) Install in your editor:

• For VSCode get the uniteai extension. Eg in VSCode, Ctrl-P then ext install uniteai.uniteai .

• For VSCodium, VSCode Marketplace files are not compatible, so you'll need to either:

  • Download the prepackaged uniteai.vsix extension, then:

    codium --install-extension clients/vscode/uniteai.vsix
    

  • DIY:

    npm install -g @vscode/vsce
    git clone https://github.com/freckletonj/uniteai
    cd uniteai/clients/vscode
    vsce package
    codium --install-extension uniteai-version.vsix
    

• For Emacs, copy the lsp-mode config to your init.el.

• For other editors with LSP support (most do), we just need to copy the emacs/vscode configuration, and translate it to your editor. Please submit a PR with new editor configs!

6.) Config:

When you first open a compatible file in your editor, the LSP will start.

It will prompt you to create a config file in your local dir, or your home dir.

You will need to edit this file, and then restart your editor.

Granular installs

If you did pip install "uniteai[all]", ignore this section!

Your config determines what modules/features are loaded.

The following makes sure to get your dependencies for each feature. This will become more relevant when more community features are added.

Transcription dependencies

# Debian/Ubuntu
sudo apt install portaudio19-dev  # needed by PyAudio

# Mac
brew install portaudio  # needed by PyAudio

pip3 install "uniteai[transcription]"

Local LLM dependencies

pip3 install "uniteai[local_llm]"

OpenAI/ChatGPT dependencies

pip3 install "uniteai[openai]"

Keycombos

Your client configuration determines this, so if you are using the example client config examples in ./clients:

VSCode	Emacs	Effect
	M-'	Show Code Actions Menu
Ctrl-Alt-d	C-c l d	Do semantic search on a document
Ctrl-Alt-g	C-c l g	Send region to GPT, stream output to text buffer
Ctrl-Alt-c	C-c l c	Same, but ChatGPT
Ctrl-Alt-l	C-c l l	Same, but Local (eg Falcon) model
Ctrl-Alt-v	C-c l v	Start voice-to-text
Ctrl-Alt-s	C-c l s	Whatevers streaming, stop it

• I'm still figuring out what's most ergonomic, so, I'm accepting feedback.

• Ctrl-Alt-d on ubuntu means defaults to "minimize all windows". You can disable that.

• Cmd-Alt-don Mac does something else too, so, we should revisit default bindings.

Retreival Augmented Generation (RAG)

For the document feature, you can reference one of multiple document types, and lookup passages with a similar "gist" to them (semantic similarity search).

Check that your .uniteai.yaml config has uniteai.document enabled.

You can use links to: YouTube (will read transcripts), Arxiv papers, PDFs, Git repos, or any HTML.

To use this feature, write some YAML, highlight it, and hit C-c l d (emacs) or C-A-d (vscode).

query:
docs:
  - title: (optional)
    url: ...
  - title: ...
    url: ...

It will take a couple minutes for long documents to get an embedding for each chunk it finds in the document, but that then gets cached and goes fast afterward.

More details.

Contributions

Why?

Because there are so many cool tools to yet be added:

• Image creation, eg: "Write a bulleted plan for a Hero's Journey story about X, and make an image for each scene."

• Contextualize the AI via reading my emails via POP3, and possibly responding, eg: "what was that thing my accountant told me not to forget?"

• Ask my database natural language questions, eg: "what were my top 10% customers' top 3 favorite products?"

• Write-ahead for tab-completion, eg: "Once upon a ____".

• Chat with a PDF document, eg: "what do the authors mean by X?"

• Do some searches, scrape the web, and upload it all into my db.

• Sky's the limit.

How?

A Key goal of this project is to be Contributor-Friendly.

• Make an Issue with your cool concept, or bug you found.

• .todo/ is a directory of community "tickets", eg .todo/042_my_cool_feature.md. Make a ticket or take a ticket, and make a PR with your changes!

• ./todo/README.md gives some overview of the library, and advice on building against this library.

• a ./contrib directory is where you can add your custom feature. See ./uniteai/contrib/example.py.

• .uniteai.yml configuration chooses which modules to load/not load.

• The code is well-documented, robust, and simple, to reduce friction.

• Adding a feature is as simple as writing some python code, and making use of uniteai's library to directly handle issues like concurrency and communicating/modifying the text editor.

Misc

Notes on Local LLMs

The file ./llm_server.py launches a TCP server in which the LLM weights are booted up. The lsp_server will make calls to this llm_server.

The reason is that the lsp_server lifecycle is (generally*) managed by the text editor, and LLM models can be really slow to boot up. Especially if you're developing a feature, you do not want the LLM to keep being read into your GPU each time you restart the lsp_server.

* you don't have to let the editor manage the lsp_server. For instance, eglot in emacs allows you to launch it yourself, and then the editor client can just bind to the port.

Falcon LLM Issue:

If Falcon runs on multiple threads, its cache has an issue. You need a separate modelling_RW.py that makes sure it never tries to cache. https://github.com/h2oai/h2ogpt/pull/297

Replacing cos_sim with this seems to do the trick:

def cos_sin(
    self,
    seq_len: int,
    device="cuda",
    dtype=torch.bfloat16,
) -> torch.Tensor:
    t = torch.arange(seq_len, device=device).type_as(self.inv_freq)
    freqs = torch.einsum("i,j->ij", t, self.inv_freq)
    emb = torch.cat((freqs, freqs), dim=-1).to(device)

    if dtype in [torch.float16, torch.bfloat16]:
        emb = emb.float()

    cos_cached = emb.cos()[None, :, :]
    sin_cached = emb.sin()[None, :, :]

    cos_cached = cos_cached.type(dtype)
    sin_cached = sin_cached.type(dtype)

    return cos_cached, sin_cached

A separate bitsandbytes issue remains unresolved, but is less serious than the above. https://github.com/h2oai/h2ogpt/issues/104 https://github.com/TimDettmers/bitsandbytes/issues/162

License

Copyright (c) Josh Freckleton. All rights reserved.

Licensed under the Apache-2.0 license.