zrb-face

A companion for zrb: a 3D character in a small, see-through, always-on-top window. She idles naturally (breathing, shifting her weight, glancing around, stretching), shows what zrb is doing and how it feels, moves her mouth with zrb's voice, dances while you play music, gestures with your own animations, and can start a conversation herself.

Quick start

pipx install zrb-face          # or: pip install zrb-face, next to zrb; from a checkout: pip install -e .

# once, e.g. in ~/.zshrc: load the zrb side into every zrb chat
export ZRB_INIT_SCRIPTS="$ZRB_INIT_SCRIPTS:$(zrb-face init-path)"

mkdir -p ~/.config/zrb-face/models && cp my.vrm ~/.config/zrb-face/models/   # a model: see Changing the model
zrb-face my                    # (re)start the server, open a window with my.vrm
zrb chat

The window is a small desktop app; download it from the releases or build it (see The desktop app). Without it, zrb-face opens the page in your browser instead.

zrb-face opens the Windows build from WSL, the installed app on macOS, or Electron from app/ in a source checkout (ZRB_FACE_APP overrides the program). It restarts the server each time, so the server always runs the current code with your current settings; open windows reload by themselves. The server keeps running across chats; its output goes to zrb-face.log in the temp folder, and pkill -f "[z]rb_face/server.py" stops it. The order doesn't matter: the window waits for the server, reloads itself when the server restarts, and zrb skips sending while either is missing. If a model can't be loaded, the window says so in red and tries again every few seconds.

How it fits together

 zrb chat  ── POST /event ──▶  zrb-face server  ── page, models, animations, live events ──▶  window (face.html)
 + zrb_init.py                 (port 7777)                                                    in the app or a browser

| zrb_init.py, .github/workflows/ | CI: the tests and lint on every push and pull request, and an LLM code review (the repository's own review and submit-comment zrb tasks, in the root zrb_init.py) posted on pull requests. |

zrb runs in WSL on Windows, but a window that stays on top has to be a Windows program. Windows can reach a server in WSL (not the other way around in WSL's default NAT mode), so the server runs next to zrb and the window connects to it.

Running by hand

What zrb-face does, per platform:

zrb-face serve &                                                                          # the server
/mnt/c/Users/$USER/AppData/Local/zrb-face/zrb-face.exe rem </dev/null &>/dev/null &!      # Windows, from WSL
open -n -a zrb-face --args rem                                                            # macOS, installed app
~/zrb-face/app/node_modules/.bin/electron ~/zrb-face/app rem                              # Linux, or any OS from source

The window takes a model name (rem or rem.vrm; without one, the first model alphabetically) and optionally another server URL (http://localhost:7779). Start the Windows build detached as above, or zsh stops it ("suspended (tty input)"). Instead of the app you can open http://localhost:7777/?model=rem in a browser: the same page, but not see-through or always on top.

Using the window

• Move: drag it. Close: Alt+F4 on Windows, Cmd+Q on macOS.
• Camera angle: click the window, then use the keys in Keyboard. The angle is remembered.

What she shows

• Voice: the mouth follows the loudness of the speech, in 40 ms steps — one value per step of audio, windowed across the chunk boundaries of the stream, so it ends when the voice does — and she nods slightly while speaking. It needs a voice zrb plays itself (piper, openai, gemini); say and termux play the audio themselves, so zrb never sees it. The mouth still starts slightly ahead of the voice, because zrb reads audio ahead of playing it; fixing that exactly needs a frame observer in zrb's PcmUtterance._fill.
• Face: she blinks every few seconds, and her eyes are on you (the camera) the way real eyes are: small jumps around your face, a glance away now and then (often with a blink), more of them while thinking, fewer while speaking; she smiles softly every 25–50 seconds and after finishing a task.
• Body: each state loops its own animations, falling back to the idles; see Animations. On the first run of a conversation she plays the Greeting action if there is one; later turns don't greet again.
• Emotion: with zrb's emotion on (ZRB_LLM_EMOTION_ENABLED=true), zrb opens each reply with an emoji, and she shows it for 6 s with the matching VRM expression (happy, sad, angry, surprised, relaxed). The user's own emotion isn't shown: her reply answers it.
• Gestures: zrb gets her tools, all auto-approved: FaceActions lists her animations by pool with what each looks like and how long it takes, FaceAction plays one once alongside its words (an action by name: Greeting, Goodbye, …; any other clip by pool/name: thinking/Thinking), and FaceLook shows zrb her window as you see it (her pose and expression from your side, plus what she is doing), to check how a gesture comes across or when you ask how she looks. FaceModels lists the avatars (the .vrm models) and which one is showing, and FaceModel switches every open window to another one, when you ask her to look different. The actions and models are read on every call, so new ones work without restarting zrb. FaceLook sees only her own window, never your screen, and needs a model that reads images (or zrb's ZRB_LLM_MULTIMODAL_MODEL).
• Music: when something plays music (a browser, Spotify, …) she dances with the dance/ animations while idle; anything zrb does takes over, and she goes back to dancing after. The server watches the system's media status: Windows' media controls (also from WSL), playerctl on Linux, or on macOS media-control (brew install media-control; it also sees a browser tab playing) and else Spotify and Music by AppleScript (which asks permission to control them the first time). Any playing media counts, a talk video too. Use zrb-face music on / off to override it; ZRB_FACE_MUSIC=off stops the watching.
• Talking first (off by default): with ZRB_FACE_TALK_FIRST=idle she speaks up after you've been quiet for a while during waking hours (8:00–22:00); with camera, when you sit back down after being away. She never interrupts a running turn or her own speech, waits a cooldown between such turns, and replies … (shown, not spoken) when there's nothing worth saying.

The mouth, the blink and the smile need the model's aa, blink and relaxed expressions. A model without them still nods while speaking.

Wiring it into zrb

zrb_face/zrb_init.py is all zrb needs; load it through ZRB_INIT_SCRIPTS as in Quick start (zrb-face init-path prints where it is). It owns zrb's speech (enable_speech), so other init scripts should only choose the voice (CFG.DEFAULT_LLM_SPEECH_BACKEND, CFG.DEFAULT_LLM_SPEECH_VOICE) and not call enable_speech themselves; whichever script calls it last would win.

Configuration

Environment variables

The zrb-side names follow zrb's CFG.ENV_PREFIX (ZRB by default). The server and the zrb-face command run without zrb, so their names are fixed.

zrb settings it depends on

Command line and URL

Keyboard (click the window first)

The camera angle is saved in the window's storage and comes back next time.

Files and folders

Models and animations are looked up in folders, in this order; the first file with a given name wins, so your own files override the shipped ones. zrb-face dirs shows the list.

In the code

Constants with no setting of their own; edit the file to change them.

Changing the model

Put any VRM file (VRM 0.x or 1.0) in ~/.config/zrb-face/models/ (or a folder in ZRB_FACE_MODELS) as <name>.vrm and open it by name: zrb-face <name>. Without a name, the window shows the first model alphabetically. To switch while she runs, zrb-face model <name> (or ask zrb: its FaceModel tool) reloads every open window with the new model; the server remembers it for windows opened later, until it restarts. GET /model says which one is chosen and which ones the open windows show. No model ships with the package. Model files are kept out of git (*.vrm is ignored). Nothing in the code depends on the model: the camera fits it, and the pose and idle come from the animations, which three-vrm retargets onto any VRM, as VRoid Hub's playground does.

Where to get models:

• VRoid Studio: free, for Windows and macOS. Make your own and export it as VRM. You own it, and it has every expression.
• VRoid Hub: thousands of models. Only the ones marked downloadable can be downloaded, and each has its own conditions (avatar use, commercial use, redistribution, changes). Read them before you change or share the model.
• BOOTH: search for VRM and filter for free items. The terms are on each item's page, often only in Japanese.
• Open Source Avatars: free VRM avatars with direct download links. The license is set per collection (CC0, CC-BY, …), so check it.
• OpenGameArt: VRoid Studio CC0 models: public domain sample models from early VRoid Studio.

Fan models of existing characters (Cortana, Kusanagi, and others) are fine on your own desktop, but don't commit or share them.

Animations

Every <pool>/<name>.vrma (VRM Animation) in the animation folders loads when the page starts. The pool, the subfolder, says when it plays:

A file whose VRMC_vrm_animation leaves specVersion out is read as "1.0", the version most exporters mean, so clips from an older exporter still load instead of vanishing with a line in the console; one that says "0" is VRM 0.x, in centimetres, and is skipped with a warning.

Any other pool loads too, and plays once something asks for it (zrb-face action <pool>/<name>, or zrb's FaceAction).

Pools nest: a subfolder's clips belong to the pool above it, as one group in the random pick. idle/calm/*.vrma loops with the idles, as often as any one other idle; actions/greetings/hi.vrma is the action greetings/hi. Overrides go by the whole path: your actions/greetings/hi.vrma replaces only that one. A clip is always played by its full path (thinking/Thinking, actions/greetings/hi); an action's short name (greetings/hi) works too, unless it is some other clip's full path (an actions/thinking/Thinking.vrma is actions/thinking/Thinking, as thinking/Thinking is the loop).

To add your own, drop the file in, e.g. ~/.config/zrb-face/anim/actions/Wave.vrma, and restart with zrb-face so the window loads it; zrb's tools see it at once. The folders cascade per file: yours replaces a shipped clip of the same <pool>/<name>, adds any new one, and every shipped clip you don't have still plays. A description, which FaceActions shows the agent, is read from the file's animation extras.description (optional). A file named like a shipped one replaces it: ~/.config/zrb-face/anim/idle/Relax.vrma. Try an action with zrb-face action Wave.

Within a pool, clips play in random order with crossfades, picked by group: the first part of the name before a - (stretch-side belongs to stretch), so a family of clips comes up as often as any single other one. On top of whatever plays: the head tilts while thinking, nods while speaking, the eyes follow you, and the face shows zrb's emotion.

The shipped clips, by pool:

The page looks a clip up by name, so zrb-face action Greeting plays actions/Greeting.vrma; zrb-face actions lists everything with its length.

Where to get more VRMA:

• VRoid's official 7-motion pack (show full body, greeting, peace sign, shoot, spin, model pose, squat), free on BOOTH. Free to use, commercial use with the credit "Animation credits to pixiv Inc.'s VRoid Project", but its terms forbid redistributing the files, so keep them out of git. E.g. cp VRMA_02.vrma ~/.config/zrb-face/anim/actions/Greeting.vrma.
• Dances: motion capture through bvh2vrma, e.g. the AIST++ dance dataset (CC BY 4.0, credit needed) or the CMU library (free). Mixamo's dances can be converted too, but its terms forbid sharing the files, so keep them in your own folder.
• Your own: bvh2vrma (VRM Consortium, MIT) converts BVH motion capture to VRMA, and VRoid Studio exports poses and animations as VRMA.

The desktop app

Each release has a zip per platform:

• Windows (zrb-face-win32-x64.zip): unzip into %LOCALAPPDATA%\zrb-face\, so zrb-face finds zrb-face.exe there from WSL.
• macOS (zrb-face-darwin-arm64.zip for Apple silicon, -x64 for Intel): move zrb-face.app to /Applications. The app isn't signed, so clear macOS's block once with xattr -dr com.apple.quarantine /Applications/zrb-face.app.
• Linux (zrb-face-linux-x64.zip): unzip anywhere and set ZRB_FACE_APP to its zrb-face program. On Wayland (GNOME), a window can't keep itself on top, so pin it by hand.

To build it yourself:

cd app
npm install
npm start                                   # run from source
npm run pack:win                            # or pack:mac / pack:linux -> dist/
cp -r dist/zrb-face-win32-x64 /mnt/c/Users/$USER/AppData/Local/zrb-face

If GitHub downloads are slow, prefix the build with ELECTRON_MIRROR=https://npmmirror.com/mirrors/electron/, then check the zip in ~/.cache/electron against GitHub's SHASUMS256.txt.

Test

python3 test_face.py                        # with a Python that has zrb installed

It checks the search paths, the event relay, the remembered model choice, the music state and watcher (media-control and AppleScript), and that every shipped .vrma is real glTF with the humanoid bones and valid channel references.

CI (.github/workflows/test.yml) runs it with a pyflakes lint on every push to main and every pull request: Python 3.11–3.14 on Linux, and 3.13 on Windows and macOS. On pull requests from a branch of this repository (not a fork), code-review.yml also has zrb review the diff and post the report as a comment (zrb submit-comment, defined in the root zrb_init.py); it needs the OPENAI_API_KEY secret (and optionally ZRB_LLM_MODEL), and skips itself without it.

Releasing

Bump the version in pyproject.toml and app/package.json, then push a tag: git tag 0.5.0 && git push --tags. .github/workflows/release.yml publishes the package to PyPI and attaches the app zips to the GitHub release. PyPI publishing is trusted publishing, so first add this repository, the workflow release.yml and the environment pypi as a trusted publisher of zrb-face on pypi.org.

Credits

The models used during development, not included: ram and rem made in mexico (Re:Zero fan models) by pon3mx on VRoid Hub; search for them there. Ram: use, changes, commercial use and redistribution allowed, no credit needed. Rem: the same but no commercial use. Both have mouth, blink and emotion expressions. They are fan models of the Re:Zero characters, so keep them on your own machine.