zrb-face

A companion for zrb: a 3D character in a small, see-through, always-on-top window. She idles naturally (breathing, shifting her weight, glancing around, stretching), shows what zrb is doing and how it feels, moves her mouth with zrb's voice, dances while you play music, gestures with your own animations, and can start a conversation herself.

Quick start

pipx install zrb-face          # or: pip install zrb-face, next to zrb; from a checkout: pip install -e .

# once, e.g. in ~/.zshrc: load the zrb side into every zrb chat
export ZRB_INIT_SCRIPTS="$ZRB_INIT_SCRIPTS:$(zrb-face init-path)"

mkdir -p ~/.config/zrb-face/models && cp my.vrm ~/.config/zrb-face/models/   # a model: see Changing the model
zrb-face my                    # (re)start the server, open a window with my.vrm
zrb chat

The window is a small desktop app; download it from the releases or build it (see The desktop app). Without it, zrb-face opens the page in your browser instead.

zrb-face opens the Windows build from WSL, the installed app on macOS, or Electron from app/ in a source checkout (ZRB_FACE_APP overrides the program). It restarts the server each time, so the server always runs the current code with your current settings; open windows reload by themselves. The server keeps running across chats; its output goes to zrb-face.log in the temp folder, and pkill -f "[z]rb_face/server.py" stops it. The order doesn't matter: the window waits for the server, reloads itself when the server restarts, and zrb skips sending while either is missing. If a model can't be loaded, the window says so in red and tries again every few seconds.

How it fits together

 zrb chat  ── POST /event ──▶  zrb-face server  ── page, models, animations, live events ──▶  window (face.html)
 + zrb_init.py                 (port 7777)                                                    in the app or a browser

zrb runs in WSL on Windows, but a window that stays on top has to be a Windows program. Windows can reach a server in WSL (not the other way around in WSL's default NAT mode), so the server runs next to zrb and the window connects to it.

Running by hand

What zrb-face does, per platform:

zrb-face serve &                                                                          # the server
/mnt/c/Users/$USER/AppData/Local/zrb-face/zrb-face.exe rem </dev/null &>/dev/null &!      # Windows, from WSL
open -n -a zrb-face --args rem                                                            # macOS, installed app
~/zrb-face/app/node_modules/.bin/electron ~/zrb-face/app rem                              # Linux, or any OS from source

The window takes a model name (rem or rem.vrm; without one, the first model alphabetically) and optionally another server URL (http://localhost:7779). Start the Windows build detached as above, or zsh stops it ("suspended (tty input)"). Instead of the app you can open http://localhost:7777/?model=rem in a browser: the same page, but not see-through or always on top.

Using the window

• Move: drag it. Close: Alt+F4 on Windows, Cmd+Q on macOS.
• Camera angle: click the window, then use the keys in Keyboard. The angle is remembered.

What she shows

• Voice: the mouth follows the loudness of the speech, in 40 ms steps, and she nods slightly while speaking. It needs a voice zrb plays itself (piper, openai, gemini); say and termux play the audio themselves, so zrb never sees it. The mouth starts slightly ahead of the voice, because zrb reads audio ahead of playing it; fixing that exactly needs a frame observer in zrb's PcmUtterance._fill.
• Face: she blinks every few seconds, her eyes follow you (the camera), and she smiles softly every 25–50 seconds and after finishing a task.
• Body: each state loops its own animations, falling back to the idles; see Animations. On SessionStart she plays the greeting action if there is one.
• Emotion: with zrb's emotion on (ZRB_LLM_EMOTION_ENABLED=true), zrb opens each reply with an emoji, and she shows it for 6 s with the matching VRM expression (happy, sad, angry, surprised, relaxed), and plays the action of the same name if there is one (actions/happy.vrma). The user's own emotion isn't shown: her reply answers it.
• Gestures: with animations in actions/, zrb gets a FaceAction tool and plays them itself, alongside its words (wave, nod, bow, …).
• Music: when something plays music (a browser, Spotify, …) she dances with the dance/ animations while idle; anything zrb does takes over, and she goes back to dancing after. The server watches the system's media status: Windows' media controls (also from WSL) or playerctl on Linux. Any playing media counts, a talk video too. On macOS, or to override it, use zrb-face music on / off; ZRB_FACE_MUSIC=off stops the watching.
• Talking first (off by default): with ZRB_FACE_TALK_FIRST=idle she speaks up after you've been quiet for a while during waking hours (8:00–22:00); with camera, when you sit back down after being away. She never interrupts a running turn or her own speech, waits a cooldown between such turns, and replies … (shown, not spoken) when there's nothing worth saying.

The mouth, the blink and the smile need the model's aa, blink and relaxed expressions. A model without them still nods while speaking.

Wiring it into zrb

zrb_face/zrb_init.py is all zrb needs; load it through ZRB_INIT_SCRIPTS as in Quick start (zrb-face init-path prints where it is). It owns zrb's speech (enable_speech), so other init scripts should only choose the voice (CFG.DEFAULT_LLM_SPEECH_BACKEND, CFG.DEFAULT_LLM_SPEECH_VOICE) and not call enable_speech themselves; whichever script calls it last would win.

Configuration

Environment variables

The zrb-side names follow zrb's CFG.ENV_PREFIX (ZRB by default). The server and the zrb-face command run without zrb, so their names are fixed.

zrb settings it depends on

Command line and URL

Keyboard (click the window first)

The camera angle is saved in the window's storage and comes back next time.

Files and folders

Models and animations are looked up in folders, in this order; the first file with a given name wins, so your own files override the shipped ones. zrb-face dirs shows the list.

In the code

Constants with no setting of their own; edit the file to change them.

Changing the model

Put any VRM file (VRM 0.x or 1.0) in ~/.config/zrb-face/models/ (or a folder in ZRB_FACE_MODELS) as <name>.vrm and open it by name: zrb-face <name>. Without a name, the window shows the first model alphabetically. No model ships with the package. Model files are kept out of git (*.vrm is ignored). Nothing in the code depends on the model: the camera fits it, and the pose and idle come from the animations, which three-vrm retargets onto any VRM, as VRoid Hub's playground does.

Where to get models:

• VRoid Studio: free, for Windows and macOS. Make your own and export it as VRM. You own it, and it has every expression.
• VRoid Hub: thousands of models. Only the ones marked downloadable can be downloaded, and each has its own conditions (avatar use, commercial use, redistribution, changes). Read them before you change or share the model.
• BOOTH: search for VRM and filter for free items. The terms are on each item's page, often only in Japanese.
• Open Source Avatars: free VRM avatars with direct download links. The license is set per collection (CC0, CC-BY, …), so check it.
• OpenGameArt: VRoid Studio CC0 models: public domain sample models from early VRoid Studio.

Fan models of existing characters (Cortana, Kusanagi, and others) are fine on your own desktop, but don't commit or share them.

Animations

Every <pool>/<name>.vrma (VRM Animation) in the animation folders loads when the page starts. The pool, the subfolder, says when it plays:

Any other pool loads too, and plays once something asks for it.

To add your own, drop the file in, e.g. ~/.config/zrb-face/anim/actions/wave.vrma, and restart with zrb-face (and zrb, so the FaceAction tool knows the new name). A file named like a shipped one replaces it: ~/.config/zrb-face/anim/idle/rest.vrma. Try an action with zrb-face action wave.

Within a pool, clips play in random order with crossfades, picked by group: the first part of the name (stretch-side belongs to stretch). A random group, then a random clip in it, so the stretches together come up as often as any single other idle. On top of whatever plays: the head tilts while thinking, nods while speaking, the eyes follow you, and the face shows zrb's emotion.

The idles and dances are ours (CC0), generated by python3 make_idle.py. The dances are 8-beat loops at 120 BPM: bounce (down on every beat, arms pumping), sway (side to side every two beats), step-touch (stepping side to side, hands up on the touch) and hands-up (arms high, waving). They don't follow the song's tempo yet. The idles:

Every clip also has breathing, uneven head drift, knees that flex a little, and fingers that relax and tighten. The hands are loosely curled, the arms hang a little away from the body, and the weight shifts move the hips over the standing leg. The motion follows VRoid's own VRMA clips (every joint moving, fingers included), common VTuber idle loops (breathing, sway, head pitch, roll and yaw), and how people actually stretch (the whole-body yawn-stretch; a side bend with the hips shifting the other way).

To change or add idles, edit CLIPS in make_idle.py (eased keyframes of poses) and run it again. Its docstring lists which way each joint turns. Poses can also move the hips (hipsMove) and set the blink and aa expressions; the page takes the larger of the clip's value and its own blinking and lip sync.

More free VRMA:

• VRoid's official 7-motion pack (show full body, greeting, peace sign, shoot, spin, model pose, squat), free on BOOTH. Free to use, commercial use with the credit "Animation credits to pixiv Inc.'s VRoid Project", but its terms forbid redistributing the files, so keep them out of git. E.g. cp VRMA_02.vrma ~/.config/zrb-face/anim/actions/greeting.vrma.
• Dances: motion capture through bvh2vrma, e.g. the AIST++ dance dataset (CC BY 4.0, credit needed) or the CMU library (free). Mixamo's dances can be converted too, but its terms forbid sharing the files, so keep them in your own folder.
• Your own: bvh2vrma (VRM Consortium, MIT) converts BVH motion capture to VRMA, and VRoid Studio exports poses and animations as VRMA.

The desktop app

Each release has a zip per platform:

• Windows (zrb-face-win32-x64.zip): unzip into %LOCALAPPDATA%\zrb-face\, so zrb-face finds zrb-face.exe there from WSL.
• macOS (zrb-face-darwin-arm64.zip for Apple silicon, -x64 for Intel): move zrb-face.app to /Applications. The app isn't signed, so clear macOS's block once with xattr -dr com.apple.quarantine /Applications/zrb-face.app.
• Linux (zrb-face-linux-x64.zip): unzip anywhere and set ZRB_FACE_APP to its zrb-face program. On Wayland (GNOME), a window can't keep itself on top, so pin it by hand.

To build it yourself:

cd app
npm install
npm start                                   # run from source
npm run pack:win                            # or pack:mac / pack:linux -> dist/
cp -r dist/zrb-face-win32-x64 /mnt/c/Users/$USER/AppData/Local/zrb-face

If GitHub downloads are slow, prefix the build with ELECTRON_MIRROR=https://npmmirror.com/mirrors/electron/, then check the zip in ~/.cache/electron against GitHub's SHASUMS256.txt.

Test

python3 test_face.py                        # with a Python that has zrb installed

Releasing

Bump the version in pyproject.toml and app/package.json, then push a tag: git tag 0.1.0 && git push --tags. .github/workflows/release.yml publishes the package to PyPI and attaches the app zips to the GitHub release. PyPI publishing is trusted publishing, so first add this repository, the workflow release.yml and the environment pypi as a trusted publisher of zrb-face on pypi.org.

Credits

zrb_face/anim/ and make_idle.py: ours, CC0.

The models used during development, not included: ram and rem made in mexico (Re:Zero fan models) by pon3mx on VRoid Hub; search for them there. Ram: use, changes, commercial use and redistribution allowed, no credit needed. Rem: the same but no commercial use. Both have mouth, blink and emotion expressions. They are fan models of the Re:Zero characters, so keep them on your own machine.