zrb-face

A companion for zrb: a 3D character in a small, see-through, always-on-top window. She idles naturally (breathing, shifting her weight, glancing around, stretching), shows what zrb is doing and how it feels, moves her mouth with zrb's voice, dances while you play music, gestures with your own animations, and can start a conversation herself.

Quick start

pipx install zrb-face          # or: pip install zrb-face, next to zrb; from a checkout: pip install -e .

# once, e.g. in ~/.zshrc: load the zrb side into every zrb chat
export ZRB_INIT_SCRIPTS="$ZRB_INIT_SCRIPTS:$(zrb-face init-path)"

mkdir -p ~/.config/zrb-face/models && cp my.vrm ~/.config/zrb-face/models/   # a model: see Changing the model
zrb-face my                    # (re)start the server, open a window with my.vrm
zrb chat

The window is a small desktop app; download it from the releases or build it (see The desktop app). Without it, zrb-face opens the page in your browser instead.

zrb-face opens the Windows build from WSL, the installed app on macOS, or Electron from app/ in a source checkout (ZRB_FACE_APP overrides the program). It restarts the server each time, so the server always runs the current code with your current settings; open windows reload by themselves. The server keeps running across chats; its output goes to zrb-face.log in the temp folder, and pkill -f "[z]rb_face/server.py" stops it. The order doesn't matter: the window waits for the server, reloads itself when the server restarts, and zrb skips sending while either is missing. If a model can't be loaded, the window says so in red and tries again every few seconds.

How it fits together

 zrb chat  ── POST /event ──▶  zrb-face server  ── page, models, animations, live events ──▶  window (face.html)
 + zrb_init.py                 (port 7777)                                                    in the app or a browser

scripts/ holds the tools that make the animations; they aren't part of the installed package (the wheel holds only zrb_face/), and the app never runs them. | zrb_init.py, .github/workflows/ | CI: the tests and lint on every push and pull request, and an LLM code review (the repository's own review and submit-comment zrb tasks, in the root zrb_init.py) posted on pull requests. |

zrb runs in WSL on Windows, but a window that stays on top has to be a Windows program. Windows can reach a server in WSL (not the other way around in WSL's default NAT mode), so the server runs next to zrb and the window connects to it.

Running by hand

What zrb-face does, per platform:

zrb-face serve &                                                                          # the server
/mnt/c/Users/$USER/AppData/Local/zrb-face/zrb-face.exe rem </dev/null &>/dev/null &!      # Windows, from WSL
open -n -a zrb-face --args rem                                                            # macOS, installed app
~/zrb-face/app/node_modules/.bin/electron ~/zrb-face/app rem                              # Linux, or any OS from source

The window takes a model name (rem or rem.vrm; without one, the first model alphabetically) and optionally another server URL (http://localhost:7779). Start the Windows build detached as above, or zsh stops it ("suspended (tty input)"). Instead of the app you can open http://localhost:7777/?model=rem in a browser: the same page, but not see-through or always on top.

Using the window

• Move: drag it. Close: Alt+F4 on Windows, Cmd+Q on macOS.
• Camera angle: click the window, then use the keys in Keyboard. The angle is remembered.

What she shows

• Voice: the mouth follows the loudness of the speech, in 40 ms steps — one value per step of audio, windowed across the chunk boundaries of the stream, so it ends when the voice does — and she nods slightly while speaking. It needs a voice zrb plays itself (piper, openai, gemini); say and termux play the audio themselves, so zrb never sees it. The mouth still starts slightly ahead of the voice, because zrb reads audio ahead of playing it; fixing that exactly needs a frame observer in zrb's PcmUtterance._fill.
• Face: she blinks every few seconds, and her eyes are on you (the camera) the way real eyes are: small jumps around your face, a glance away now and then (often with a blink), more of them while thinking, fewer while speaking; she smiles softly every 25–50 seconds and after finishing a task.
• Body: each state loops its own animations, falling back to the idles; see Animations. On SessionStart she plays the greeting action if there is one.
• Emotion: with zrb's emotion on (ZRB_LLM_EMOTION_ENABLED=true), zrb opens each reply with an emoji, and she shows it for 6 s with the matching VRM expression (happy, sad, angry, surprised, relaxed), and plays the action of the same name if there is one (actions/happy.vrma). The user's own emotion isn't shown: her reply answers it.
• Gestures: zrb gets her tools, all auto-approved: FaceActions lists her animations by pool with what each looks like and how long it takes, FaceAction plays one once alongside its words (an action by name: wave, nod, bow, …; any other clip by pool/name: thinking/idea), and FaceLook shows zrb her window as you see it (her pose and expression from your side, plus what she is doing), to check how a gesture comes across or when you ask how she looks. FaceModels lists the avatars (the .vrm models) and which one is showing, and FaceModel switches every open window to another one, when you ask her to look different. The actions and models are read on every call, so new ones work without restarting zrb. FaceLook sees only her own window, never your screen, and needs a model that reads images (or zrb's ZRB_LLM_MULTIMODAL_MODEL).
• Music: when something plays music (a browser, Spotify, …) she dances with the dance/ animations while idle; anything zrb does takes over, and she goes back to dancing after. The server watches the system's media status: Windows' media controls (also from WSL), playerctl on Linux, or on macOS media-control (brew install media-control; it also sees a browser tab playing) and else Spotify and Music by AppleScript (which asks permission to control them the first time). Any playing media counts, a talk video too. Use zrb-face music on / off to override it; ZRB_FACE_MUSIC=off stops the watching.
• Talking first (off by default): with ZRB_FACE_TALK_FIRST=idle she speaks up after you've been quiet for a while during waking hours (8:00–22:00); with camera, when you sit back down after being away. She never interrupts a running turn or her own speech, waits a cooldown between such turns, and replies … (shown, not spoken) when there's nothing worth saying.

The mouth, the blink and the smile need the model's aa, blink and relaxed expressions. A model without them still nods while speaking.

Wiring it into zrb

zrb_face/zrb_init.py is all zrb needs; load it through ZRB_INIT_SCRIPTS as in Quick start (zrb-face init-path prints where it is). It owns zrb's speech (enable_speech), so other init scripts should only choose the voice (CFG.DEFAULT_LLM_SPEECH_BACKEND, CFG.DEFAULT_LLM_SPEECH_VOICE) and not call enable_speech themselves; whichever script calls it last would win.

Configuration

Environment variables

The zrb-side names follow zrb's CFG.ENV_PREFIX (ZRB by default). The server and the zrb-face command run without zrb, so their names are fixed.

zrb settings it depends on

Command line and URL

Keyboard (click the window first)

The camera angle is saved in the window's storage and comes back next time.

Files and folders

Models and animations are looked up in folders, in this order; the first file with a given name wins, so your own files override the shipped ones. zrb-face dirs shows the list.

In the code

Constants with no setting of their own; edit the file to change them.

Changing the model

Put any VRM file (VRM 0.x or 1.0) in ~/.config/zrb-face/models/ (or a folder in ZRB_FACE_MODELS) as <name>.vrm and open it by name: zrb-face <name>. Without a name, the window shows the first model alphabetically. To switch while she runs, zrb-face model <name> (or ask zrb: its FaceModel tool) reloads every open window with the new model; the server remembers it for windows opened later, until it restarts. GET /model says which one is chosen and which ones the open windows show. No model ships with the package. Model files are kept out of git (*.vrm is ignored). Nothing in the code depends on the model: the camera fits it, and the pose and idle come from the animations, which three-vrm retargets onto any VRM, as VRoid Hub's playground does.

Where to get models:

• VRoid Studio: free, for Windows and macOS. Make your own and export it as VRM. You own it, and it has every expression.
• VRoid Hub: thousands of models. Only the ones marked downloadable can be downloaded, and each has its own conditions (avatar use, commercial use, redistribution, changes). Read them before you change or share the model.
• BOOTH: search for VRM and filter for free items. The terms are on each item's page, often only in Japanese.
• Open Source Avatars: free VRM avatars with direct download links. The license is set per collection (CC0, CC-BY, …), so check it.
• OpenGameArt: VRoid Studio CC0 models: public domain sample models from early VRoid Studio.

Fan models of existing characters (Cortana, Kusanagi, and others) are fine on your own desktop, but don't commit or share them.

Animations

Every <pool>/<name>.vrma (VRM Animation) in the animation folders loads when the page starts. The pool, the subfolder, says when it plays:

Any other pool loads too, and plays once something asks for it (zrb-face action <pool>/<name>, or zrb's FaceAction).

Pools nest: a subfolder's clips belong to the pool above it, as one group in the random pick. idle/calm/*.vrma loops with the idles, as often as any one other idle; actions/greetings/hi.vrma is the action greetings/hi. Overrides go by the whole path: your actions/greetings/hi.vrma replaces only that one. A clip is always played by its full path (thinking/idea, actions/greetings/hi); an action's short name (greetings/hi) works too, unless it is some other clip's full path (an actions/thinking/idea.vrma is actions/thinking/idea, as thinking/idea is the loop).

To add your own, drop the file in, e.g. ~/.config/zrb-face/anim/actions/wave.vrma, and restart with zrb-face so the window loads it; zrb's tools see it at once. The folders cascade per file: yours replaces a shipped clip of the same <pool>/<name>, adds any new one, and every shipped clip you don't have still plays. A description, which FaceActions shows the agent, is read from the file's animation extras.description (the generators write one; see NOTES in the scripts). A file named like a shipped one replaces it: ~/.config/zrb-face/anim/idle/rest.vrma. Try an action with zrb-face action wave.

Within a pool, clips play in random order with crossfades, picked by group: the first part of the name (stretch-side belongs to stretch). A random group, then a random clip in it, so the stretches together come up as often as any single other idle. On top of whatever plays: the head tilts while thinking, nods while speaking, the eyes follow you, and the face shows zrb's emotion.

The shipped clips come from two generators, and every one passes the same check before it is written (see below).

• Motion capture (python3 scripts/make_mocap.py): real people, from the CMU Graphics Lab Motion Capture Database, retargeted onto the VRM skeleton. That's where the natural weight, timing and follow-through come from: waiting, stretching, talking with the hands, waving, laughing, crying, dancing. The script downloads the takes it needs (as BVH, from una-dinosauria/cmu-mocap) into ~/.cache/zrb-face/cmu, maps each joint's world rotation onto the matching VRM bone (correcting for how differently the two skeletons stand at rest), calibrates the torso and legs on the actor's own neutral stance (the skeleton's zero pose isn't how a person stands, and mapped across as-is it leaves her leaning and her feet trailing), keeps the hips' sway but not where the actor walked to, stands her on the floor, faces her forward and turns her head to you on average (a conversation take's actor looks at the other person), keeps only part of the actors' side-to-side sway and head tilt (CALM: half in the loops, a third while speaking; acted big, it reads as tipsy), smooths the noisy hands, and picks each loop's start and end where the pose and the hips match best, blending the seam. CMU doesn't capture fingers, so the hands keep a relaxed curl.
• Keyframes (python3 scripts/make_idle.py): eased poses, for what no take has (a V sign, a thumbs-up, fidgeting, the beat-exact dances). The motion passes through the poses on a smooth curve rather than stopping at each, and each bone trails a little behind the one before it (the hand after the forearm after the upper arm), so a gesture travels out along the arm. Arms that must reach somewhere (the chin, a V sign by the face, hands clapping) are placed by where the wrist goes, not by angles: reach() solves the arm (two-bone IK), with the elbow and the palm pointing where asked.

Both plant the feet: wherever a foot stands (near the floor and not swinging, with some hysteresis so a hovering foot doesn't flicker), it is pinned where it stands and the legs are solved (IK) to keep it there, so when the hips shift, sway or dip, the knees bend instead of the feet sliding; a foot lets go only when it lifts, and no foot goes through the floor. The knee is a hinge that faces where the thigh's front does, so it never turns sideways or backward, and it eases into straight instead of snapping. That's also what gives the model's spring bones (hair, coat, chest) real motion to swing with: VRM physics runs on the model's own spring bones, every frame.

The idles:

The dances: macarena (mocap) and, at 120 BPM in 8 beats, bounce (down on every beat, arms pumping), sway (side to side every two beats), step-touch (stepping side to side, hands up on the touch), hands-up (arms high, waving) and twist (the hips and shoulders swinging one way and back). They don't follow the song's tempo yet.

The state loops, so something is already happening before you drop your own clip in:

The actions, each of which starts and ends standing so it can be cut into any loop. happy, sad, angry, surprised and relaxed are named after the VRM emotion presets, so zrb's own emotions drive the body as well as the face (the page looks the clip up by the emotion's name):

Every keyframed clip also has breathing (the chest and shoulders rising), uneven head drift, knees that flex a little, and fingers that relax and tighten.

To change or add keyframed clips, edit the pools in scripts/make_idle.py and run it again. Its docstring lists which way each joint turns; reach() places an arm by its wrist, and arm() builds one from angles. To add a motion-captured one, add a line to CLIPS in scripts/make_mocap.py (the CMU take, the seconds to use, the pool; the index lists every take) and run python3 scripts/make_mocap.py <name>. Your own BVH with CMU joint names works too: python3 scripts/make_mocap.py --bvh take.bvh --start 2 --end 9 --pool idle --name mine --out ~/.config/zrb-face/anim.

Before either writes a file, audit()/check() checks every frame: nothing sinks through the floor or through her body, no hand reaches further than an arm, no elbow folds past its stop (about 155°), no hand goes inside her head (touching her face is fine), her head never drops below her hips, and every clip ends where it starts. Poses can also move the hips (hipsMove) and set expressions (blink, aa, happy, angry, sad, relaxed, surprised). The actions wear a face (FACES in scripts/make_idle.py): a laugh closes her eyes in a smile and opens her mouth in bursts, a sad clip frowns, the friendly gestures smile. The page takes the larger of the clip's expression and its own blinking, lip sync, smile and emotion. A model without one of the expressions (many have no surprised) just skips it.

More free VRMA:

• VRoid's official 7-motion pack (show full body, greeting, peace sign, shoot, spin, model pose, squat), free on BOOTH. Free to use, commercial use with the credit "Animation credits to pixiv Inc.'s VRoid Project", but its terms forbid redistributing the files, so keep them out of git. E.g. cp VRMA_02.vrma ~/.config/zrb-face/anim/actions/greeting.vrma.
• Dances: motion capture through bvh2vrma, e.g. the AIST++ dance dataset (CC BY 4.0, credit needed) or the CMU library (free). Mixamo's dances can be converted too, but its terms forbid sharing the files, so keep them in your own folder.
• Your own: bvh2vrma (VRM Consortium, MIT) converts BVH motion capture to VRMA, and VRoid Studio exports poses and animations as VRMA.

The desktop app

Each release has a zip per platform:

• Windows (zrb-face-win32-x64.zip): unzip into %LOCALAPPDATA%\zrb-face\, so zrb-face finds zrb-face.exe there from WSL.
• macOS (zrb-face-darwin-arm64.zip for Apple silicon, -x64 for Intel): move zrb-face.app to /Applications. The app isn't signed, so clear macOS's block once with xattr -dr com.apple.quarantine /Applications/zrb-face.app.
• Linux (zrb-face-linux-x64.zip): unzip anywhere and set ZRB_FACE_APP to its zrb-face program. On Wayland (GNOME), a window can't keep itself on top, so pin it by hand.

To build it yourself:

cd app
npm install
npm start                                   # run from source
npm run pack:win                            # or pack:mac / pack:linux -> dist/
cp -r dist/zrb-face-win32-x64 /mnt/c/Users/$USER/AppData/Local/zrb-face

If GitHub downloads are slow, prefix the build with ELECTRON_MIRROR=https://npmmirror.com/mirrors/electron/, then check the zip in ~/.cache/electron against GitHub's SHASUMS256.txt.

Test

python3 test_face.py                        # with a Python that has zrb installed

It checks the search paths, the event relay, the remembered model choice, the music state and watcher (media-control and AppleScript), and every shipped .vrma: real glTF, the humanoid bones and channel references, and ends that meet, so the loops close. It also runs the same pose audit() the generator does before writing a file, regenerates every keyframed clip and fails if a shipped one is stale or a shipped file comes from neither generator, checks that standing feet don't slide, that reach() lands the wrist, and that a BVH take converts into a clip that loops, and fails if an emotion zrb can send has no action to play.

CI (.github/workflows/test.yml) runs it with a pyflakes lint on every push to main and every pull request: Python 3.11–3.14 on Linux, and 3.13 on Windows and macOS. On pull requests from a branch of this repository (not a fork), code-review.yml also has zrb review the diff and post the report as a comment (zrb submit-comment, defined in the root zrb_init.py); it needs the OPENAI_API_KEY secret (and optionally ZRB_LLM_MODEL), and skips itself without it.

Releasing

Bump the version in pyproject.toml and app/package.json, then push a tag: git tag 0.1.0 && git push --tags. .github/workflows/release.yml publishes the package to PyPI and attaches the app zips to the GitHub release. PyPI publishing is trusted publishing, so first add this repository, the workflow release.yml and the environment pypi as a trusted publisher of zrb-face on pypi.org.

Credits

scripts/make_idle.py, scripts/make_mocap.py and the keyframed clips in zrb_face/anim/: ours, CC0.

The motion-captured clips in zrb_face/anim/ (listed in scripts/make_mocap.py's CLIPS): the data used in this project was obtained from mocap.cs.cmu.edu. The database was created with funding from NSF EIA-0196217. CMU allows any use, including in products, except reselling the data itself.

The models used during development, not included: ram and rem made in mexico (Re:Zero fan models) by pon3mx on VRoid Hub; search for them there. Ram: use, changes, commercial use and redistribution allowed, no credit needed. Rem: the same but no commercial use. Both have mouth, blink and emotion expressions. They are fan models of the Re:Zero characters, so keep them on your own machine.