humanoid-companion

A small humanoid robot you can talk to. It walks with a reinforcement-learned policy, waves, nods and dances while it talks, shows a face with lip sync, and speaks — and its brain is a language model you run yourself (Ollama, llama.cpp, vLLM, or any OpenAI-compatible API).

Today its body is a simulated ROBOTIS OP3 in MuJoCo, driven by exactly the control loop a real robot would run. The goal is to put the same code on a real 50 cm robot.

▶ Watch with sound (35 s). Left: the robot's face. Right: its body, in real time.

Every reply was generated live by a local model; no safety stops. The robot never says "owner": give it your name and it uses it.

What is inside

Quick start

Python 3.12 and uv. macOS or Linux; no GPU needed to run it.

git clone https://github.com/YauhenBichel/humanoid-companion && cd humanoid-companion
uv sync --frozen

# 1. Watch it walk (bundled policy, C MuJoCo, 10 s at 0.5 m/s)
uv run humanoid-sim --vx 0.5 --seconds 10 --record walk.mp4
bin/viewer --vx 0.4 --seconds 20          # live 3D viewer on macOS (mjpython)

# 2. Give it a brain: any OpenAI-compatible chat server, e.g. Ollama
ollama pull llama3.1:8b
export HUMANOID_LLM_MODEL=llama3.1:8b     # HUMANOID_LLM_BASE_URL defaults to Ollama, 127.0.0.1:11434/v1
export HUMANOID_USER_NAME=Alex            # optional: it calls you by name

# 3. Talk to it: face and body side by side in your browser; type, it answers, speaks and moves
uv run humanoid-talk --open

A voice (optional). The robot speaks through any server with the OpenAI speech API. Kokoro is small and natural:

docker run -p 8880:8880 ghcr.io/remsky/kokoro-fastapi-cpu    # HUMANOID_TTS_BASE_URL defaults to 127.0.0.1:8880/v1

For speaking to it (humanoid-talk --mic, needs uv sync --extra voice), run a Whisper server with the OpenAI transcription API, e.g. Speaches on port 8000. For the native Belarusian goodbye, run belarusian-tts (port 11810) and set HUMANOID_USER_NAME_BE to your name in Belarusian (e.g. Яўген).

Record a conversation as a split-screen video with sound:

uv run humanoid-talk --text "Hi! I'm back." --text "Can you dance for me?" \
    --text "Please walk forward a little and then turn left." --record demo/

Configuration

How it works

flowchart LR
    you([you: text or voice]) --> conv[conversation<br/>persona + JSON schema]
    conv -- "say, expression, gesture" --> face[face page<br/>lip sync, captions]
    conv -- "say" --> tts[speech server]
    tts --> face
    conv -- "action: walk + instruction" --> brain[walking planner<br/>schema, clamps, stop words]
    brain -- "vx, vy, yaw rate, duration" --> loop[50 Hz control loop<br/>safety stops]
    gest[gestures<br/>head + arm offsets] --> loop
    loop <--> io[RobotIO<br/>MuJoCo today, servos later]
    llm[(your LLM)] -.-> conv
    llm -.-> brain

One turn: the face shows thinking; the model replies; the speech and the walking plan are fetched at the same time; then the robot speaks while its body acts. Arm gestures come before a walk, never during it (measured: both arms up tips it over while walking).

The deployment path is written for a real robot: RobotIO is the only thing that knows about MuJoCo. The observation builder reproduces Playground's exactly (including its one-step action memory and sensor timing), which the parity tests pin.

Results (simulation)

Walking, bundled policy, 10 s at a commanded 0.5 m/s in C MuJoCo (not the MJX it was trained in, so this is also a sim-to-sim test): 5.27 m (0.53 m/s), 0.35 m sideways drift, no safety stop in 500 control steps. In training's own evaluation (128 episodes in MJX): mean 0.519 m/s, 0 falls.

Speed range (docs/speed-sweep.md): walks at 0.3–0.8 m/s forward (top ~0.65 m/s) and backwards at 0.3 m/s. Commands of 0.2 m/s and below make it stand still — a dead zone from Playground's reward shaping (mujoco_playground#361) — so the planner raises slow commands to the slowest real walk.

Gestures (docs/gesture-stability.md): all five pass standing; nod and look-around also pass while walking. A raised arm alone tipped the robot over until the other arm counter-balanced.

Training: 103 M environment steps in 96 minutes on a 16-core CPU (AMD Ryzen AI MAX+ 395, JAX split into 16 CPU devices: 2.8× faster than one). It also runs on that chip's GPU — see strix-halo-jax.

Train your own

uv run python -m humanoid_companion.train --name baseline --cpu-devices 16 --num-envs 4096 --num-timesteps 100000000
uv run python -m humanoid_companion.evaluate --run runs/baseline --no-video
uv run python -m humanoid_companion.export --run runs/baseline          # -> exports/baseline/policy.npz
uv run python -m humanoid_companion.speed_sweep --policy exports/baseline/policy.npz --name baseline
uv run humanoid-talk --open --policy exports/baseline/policy.npz --speed-sweep speed-sweep-baseline.json

--randomize trains the hardened environment (pushes, latency, encoder offsets, randomized friction, masses and gains) meant for a real robot.

Roadmap

1. Hardened policy (domain randomization) and its sim-to-sim numbers.
2. DynamixelRobotIO: the real servos and IMU, after a system identification.
3. A real OP3-class robot walking with this code; the face on a small screen on its head.
4. Seeing: a camera, and the model told what is in front of it.

Ideas and pull requests are welcome: CONTRIBUTING.md.

Contributors

Licence and disclaimer

Apache-2.0 (LICENSE, NOTICE). The OP3 model and the Op3Joystick environment come from MuJoCo Playground (Apache-2.0). Not affiliated with ROBOTIS or Google DeepMind.

This is research software for a simulated robot, provided as is. It is not a safety-rated robot controller: a real robot running it can fall, pinch or hit, so keep people clear and a hand on the power switch. It is not a medical or care device and must not be used as one.