serial-hid-kvm 0.1.0

KVM control via CH9329 USB HID emulator + HDMI capture with interactive preview and TCP API

serial-hid-kvm

Standalone KVM (Keyboard, Video, Mouse) controller using a CH9329 USB HID emulator over serial and a USB HDMI capture device. Provides an interactive preview window with full keyboard/mouse pass-through, a web-based remote viewer, and a JSON Lines API for programmatic automation.

What It Does

Turn two cheap USB devices (~$15 total) into a software KVM switch. Control any PC's keyboard and mouse over USB while watching its screen via HDMI capture — all from another PC, a script, or an AI agent.

flowchart LR
    subgraph Controller["Controller PC"]
        APP[serial-hid-kvm]
    end

    subgraph Target["Target PC"]
        HID["Keyboard/Mouse"]
        HDMI_OUT["HDMI out"]
    end

    APP -- USB --> CH340 -- Serial --> CH9329 -- USB --> HID
    HDMI_OUT -- HDMI --> CAP[HDMI Capture] -- USB --> APP

Features

• Interactive preview window — see the target screen in real-time, type and click as if you were sitting in front of it
• Full keyboard capture (Windows) — Win32 low-level hook captures all keys including Win, Alt+Tab when the preview window is focused (Ctrl+Alt+Del via Ctrl+Alt+End)
• Web-based remote viewer — browser-based remote desktop with keyboard/mouse over WebSocket (--web)
• API server (--api) — JSON Lines protocol over TCP socket for programmatic automation from any language; also available as an MCP server via mcp-serial-hid-kvm
• Python client library — thread-safe client with auto-reconnect for scripting and AI agent integration
• Multiple concurrent clients — one KVM server, many API clients (AI agents, scripts, etc.)
• Headless mode — run without preview window; combine with --api and/or --web
• Multi-layout support — US, JP (JIS), UK, DE, FR keyboards for API text input (some special characters like umlauts, accented characters, and £ are not yet supported in non-US/JP layouts)
• Audio streaming — HDMI audio playback in preview window and web viewer via --audio-device (optional)
• Auto-detection — finds CH340 serial adapter and HDMI capture device automatically

Use Cases

• AI-driven PC automation — let Claude or other AI agents control a PC via mcp-serial-hid-kvm
• Remote KVM over network — lightweight IP-KVM with commodity hardware
• Automated testing — drive real hardware UI tests via the API
• Headless server management — access BIOS/UEFI setup, OS installers, or machines without network
• Security testing — HID-level input that is indistinguishable from a real keyboard

Hardware Requirements

Device	Purpose	Typical Cost
CH9329 + CH340 cable	USB HID emulator via serial	~$5
USB HDMI capture dongle	Screen capture	~$10

The CH9329 appears as a standard USB keyboard/mouse to the target PC — no drivers needed on the target side.

Installation

Requires Python 3.10+.

# Ubuntu/Debian: venv package may need to be installed first
sudo apt install python3-venv

python3 -m venv .venv
source .venv/bin/activate
pip install -e .

# With audio support (for web viewer audio streaming)
pip install -e ".[audio]"

On Linux, add your user to the required groups and install dependencies:

sudo usermod -a -G dialout $USER  # Serial port access
sudo usermod -a -G video $USER    # Video capture access
# Log out and back in for group changes to take effect

sudo apt install xdotool  # Required for preview window focus detection

Quick Start

# Preview window only (default)
serial-hid-kvm

# Preview + API server
serial-hid-kvm --api

# Headless with API server
serial-hid-kvm --headless --api

# Headless with web viewer
serial-hid-kvm --headless --web

# Headless with both API + web
serial-hid-kvm --headless --api --web

# Web viewer accessible from LAN  ⚠️ USE WITH CAUTION — see Remote Access and Security
serial-hid-kvm --headless --web --web-host 0.0.0.0

# API server accessible from LAN  ⚠️ USE WITH CAUTION — see Remote Access and Security
serial-hid-kvm --headless --api --api-host 0.0.0.0

# Both API + web accessible from LAN  ⚠️ USE WITH CAUTION — see Remote Access and Security
serial-hid-kvm --headless --api --api-host 0.0.0.0 --web --web-host 0.0.0.0

# Custom settings
serial-hid-kvm --headless --api --api-port 9400 --web --web-port 8080

# Web viewer with audio (use list-devices to find audio device index)
serial-hid-kvm --headless --web --audio-device 3

# Japanese keyboard layout
serial-hid-kvm --target-layout jp106

# List capture and audio devices
serial-hid-kvm list-devices

# Use config file
serial-hid-kvm --config my.yaml

By default, only the preview window runs. API server (--api, default port 9329) and web viewer (--web, default port 9330) are opt-in. Both bind to 127.0.0.1 by default; use --api-host / --web-host to change.

Remote Access and Security

The API server and web viewer have no built-in authentication or TLS. By default they bind to 127.0.0.1 and are not exposed to the network. Binding to 0.0.0.0 exposes full keyboard/mouse control of the target to anyone who can reach the port. To access remotely, use an external tool rather than binding to 0.0.0.0 directly:

• SSH tunnel — simplest, no extra software: ssh -L 9330:127.0.0.1:9330 kvm-host
• Tailscale / WireGuard — VPN-level protection without exposing ports
• Caddy / nginx — reverse proxy with TLS and authentication if needed

Configuration

Settings are resolved with priority: CLI args > environment variables > config file > defaults.

CLI Arguments

serial-hid-kvm [OPTIONS] [COMMAND]

Commands:
  list-devices              List capture devices and exit

Options:
  --headless                Run without preview window
  --log-file PATH           Also write logs to a file (in addition to stderr)
  --debug-keys              Print keycode debug output to console
  --show-cursor             Show mouse cursor on preview window
  -c, --config FILE         Path to YAML config file

  --api                     Enable API server (JSON Lines over TCP socket)
  --api-host ADDR           API server bind address (default: 127.0.0.1)
  -p, --api-port PORT       API server port (default: 9329)

  --web                     Enable web-based remote desktop viewer
  --web-host ADDR           Web viewer bind address (default: 127.0.0.1)
  --web-port PORT           Web viewer port (default: 9330)
  --web-fps FPS             Web viewer frame rate (default: 20)
  --web-quality Q           Web viewer JPEG quality 1-100 (default: 60)

  --audio-device DEV        Audio input device index or name for web viewer
                            (disabled by default; requires pip install .[audio])

  -s, --serial-port PORT    Serial port (e.g. COM3, /dev/ttyUSB0)
  --serial-baud BAUD        Serial baud rate (default: 9600)

  --screen-width PX         Target screen width (default: 1920)
  --screen-height PX        Target screen height (default: 1080)

  --capture-device DEV      Capture device index or path
  --capture-width PX        Capture resolution width (default: 1920)
  --capture-height PX       Capture resolution height (default: 1080)

  --target-layout NAME      Target keyboard layout: us104, jp106, uk105, de105, fr105
  --host-layout NAME        Host keyboard layout (default: auto-detect, 'none' to disable)
  --layouts-dir DIR         Directory for custom layout YAML files

Note: --target-layout affects character-to-keycode conversion in the API (type_text, send_key) and the preview window's character mapping. The web viewer forwards physical key positions via event.code and does not use this setting. --host-layout is used by the preview window on Linux to translate between host and target keyboard layouts (see Preview Window Keyboard Handling below).

Environment Variables

All use the SHKVM_ prefix:

Variable	Default	Description
SHKVM_SERIAL_PORT	auto-detect	Serial port (/dev/ttyUSB0, COM3, etc.)
SHKVM_SERIAL_BAUD	9600	Baud rate for CH9329
SHKVM_SCREEN_WIDTH	1920	Target screen resolution width
SHKVM_SCREEN_HEIGHT	1080	Target screen resolution height
SHKVM_CAPTURE_DEVICE	auto-detect	Capture device index or path
SHKVM_CAPTURE_WIDTH	1920	Capture resolution width
SHKVM_CAPTURE_HEIGHT	1080	Capture resolution height
SHKVM_TARGET_LAYOUT	us104	Target keyboard layout for API and preview
SHKVM_HOST_LAYOUT	auto	Host keyboard layout (auto, layout name, or none)
SHKVM_LAYOUTS_DIR	—	Directory for custom layout YAML files
SHKVM_API	false	Enable API server (1/true)
SHKVM_API_HOST	127.0.0.1	API server bind address
SHKVM_API_PORT	9329	API server port
SHKVM_WEB	false	Enable web viewer (1/true)
SHKVM_WEB_HOST	127.0.0.1	Web viewer bind address
SHKVM_WEB_PORT	9330	Web viewer port
SHKVM_WEB_FPS	20	Web viewer frame rate
SHKVM_WEB_QUALITY	60	Web viewer JPEG quality (1-100)
SHKVM_AUDIO_DEVICE	—	Audio input device index or name (web viewer)
SHKVM_DEBUG_KEYS	0	Enable keycode debug output (1/true)
SHKVM_SHOW_CURSOR	0	Show mouse cursor on preview window (1/true)
SHKVM_LOG_FILE	—	Also write logs to a file

Config File

YAML config file is searched in this order:

1. --config FILE CLI argument
2. SHKVM_CONFIG environment variable
3. ./serial-hid-kvm.yaml (current directory)
4. ~/.config/serial-hid-kvm/config.yaml (Linux) / %APPDATA%\serial-hid-kvm\config.yaml (Windows)

Example serial-hid-kvm.yaml:

serial_port: /dev/ttyUSB0
serial_baud: 9600
screen_width: 1920
screen_height: 1080
capture_device: /dev/video0
target_layout: jp106
host_layout: auto
api_enabled: false
api_host: 127.0.0.1
api_port: 9329
web_enabled: false
web_host: 127.0.0.1
web_port: 9330
web_fps: 20
web_quality: 60
audio_device: null
debug_keys: false
show_cursor: false

Preview Window

The interactive preview window shows the target screen in real-time with full keyboard and mouse pass-through.

Controls

• Keyboard: All keys forwarded to target when window is focused
• Mouse: Click, drag, and scroll forwarded with coordinate mapping; cursor hidden by default (--show-cursor to show)
• Ctrl+Alt+End: Send Ctrl+Alt+Del to target (like RDP/VNC)
• Ctrl+Alt+Q: Quit the preview (and the server)

On Windows, a low-level keyboard hook captures all keys (including Win key). On Linux, pynput is used (keys are not suppressed on the host). The mouse cursor is hidden by default because the target OS cursor is visible in the captured video. However, the video has inherent latency so the cursor in the video lags behind your actual mouse position. --show-cursor shows a local cursor that tracks your mouse instantly, reducing the feeling of input lag during precise operations.

Keyboard Handling

On Windows, the preview window captures physical scan codes and converts them directly to HID keycodes. Keyboard input works correctly without layout configuration.

On Linux, pynput delivers characters rather than scan codes. When the host and target keyboard layouts differ, characters must be reverse-mapped to the correct physical HID keycodes. This is handled automatically:

1. Detection: --host-layout auto (default) detects the host layout via gsettings (GNOME compositor) and setxkbmap (X11/Xwayland)
2. Reverse lookup: When host and target layouts differ, the preview builds a character→keycode map for the host layout. Incoming characters are looked up in the host map to recover the physical key position, then sent as HID keycodes
3. Keycode sending: The physical keycode is sent to the target with the currently held modifier bits

Wayland hybrid mode: On GNOME Wayland, pynput runs through Xwayland, which may have a different keymap from the compositor. For example, the compositor may be set to JP while Xwayland defaults to US. When this mismatch is detected:

• Unshifted characters match the compositor (display) layout → mapped via char_to_hid using the target layout
• Shifted characters match the Xwayland layout → reverse-looked up via the host character map

Use --debug-keys to see which path each key takes. Debug output labels:

• WAYLAND-REVERSE: Shifted character reverse-looked up via Xwayland host layout (Wayland hybrid)
• HOST REVERSE: Character reverse-looked up via host layout (pure X11)
• CHAR: Character mapped via char_to_hid using the target layout (fallback)
• CHAR UNMAPPED: Character not found in any layout map

Known limitation — layout-specific physical keys (e.g. JIS ろ/¥): Some keyboards have physical keys that don't exist on other layouts. For example, JIS keyboards have ろ (International1, HID 0x87) and ¥ (International3, HID 0x89), which produce \/_ and ¥/| respectively. On Xwayland with a US keymap, these keys are mapped to the same characters as standard US keys (_ = Shift+-, | = Shift+\), making them indistinguishable at the character level. The preview window sends the standard key position in these cases. The web viewer handles all keys correctly because it uses event.code (W3C physical key positions) which is layout-independent.

Web Viewer

The web viewer provides a browser-based remote desktop interface, similar to VNC but over HTTP + WebSocket. Enable it with --web:

serial-hid-kvm --headless --web
# → Web viewer listening on http://127.0.0.1:9330

Then open http://localhost:9330 in a browser. To allow access from other machines, use --web-host 0.0.0.0 (see Remote Access and Security for important caveats).

Features:

• Real-time JPEG video stream over WebSocket (binary frames, no base64 overhead)
• Full keyboard capture via event.code (physical key positions) — layout-independent and works correctly with all keys including JIS-specific keys (ろ, ¥)
• Mouse click, drag, scroll with coordinate normalisation to CH9329 absolute range
• Ctrl+Alt+Del button in toolbar
• Fullscreen mode
• FPS counter
• Auto-reconnect on WebSocket disconnect (2-second interval)
• Dark theme, responsive canvas with aspect ratio preservation
• Focus-loss detection: all keys released when canvas loses focus (no stuck keys)
• Audio streaming with Unmute/Mute button (when --audio-device is set)
• Runs alongside the API server and preview window simultaneously

As with the preview window, the mouse cursor is hidden by default. The Cursor button in the toolbar shows a local cursor that tracks your mouse instantly, reducing the feeling of input lag caused by video stream latency.

Known limitation: The web viewer supports up to 6 simultaneous key presses (the HID protocol maximum), but browser-level key event quirks may affect complex multi-key scenarios. For reliable text input, use the API's type_text method.

Audio

HDMI audio can be streamed to both the preview window and the web viewer. Audio is disabled by default — specify an audio input device to enable it:

# List available audio devices
serial-hid-kvm list-devices

# Preview window with audio
serial-hid-kvm --audio-device 3

# Web viewer with audio
serial-hid-kvm --headless --web --audio-device 3

Requires the audio extra: pip install serial-hid-kvm[audio] (installs sounddevice). The audio device is typically the HDMI capture dongle's audio input. The preview window plays audio immediately; the web viewer requires clicking the Unmute button (browser autoplay policy).

API Server

JSON Lines protocol over TCP socket. Enable with --api. One JSON object per line, newline-delimited. An MCP server wrapper is available at mcp-serial-hid-kvm for integration with Claude and other AI agents.

Request / Response

→  {"id":"1", "method":"type_text", "params":{"text":"hello\n"}}
←  {"id":"1", "ok":true, "result":{"chars_typed":6}}

→  {"id":"2", "method":"ping", "params":{}}
←  {"id":"2", "ok":true, "result":{"pong":true}}

Available Methods

Method	Parameters	Description
type_text	text, char_delay_ms?	Type text with {tag} support
send_key	key, modifiers?	Single key press (e.g., enter, f5)
send_key_sequence	steps, default_delay_ms?	Multiple key steps with delays
mouse_move	x, y, relative?	Move mouse cursor (preserves button state during drag)
mouse_click	button?, x?, y?	Click at position
mouse_down	button?, x?, y?	Press and hold button (for drag start)
mouse_up	button?, x?, y?	Release button (for drag end)
mouse_scroll	amount	Scroll wheel (-127 to 127)
capture_frame	quality?	Capture screen as base64 JPEG
get_device_info	—	Serial port, capture device, config info
list_capture_devices	—	List video devices
set_capture_device	device	Switch capture device
set_capture_resolution	width, height	Change capture resolution
ping	—	Health check

Text Tags

type_text supports inline tags for special keys:

ls -la{enter}             # type command and press Enter
{ctrl+c}                  # Ctrl+C
{alt+f4}                  # Alt+F4
Hello{{World}}            # literal braces: Hello{World}
path{0x87}file            # raw HID keycode (0x87 = JIS backslash)

Quick Test

# Start with API enabled
serial-hid-kvm --headless --api

# Using netcat
echo '{"id":"1","method":"ping","params":{}}' | nc localhost 9329

# Using Python
python3 -c "
from serial_hid_kvm.client import KvmClient
c = KvmClient(); c.connect()
print(c.ping())
c.type_text('Hello from Python!')
"

Python Client Library

from serial_hid_kvm.client import KvmClient

client = KvmClient("127.0.0.1", 9329)
client.connect()

# Keyboard
client.type_text("echo hello{enter}")
client.send_key("f5")
client.send_key("c", modifiers=["ctrl"])

# Mouse
client.mouse_move(960, 540)
client.mouse_click("left", x=100, y=200)
client.mouse_scroll(-3)

# Drag (select text, drag-and-drop, resize windows, etc.)
client.mouse_down("left", x=100, y=200)
client.mouse_move(500, 200)
client.mouse_up("left", x=500, y=200)

# Screen capture
jpeg_bytes, width, height = client.capture_frame_jpeg()

# Device info
info = client.get_device_info()

The client is thread-safe and auto-reconnects on connection loss.

License

MIT