Monkey see, monkey do

msmd lets you draw a rectangle over text on a macOS screen, recognizes that region with Apple's built-in Vision framework, and emits the recognized characters as keyboard events. An amber border remains around the capture region while OCR and typing are active.

The first release targets macOS. Capture, OCR, and keyboard output are separate interfaces so Linux and Windows adapters can be added without changing the text tracking or command-line flow.

Use this only on sites and accounts where automation is permitted. Automated results are not meaningful typing scores and may violate a site's rules.

Install with uv

Install uv and make sure your Python build has Tk support. Then let uv create and synchronize the managed project environment:

uv sync

There is no environment activation step. Run commands through uv run so uv uses the locked project environment.

On first use, macOS will ask for two permissions for your terminal or Python:

• Screen & System Audio Recording, so the selected pixels can be captured.
• Accessibility, so keyboard events can be sent to the browser.

Enable them under System Settings → Privacy & Security, then quit and relaunch the terminal if macOS does not apply the permission immediately.

Run

Start with a dry run to verify OCR without typing:

uv run msmd --dry-run

Then run a single capture:

uv run msmd

The interaction is:

1. Drag a rectangle tightly around the typing-test text.
2. Press Enter to confirm it, or Esc to cancel.
3. During the three-second countdown, click the browser's typing area.
4. Keep the target field focused while the recognized characters are emitted.

For a test whose viewport changes or scrolls, use experimental continuous mode:

uv run msmd --continuous --typing-delay 0.01

Stop continuous mode with Ctrl+C in the launching terminal. The tracker uses exact overlap between consecutive OCR frames and waits for two matching frames before accepting a completely replaced viewport. This reduces duplicate typing, but OCR mistakes or animated layouts can still produce incorrect input.

For a typing test with a fixed time limit, --duration implies continuous mode and stops automatically after the requested number of seconds. The timer starts after the focus countdown:

uv run msmd --duration 30
uv run msmd --duration 60 --typing-delay 0.01

Useful options:

--countdown SECONDS       Time available to focus the browser
--typing-delay SECONDS    Delay after each generated character
--scan-interval SECONDS   Delay between continuous OCR frames
--duration SECONDS        Stop continuous OCR after this many seconds
--language LANGUAGE       Vision language, such as en-US or de-DE

Current limitations

• Region selection currently targets the primary display.
• OCR quality depends on contrast, font size, animation, and the selected bounds.
• Unicode events work for ordinary typing-test text, but a site that requires physical key codes may need a keyboard-layout-aware writer later.
• Continuous mode is intentionally conservative but cannot guarantee that noisy OCR will never retype text.

Development

uv sync
uv run pytest

Platform-neutral behavior lives in core.py; macOS-specific screenshot, Vision, and keyboard implementations live in platforms/macos.py. A future Linux adapter can use X11/Wayland capture plus Tesseract, while Windows can use Windows Graphics Capture plus Windows OCR without changing the rest of the application.