qamule-scholar

qamule-scholar prepares the local SenseVoice model and starts QAMule Scholar's local Android projection workspace.

Platform

This package is supported only on macOS. Its build hook rejects non-macOS builds and its published wheel is tagged for macOS. It requires Python 3.11 or newer.

Install

Install it from the workspace during development:

uv sync --package qamule-scholar

Local model

Run the workspace with:

uv run qamule-scholar

The command checks the default ModelScope cache before starting the local web service. If the supported int8 SenseVoice ONNX snapshot is absent, it asks for a y or n confirmation, downloads poloniumrock/SenseVoiceSmallOnnx through ModelScope, validates model.int8.onnx and tokens.txt, then opens the browser workspace after the service is listening.

To start the local service without opening a browser, use -q:

uv run qamule-scholar -q

Android projection

Scholar includes a local Android screen projection workspace. It packages the device-side DEX JAR and exposes the workspace through a loopback web page. The default URL is http://127.0.0.1:26828/. Use --port to choose another local port:

uv run qamule-scholar --port 28080

By default, Scholar does not pass a device server port to uiautomator2. Declare one explicitly when needed:

uv run qamule-scholar --device-port 26831

The page lists ADB-authorized devices as direct-select cards; click a card to start projection. Use the + entry to add more device streams, and disconnect each stream from its own card. Device selection, connection, and disconnection are not available as CLI commands. The browser must support WebCodecs H.264 decoding, such as a current Chrome or Edge build.

While a device is projected, browser screen gestures and the current-device controls act directly on that device before recording starts. These direct actions do not create recording artifacts; the device card and its current controls remain locked until each action completes.

Recording

Select Start Recording in the sidebar to create a local session. Scholar starts microphone capture from the system-default input and transcribes detected speech with the local SenseVoice model. The most recent recognized sentences appear in the floating voice panel and are also written locally. If macOS denies microphone access, device recording continues and the panel reports the reason. Before the first recording, allow the application used to start Scholar in System Settings > Privacy & Security > Microphone. This is usually Terminal, Visual Studio Code, or the terminal application in which uv run qamule-scholar was started. Also select a usable input in System Settings > Sound > Input, then start a new recording.

For every device action while a session is active, Scholar saves a JPEG screenshot, raw UI hierarchy XML, and a simplified UI hierarchy text file before sending the action to the device. The text file follows u2cli dump-hierarchy output: invisible and zero-area nodes are omitted, and trivial single-child wrapper nodes are collapsed. It does not automatically capture a second device state after that action. Actions performed outside an active recording remain direct controls and create no artifacts.

Sessions are created below ./qamule-scholar from the directory where the command was run:

qamule-scholar/
    recording-<UTC timestamp>-<id>/
        recording.json
        transcript.txt
        step_000/
            screenshot.jpg
            hierarchy.xml
            hierarchy.txt
        step_001/
            screenshot.jpg
            hierarchy.xml
            hierarchy.txt

recording.json is the complete session manifest, including devices, recorded actions, and SenseVoice segments in one ordered events list. Events are ordered using internal capture and recognition times before serialization, so actions during or after a segment follow that voice event. transcript.txt contains the same recognized speech in a concise line-oriented format. Each action event records both hierarchy artifact paths as hierarchy and hierarchy_text.

Each recorded action identifies its source device with serial; device labels and the most recently observed window_size appear once in the top-level devices list. Tap and swipe coordinates remain normalized in the inclusive range 0 through 1.

Inspect a recorded tap

To find the smallest hierarchy node containing a recorded tap without opening the full XML, run:

uv run qamule-scholar tap-node qamule-scholar/recording-<id> --step 0

--step 0 selects step_000; omit it to inspect every recorded tap. The command converts the recorded normalized coordinates to device pixels and prints only concise u2cli dump-hierarchy-style node summaries, one per line. By default it returns up to five nodes. Use --depth N to choose a different limit: the smallest bounds match is first, then its ancestors. Trivial single-child wrapper nodes are collapsed like u2cli; if the root is reached first, it returns every available node:

uv run qamule-scholar tap-node qamule-scholar/recording-<id> --step 0 --depth 5

The bridge is bound to loopback. At startup it temporarily pushes agt-screen-server.jar, creates an ADB forward to its Android abstract socket, and runs the DEX with app_process. On shutdown it closes the local forward, stops the device-side process, and removes the temporary JAR. Screen capture uses adbutils against the local ADB server for this transport; device interactions remain the responsibility of Scholar's U2DeviceGateway, which delegates to the reusable u2cli service.