ChartRemotely — agent

Remote control of desktop charting applications on macOS. Say a company name into any Apple device and the chart on a wall monitor changes — including a monitor in another building.

This repository is the agent: the piece that runs on the Mac driving the display. It is deliberately small and readable, because installing it means granting Accessibility permission to software that types into a trading application. You should be able to audit that decision in a few minutes.

What it does, precisely

thinkorswim has no API, no URL scheme, and no automation surface. The agent drives it the way a screen reader would — through the macOS Accessibility API and synthetic input events. It is the same category of software as AutoHotkey or Keyboard Maestro, and it never touches a broker API.

What it cannot do

The command vocabulary is closed, and it is one short file: vocab.py.

There is no command that opens an order ticket, submits a trade, moves money, or reads account balances. Adding one would mean editing that file, in public, in this repository.

What a snapshot shows

Only the pane that holds the chart. The top bar with the account number, and the left-hand account, watchlist and news gadgets, never leave the machine. When the agent cannot tell the chart's pane apart from the whole window, it refuses rather than send the window.

The chart pane still shows whatever thinkorswim draws inside it. On a Mac that drives a display, set the chart up for being looked at from elsewhere:

• collapse the Active Trader ladder on the right (it shows P/L Open, P/L Day and Net Pos);
• turn off the position, open-order and trade-history markers in the chart's own settings.

Snapshots need macOS Screen Recording for the Python that runs the relay, as well as Accessibility.

Why the resolver exists

Dictation matches speech against the whole English language, so a security name arrives mangled: Palantir becomes "Volunteer", Qualcomm becomes "Callalon", Coin becomes "Kuehn". The resolver repairs what it can against the SEC company registry — exact ticker, exact name, prefix, word-subset, name-coverage, consonant skeleton, then fuzzy — and refuses anything that does not convince, rather than charting a plausible wrong answer.

Every case in tests/test_resolve.py is a real transcript or a real failure. The suite asserts what must resolve and what must be refused; loosening a threshold to fix a miss usually breaks a refusal.

Commands

chartremotely show palantir --scale scalp   put a security on the chart
chartremotely show "john deere"             spoken names resolve to tickers
chartremotely scale half                    change the aggregation
chartremotely read                          report symbol and scale
chartremotely scales                        list the time frames on offer
chartremotely studies --row-height 2.5      rebuild the study set
chartremotely learn                         rediscover the chart's controls
chartremotely setup                         pair this Mac and make its voice Shortcut
chartremotely permissions                   report Accessibility and Screen Recording
chartremotely serve                         run the local listener
chartremotely pair                          adopt this display to an operator
chartremotely relay                         hold a connection open for the operator
chartremotely doctor                        check everything the agent needs

serve and relay are two different transports and most setups want both. serve listens on the local network, which is what an Apple Shortcut on the same tailnet talks to. relay dials out to the operator and holds the connection open, which is the only way a command from the other side of the world reaches this machine - no inbound port, no router to open, no certificate to keep alive. Pairing alone does not start it: a display that is paired but has no relay running reports connected: false and silently ignores everything sent to it.

Voice

"Hey Siri, ChartRemotely", then one sentence:

"Palantir, half, on mac mini." · "Shopify thirty minutes." · "Nvidia daily." · "GE hourly on the studio."

Say the company, the scale, and - if it is not this Mac - "on" the display. The Shortcut records for about six seconds (tap to finish sooner where your device offers it) and sends the recording to the Mac that made it. That Mac hears it with Whisper, running locally on its own GPU (mlx-whisper, Apple silicon only) - nothing is sent to Apple or anyone else to transcribe - primed with the words it expects: the ten scale mnemonics, bar phrases like "thirty minutes", the companies you chart lately and the names of your displays. The recording is deleted as soon as it is heard; only the text is logged.

If something is missing or unclear you are asked, and the Shortcut listens again (up to three times), keeping what it already understood: "I heard Nvidia but no scale. Say the scale, like half or daily. I'm listening." - "Daily." Or "Did you mean PS (Pershing Square) or MSGS (Madison Square Garden)? Say the one you meant. I'm listening."

First run. chartremotely setup fetches the speech model, whisper-large-v3-turbo (about 1.6 GB, once, from Hugging Face). The listener loads it when it starts, so a sentence takes about a second to hear. Setup also learns your display names (the free chart_agent_status, asked at most every four hours) so "on the studio" is understood.

One Shortcut: talk, or type when it asks. When the recording has nothing in it - no microphone, silence, noise, or a Mac that cannot hear yet

• "ChartRemotely" shows one text box instead: Type your request, e.g. Palantir half on mac mini. Type the same sentence and press Return; it is understood the same way and costs the same. If something is missing, the box comes back with the question ("I heard Nvidia but no scale. Type the scale, like half or daily."). On a Mac the Shortcut goes straight to the box: Record Audio waits forever on a Mac without a microphone, and a Mac has a keyboard.

The Shortcut talks to one Mac — the one that made it. That Mac looks up the company and the scale itself (free: a table, never the chart), then acts as an ordinary patron of the operator: it calls the priced chart_show_chart tool with your npub, your sign-in token and the display named at "Where?" (this Mac when blank). A voice change costs what chart_show_chart costs. Siri answers within about three seconds — "Chart PLTR at half scale sent to Mac-mini. Good luck." — and the chart changes a moment later through that display's relay, which also takes its picture. Names may be loose: for a display paired as "Mac mini", "mac-mini", "mini mac", "mini", "macm" and even "mack meeny" all work. A refusal that comes back quickly — a balance too low, a display that is unknown or not answering — is spoken instead, in the operator's words. "Read" calls chart_read_chart and speaks what the chart shows.

Signing in. The first chartremotely setup proves your npub by a Nostr DM: reply from your Nostr client, and choose a long cache_duration in the reply (e.g. 30 days, or unlimited) so voice keeps working — left alone it lasts 2 hours. The token the operator returns is kept in this Mac's login Keychain ("ChartRemotely voice sign-in"), never in the config or a log; your nsec is never asked for. When it expires you hear "Your ChartRemotely sign-in expired. Answer the DM on your phone to renew." — a fresh DM is already on its way, and once you answer it (within about half an hour) the next command works again.

What Siri heard. Dictation matches against all of English, so the Mac repairs what it hears against the three small vocabularies it actually needs: companies in the SEC list, the ten time frames, and your display names. A scale may be a mnemonic, a bar size ("thirty minutes", "an hour", "four hours", "a day") or one word that sounds like exactly one mnemonic ("have", "alf" are half). A company may be mis-heard ("volunteer", "pal and tear" are Palantir; "shop if y" is Shopify); when two companies are too close to call you are asked - "Did you mean PS (Pershing Square) or MSGS (Madison Square Garden)?" - rather than shown the wrong chart, and the symbols you chart lately settle a near-tie. understand() parses a whole sentence the same way ("Palantir half on the mini") - that is what the ChartRemotely Shortcut uses on the sentence Whisper heard.

Each command handled is logged as one line — time, the command's first word, the "Where?" as said, what was heard for a company or scale lookup, and the reply if it was an ERR — plus one line when its tool call ends (the tool, "Where?" and any refusal) — to ~/Library/Logs/chartremotely-serve.log (voice) or chartremotely-relay.log (from the operator). Never a token, a secret or a picture.

Running it as a service

chartremotely setup installs both transports as launchd agents (com.chartremotely.serve and com.chartremotely.relay), logging to ~/Library/Logs/chartremotely-<command>.log. Two services rather than one, so a relay that loses the network cannot take the local listener down with it.

Time frames are spoken as mnemonics chosen for phonetic distance, because digits are the worst thing to say to a recogniser - "fifteen" and "fifty" collide, and so do "one" and "won":

Install

curl -fsSL https://chartremotely.tollbooth-dpyc.com/install.sh | sh

That installs uv if it is missing, installs this agent from PyPI (uv tool install --python 3.12 'chartremotely[macos,setup]'), and runs chartremotely setup, which:

1. proves who owns the display — your existing npub by answering a Nostr DM, a key already saved on this Mac, or a new key it makes and saves for you in the Keychain and, through Safari, in iCloud Passwords;
2. pairs this Mac with that identity, with no code to copy;
3. gives the Mac its tailnet address with tailscale serve;
4. installs the listener and the relay as launchd agents;
5. asks macOS for Accessibility and Screen Recording for the Python that runs them;
6. fetches the Whisper speech model (about 1.6 GB, once) and your display names;
7. makes this Mac's Shortcut, "ChartRemotely" (talk, or type when it asks), opens it for import, and removes a "ChartRemotely Ask" an older setup left in your library.

Run it again at any time: finished steps are skipped. chartremotely doctor reports anything still missing.

To pair by hand instead, run chartremotely pair and give the code it prints to your MCP client (chart_pair_agent), or type it on the site's Profile page.

Releases

Every merge to main that changes chartremotely/ or pyproject.toml is published to PyPI on its own, once ruff and the tests pass: the next patch after the newest vX.Y.Z tag, tagged and given a GitHub Release. To release a minor or major version instead, raise version in pyproject.toml in the PR; the merge publishes that number. Pushing a v* tag by hand still publishes exactly that version.

Nothing on a Mac updates itself. Upgrade an installed agent with uv tool upgrade chartremotely.

How it finds anything

Nothing is anchored to screen coordinates. thinkorswim layouts differ per user and change through the day, so a fixed region works on one machine and nowhere else.

Structural discovery is not available either: the chart's controls cannot be reached through AXChildren. The combo box holding the symbol reports the window as its parent, yet the window does not list it as a child - the link exists upward but not downward. A full walk of 2,500 elements finds the news panel's symbol box and never the chart's.

What the tree does expose reliably is the container layout. So the agent asks the tree where the panes are and hit-tests relative to each pane's own bounds, validating a candidate before acting on it. The aggregation control anchors on the labelled "Style" button beside it. Move the chart, resize it, switch to a grid - discovery follows, and takes about a second.

Status

Early. The thinkorswim adapter works; the interfaces are still moving.

License

Apache-2.0