Computer Use

Computer use gives the voice assistant eyes and careful hands on your Mac. You ask out loud — “read me the selected paragraph,” “open the pricing page,” “fill my name into this form” — and the assistant inspects what’s on screen and acts on it. It runs in the background: it doesn’t move your pointer, raise windows, or steal focus, so you can keep working while it does.

Computer use is part of the Assistant layer, so it’s Pro and needs an agent-capable backend (Claude, Codex, Cursor, or OpenCode). The fully offline backends (Ollama, local MLX) have no tools and can’t drive your computer.

What it needs

• A Pro license and an agent backend selected in Dashboard → Assistant → Provider.
• Voice Mode connected as a tool provider — Dashboard → Assistant → Tools.
• macOS permissions:
  • Accessibility — required to read on-screen text and to deliver text or keystrokes into other apps.
  • Screen Recording — required only when the assistant needs a screenshot because a text read isn’t enough.

If a permission is missing, the assistant will tell you what to grant and where.

What it can do

• See what’s running — discover the apps it can target.
• Read — your current selection, the visible text of a window, a browser page’s text, or a browser’s URL.
• Screenshot — capture a window when text alone isn’t enough to understand the screen.
• Open — launch an app or URL (in the background by default).
• Click — activate a button, link, or control using a background-safe route.
• Fill text — draft or type text into an editable field. This is draft-only: it never submits, presses Enter, or clicks a button on its own.
• Press keys — send a real key or shortcut (Return, Tab, Escape, arrows, ⌘A, etc.) when you ask it to commit, move fields, or submit.
• Choose values — pick a dropdown, combobox, or menu value without taking focus.

How it stays safe

• Background by default. It uses accessibility actions and background-safe clicks — it doesn’t take over your pointer or bring apps to the front. If the only way left to do something would steal focus, it stops and asks you first.
• Draft, don’t send. Filling a field puts text in; it won’t submit. Pressing Enter or clicking “Send” only happens when you explicitly ask.
• Look before acting. Every action re-checks that the screen still matches what it last saw. If the window or page changed underneath it, it tells you what changed and confirms before continuing.
• One-time consent. The first time the assistant tries to act on your computer, Voice Mode shows a consent prompt. Approve once and future actions proceed; you can revoke it anytime in Dashboard → Assistant → Tools.
• Audited, not recorded. Activity is logged as metadata and byte counts — not the contents of what was read or typed.

Example voice flows

• “What does the selected text say?” → reads your selection back to you.
• “Open the GitHub releases page.” → opens it in the background; you switch when you’re ready.
• “Put my email address in the email field.” → drafts it into the field, leaves the form for you to review and submit.
• “Set the country dropdown to Canada.” → selects the value in place.
• “Take a screenshot of this window so you can see what I mean.” → captures it, then answers based on what’s there.

Pro tips

• Say what you want, not how. “Fill in the shipping form” is better than naming each field — the assistant inspects the form and finds the targets.
• It won’t submit unless you say so. If you actually want it sent, finish with “…and submit it” or “press Enter.” Otherwise you get a safe draft to review.
• Reach for backends that can act. Computer use needs Claude, Codex, Cursor, or OpenCode. If nothing happens, check you’re not on Ollama or local MLX.
• Grant Screen Recording before you need it. Most reads work with Accessibility alone, but a “look at this screen” request will stall waiting on the Screen Recording grant if it isn’t already given.