What the agent can do
See the screen. Thesee_screen tool captures a screenshot (saved under ~/.xynthis/screens/), the frontmost app name, the focused window title, and an accessibility dump of the window’s interactive elements: buttons, text fields, labels. Text-only models reason from the element labels; multimodal models also read the screenshot.
Read pixels with local OCR. When an app doesn’t expose accessibility data properly (Gmail, Figma, most web apps), the screen_ocr tool runs Apple’s Vision framework on the screenshot. It is on-device: no API key, no network, and it finishes in roughly 100-300 ms for a full Retina capture. Every recognized text element comes back with exact pixel coordinates, so the agent can click what it just read. Pass a region ({x, y, w, h}) to OCR only part of the screen, which is useful for re-checking one dialog without paying for a full-screen pass.
Click and type. mouse_move, mouse_click, keyboard_type, and keyboard_key post real input events. click_element clicks a UI element by its accessibility label (“click the button named Play”). The tool resolves the element’s actual position, so the model never guesses coordinates. open_app launches applications by name.
Drive the browser. safari_open opens URLs in Safari. For email, browser_email_automation composes a Gmail message in one call (recipient, subject, and body filled at once via Gmail’s compose URL) instead of a fragile screenshot-and-click loop. When you ask for a draft, it leaves the compose window open unsent so you can inspect it; it only presses send when you asked it to send.
Permissions
Two macOS permissions gate all of this, and both are one-time grants in System Settings → Privacy & Security:| Permission | What it unlocks |
|---|---|
| Accessibility | Mouse, keyboard, and reading window element trees |
| Screen Recording | Screenshots and OCR |
AXIsProcessTrustedWithOptions and CGPreflightScreenCaptureAccess), so a mid-conversation tool call never pops a surprise consent dialog. If a permission is missing, the tool returns a message telling the agent exactly which pane to point you at, and the agent asks you to grant it. Tools degrade rather than fail: with Screen Recording missing, see_screen still returns the accessibility tree; with Accessibility missing, it still returns the screenshot.
Safety behavior you will notice
A few deliberate guardrails shape how the agent behaves during computer use. It stops to decide after repeated looking. If the agent takes three screen observations in a row without acting, the next observation is blocked and it is forced to decide: act, finish, or report that it is blocked. This prevents the “keep re-checking the screen” loop. A real action resets the counter, so the normal observe → act → observe rhythm is unaffected. One-shot actions run exactly once. Sending an email is not retryable: a second send is a duplicate. Once a send has executed in a turn, any further send attempt in that turn is blocked and the agent must verify the outcome and report instead. Destructive tools ask first. Arbitrary shell (bash), file writes, patches, Python execution, and subagent spawning default to a per-call confirmation prompt. In non-interactive contexts (schedulers, scripts) the prompt collapses to a denial rather than running silently. You can set per-tool policy in ~/.xynthis/tool_policy.json, or bypass the gate entirely with XYNTHIS_YOLO=1 for trusted automation.
Mouse and keyboard tools are intentionally not behind that confirmation prompt: the Accessibility grant is already the user-facing permission boundary. When the OS says yes, the policy says yes.
For decisions the agent genuinely cannot make (an ambiguous or destructive plan), it uses
ask_user: the turn pauses, the question surfaces in the app or event stream, and you resume later with xynthis checkpoints resume <id> "<answer>".