> ## Documentation Index
> Fetch the complete documentation index at: https://docs.xynthis.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Computer use

> The agent can see your screen, click, type, open apps, and drive a browser on macOS.

On macOS, the agent controls the machine directly through CoreGraphics: it moves the mouse, clicks, types, presses keys, and takes screenshots to check the result. Ask it to "open System Settings and turn off Night Shift" or "reply to the Gmail thread from Sam" and it works the screen the way you would: look, act, look again.

Computer use is macOS-only. On Windows and Linux the CLI and the daemon run, but there are no screen-control tools.

## What the agent can do

**See the screen.** The `see_screen` tool captures a screenshot (saved under `~/.xynthis/screens/`), the frontmost app name, the focused window title, and an accessibility dump of the window's interactive elements: buttons, text fields, labels. Text-only models reason from the element labels; multimodal models also read the screenshot.

**Read pixels with local OCR.** When an app doesn't expose accessibility data properly (Gmail, Figma, most web apps), the `screen_ocr` tool runs Apple's Vision framework on the screenshot. It is on-device: no API key, no network, and it finishes in roughly 100-300 ms for a full Retina capture. Every recognized text element comes back with exact pixel coordinates, so the agent can click what it just read. Pass a `region` (`{x, y, w, h}`) to OCR only part of the screen, which is useful for re-checking one dialog without paying for a full-screen pass.

**Click and type.** `mouse_move`, `mouse_click`, `keyboard_type`, and `keyboard_key` post real input events. `click_element` clicks a UI element by its accessibility label ("click the button named Play"). The tool resolves the element's actual position, so the model never guesses coordinates. `open_app` launches applications by name.

**Drive the browser.** `safari_open` opens URLs in Safari. For email, `browser_email_automation` composes a Gmail message in one call (recipient, subject, and body filled at once via Gmail's compose URL) instead of a fragile screenshot-and-click loop. When you ask for a draft, it leaves the compose window open unsent so you can inspect it; it only presses send when you asked it to send.

## Permissions

Two macOS permissions gate all of this, and both are one-time grants in System Settings → Privacy & Security:

| Permission       | What it unlocks                                   |
| ---------------- | ------------------------------------------------- |
| Accessibility    | Mouse, keyboard, and reading window element trees |
| Screen Recording | Screenshots and OCR                               |

Xynthis checks permission state without triggering the system prompt (via `AXIsProcessTrustedWithOptions` and `CGPreflightScreenCaptureAccess`), so a mid-conversation tool call never pops a surprise consent dialog. If a permission is missing, the tool returns a message telling the agent exactly which pane to point you at, and the agent asks you to grant it. Tools degrade rather than fail: with Screen Recording missing, `see_screen` still returns the accessibility tree; with Accessibility missing, it still returns the screenshot.

## Safety behavior you will notice

A few deliberate guardrails shape how the agent behaves during computer use.

**It stops to decide after repeated looking.** If the agent takes three screen observations in a row without acting, the next observation is blocked and it is forced to decide: act, finish, or report that it is blocked. This prevents the "keep re-checking the screen" loop. A real action resets the counter, so the normal observe → act → observe rhythm is unaffected.

**One-shot actions run exactly once.** Sending an email is not retryable: a second send is a duplicate. Once a send has executed in a turn, any further send attempt in that turn is blocked and the agent must verify the outcome and report instead.

**Destructive tools ask first.** Arbitrary shell (`bash`), file writes, patches, Python execution, and subagent spawning default to a per-call confirmation prompt. In non-interactive contexts (schedulers, scripts) the prompt collapses to a denial rather than running silently. You can set per-tool policy in `~/.xynthis/tool_policy.json`, or bypass the gate entirely with `XYNTHIS_YOLO=1` for trusted automation.

Mouse and keyboard tools are intentionally *not* behind that confirmation prompt: the Accessibility grant is already the user-facing permission boundary. When the OS says yes, the policy says yes.

<Note>
  For decisions the agent genuinely cannot make (an ambiguous or destructive plan), it uses `ask_user`: the turn pauses, the question surfaces in the app or event stream, and you resume later with `xynthis checkpoints resume <id> "<answer>"`.
</Note>

## Trying it directly

Every tool the agent uses is also callable by hand, which is handy for checking permissions are wired up:

```bash theme={null}
xynthis tool see_screen --args '{}'
xynthis tool screen_ocr --args '{"region": {"x": 0, "y": 0, "w": 800, "h": 600}}'
```

If a permission is missing, the output includes the System Settings pane to open and what to do after granting it.