tutti/docs/conventions/troubleshooting/computer-use.md

Computer Use Troubleshooting

Use this guide for recurring cua-driver and tutti computer failures. Keep stable CLI and authorization contracts in Tutti CLI Contract; keep symptom-driven diagnosis and recovery here.

Computer use keeps reopening setup after installation or authorization

Symptom

The Desktop settings row reports that computer use is installed and authorized, but selecting /computer opens setup again. On macOS, the first visit to a Privacy & Security pane may not list CuaDriver, or Screen Recording may remain granted but not effective until the driver is restarted.

Root cause and fix

Desktop permission status and Agent Composer options refresh independently. A fresh host probe may therefore be authorized while an older Composer snapshot still reports computer use as unavailable. Treat the fresh installed and authorized host status as sufficient for the Composer interaction; the daemon still performs its authoritative readiness clamp when the session launches.

On macOS, start the user-initiated permission grant before opening System Settings so TCC has registered CuaDriver on the first visit. After both Accessibility and Screen Recording are granted but capture is not yet effective, restart CuaDriver through its app bundle and then read status again; plain polling cannot clear the process-cached permission state.

Validation

• An uninstalled driver still routes /computer to setup.
• An installed and authorized driver inserts and submits /computer without reopening setup, even if Composer options were loaded before authorization.
• CuaDriver appears on the first user-initiated visit to each macOS permission pane.
• Returning from macOS System Settings reconciles granted permissions and capture readiness without restarting Tutti.
• Windows continues to use doctor --json; macOS-only reconciliation never runs on Windows.

CuaDriver permissions are granted but every input tool is denied

Symptom

macOS Accessibility and Screen Recording are both granted. Window listing, screenshots, and the AX tree work, but native click, move_cursor, press_key, type_text, and scroll appear with allowed: false. Their denial reason is tool has denied or unknown capabilities: input.delivery_mode.

Root cause and fix

CuaDriver 0.20 adds input.delivery_mode to tools that support its explicit background/foreground input-delivery ladder. Tutti's native-tool policy is fail-closed and previously did not recognize that capability, so it rejected the complete tool whenever the otherwise authorized input action advertised the new metadata. The same catalog also declares input.pointer.move on move_cursor; that capability must accompany the existing agent_cursor.move authorization. This is not an observation-only computer session and is not a stale macOS TCC grant.

Authorize input.delivery_mode and input.pointer.move alongside the existing keyboard, pointer, and window.activate capabilities. Delivery mode does not grant a new class of input by itself; it selects delivery posture for an input tool already allowed by Tutti. Keep unknown capabilities denied.

Validation

• With CuaDriver 0.20, tutti computer tool list --json marks the input tools allowed while a tool carrying an unrelated future capability remains denied.
• Background delivery remains the default; foreground delivery is explicit and uses the live native schema.
• Restart the updated tuttid process so it loads the new policy. Existing Agent conversations do not need to be recreated because native tools are listed and authorized live on every call.

Screen Recording is enabled but Tutti says it has not taken effect

Symptom

macOS System Settings shows CuaDriver enabled for both Accessibility and Screen Recording, but Tutti reports Screen Recording as authorized but not effective. cua-driver permissions status --json may report both TCC grants as true while returning screen_recording_capturable: null and direct_capture_status: "not_checked".

Root cause and fix

CuaDriver 0.20 keeps its read-only permission status content-free and does not run the prompt-capable direct-capture probe. A nullable screen_recording_capturable therefore means the probe was not run, not that capture failed. Tutti treats the two states separately: an explicit false remains not ready, while null is accepted when Accessibility and Screen Recording are both granted. Actual computer actions still cross CuaDriver's own permission enforcement boundary.

Do not repeatedly toggle the macOS permission or reinstall CuaDriver solely because the nullable probe field is present. Use the exact daemon-attributed status payload to distinguish an unprobed result from an explicit capture failure.

Validation

• cua-driver permissions status --json attributes the response to com.trycua.driver and reports Accessibility and Screen Recording as true.
• A nullable capture probe no longer keeps Tutti in the authorization wizard.
• An explicit screen_recording_capturable: false still shows the recovery path and blocks computer-use readiness.
• The daemon and desktop apply the same nullable-versus-false interpretation.

A computer click reports success but the UI does not change

Symptom

A stable or native computer click reports that it was posted, but a fresh screenshot shows no UI change. This commonly affects Electron-based apps. The agent-cursor overlay may also appear offset from the intended point after a display-configuration change.

Quick checks

1. Capture a fresh target-window screenshot with the same explicit pid and window-id used for the action.
2. Treat a posted-click response as dispatch confirmation only. Do not use the agent-cursor overlay as evidence that the event landed at the displayed point.
3. Inspect the screenshot structured content for the target's element_token. If no suitable element exists, confirm that pixel coordinates came from the latest screenshot of the same window.
4. Use computer tool describe --name click --json before native escalation so the live cua-driver schema remains the source of truth.

Root cause

Pixel clicks use background CGEvent delivery. cua-driver can confirm that the event was dispatched, but it cannot read back the UI effect. Focus-sensitive or Electron surfaces may silently discard a background synthetic click. The agent-cursor overlay is rendered through a separate visual channel and may have a stale display offset, so its apparent position does not validate event delivery.

Fix

1. Prefer the native click element-token path from the latest screenshot when an actionable element is available.
2. Verify the result with another fresh screenshot.
3. If a background pixel click had no effect, do not repeat the same click. Follow the live native schema and retry once with delivery_mode: "foreground", which briefly fronts the target window.
4. If foreground delivery still has no visible effect, re-snapshot and re-resolve the target instead of reusing stale coordinates or tokens.

Validation

• The post-action screenshot shows the expected state change.
• The action used the same pid and window-id as its source screenshot.
• Element-token retries use a token from the latest snapshot; stale-token errors trigger a new snapshot.
• Pixel escalation follows background, verify, foreground, verify rather than repeated unverified clicks.

References

• Tutti CLI Contract: Computer command surfaces
• Injected Computer Use skill

A Windows native tool call rejects valid-looking JSON

Symptom

computer tool call reports that arguments-json is not a JSON object, or a nested powershell.exe -Command invocation changes quotes and backslashes.

Root cause and fix

JSON embedded in a command string crosses both the outer launcher and Windows shell parsing boundaries. Serialize the object with the shell's JSON library and pipe it through --arguments-json -; do not launch a second PowerShell process around the Tutti CLI command. --arguments-json remains supported for simple existing scripts, but the two input sources cannot be combined. When using Windows PowerShell 5.1, temporarily set $OutputEncoding to [Text.UTF8Encoding]::new($false) around the pipeline and restore it in a finally block; its default ASCII encoding cannot preserve non-ASCII JSON.

If the driver returns isError=true together with Windows success status 0x00000000, treat it as an inconsistent driver result rather than successful delivery. If it rejects automation of its own authorization process, choose a different application target; never automate the authorization UI.

Validation

• Stdin JSON round-trips nested values, backslashes, spaces, and non-ASCII text.
• Every state-changing action is followed by a fresh screenshot.
• An inconsistent success-code error and a protected authorization target have explicit diagnostic messages.

A Windows CUA doctor check is slow or reports UIA fallback

Symptom

Windows computer-use status takes several seconds, or the driver doctor reports that optional UI Automation support is unhealthy and is falling back to Win32-only window tools. A status response may still be authorized because the installed driver can perform the core Win32 actions.

Root cause and fix

The Windows doctor probes both the optional UIA adapter and the native Win32 path. A broken accessibility provider can make the optional probe slow or degraded even when native input remains usable. The desktop and daemon bound the doctor process at 10 seconds and classify the explicit UIA-to-Win32 fallback as a degraded-but-usable result. CuaDriver 0.18 may emit that fallback as an ANSI-colored stderr warning before the JSON document rather than inside a JSON diagnostic field. Both adapters therefore inspect the complete output, but only after validating that it contains a parseable doctor JSON result. The desktop preserves the diagnostic with reason driver-doctor-failed, while the daemon keeps host-managed computer tools available. A real timeout, malformed response, or missing driver remains an unknown/not-ready result; do not silently treat those as healthy.

Validation

• Run the exact installed cua-driver doctor --json executable and record whether the response is healthy, degraded, timed out, or unparseable.
• Confirm degraded output keeps Win32 computer actions available while the diagnostic warns that UIA-specific window operations may not work.
• Cover both a fallback stored inside the JSON document and an ANSI/stderr fallback prefix followed by an otherwise valid JSON document.
• Confirm a real doctor timeout returns within 10 seconds and does not leave a child process running.