agent-radar
2ed3415fb7
Publish Claude Code Binaries / Stage and publish binaries (push) Canceled after 0s
Publish Tutti App Runtime / Build darwin-arm64 (push) Canceled after 0s
Publish Tutti App Runtime / Build darwin-amd64 (push) Canceled after 0s
Publish Tutti App Runtime / Build linux-amd64 (push) Canceled after 0s
Publish Tutti App Runtime / Build windows-amd64 (push) Canceled after 0s
Sync Tutti Agent Skills / sync (push) Canceled after 0s
Windows Agent Adapters / Test Windows agent adapters (push) Canceled after 0s
Windows Daemon Adapters / Test Windows daemon adapters (push) Canceled after 0s
Publish Tutti App Runtime / Publish runtime catalog (push) Canceled after 0s
109 lines
7.2 KiB
Markdown
109 lines
7.2 KiB
Markdown
---
|
|
name: computer-use
|
|
description: Use to operate the supported desktop — take screenshots, click, type, press keys, scroll — through the Tutti CLI.
|
|
---
|
|
|
|
# Computer Use
|
|
|
|
Use this skill for supported desktop automation (macOS or Windows): screenshot, click, type, press keys, scroll, or move the cursor.
|
|
|
|
Drive the desktop only through `{{.CLICommand}} computer`. The Tutti daemon owns the cua-driver session. Do not use platform-native scripts, direct accessibility APIs, or the standalone cua-driver CLI; those are outside the managed session.
|
|
|
|
## Stable Window Workflow
|
|
|
|
Stable commands implement a window-coordinate workflow. They do not have a shared `--scope` flag.
|
|
|
|
1. Start with `{{command "computer.screenshot"}}`. Prefer an explicit `--pid <pid> --window-id <window-id>` pair; omit both only when automatic visible-window selection is intentional.
|
|
2. Choose `click`, `double-click`, `right-click`, and `scroll` coordinates from that window screenshot. Coordinates are local to its image, so reuse the same explicit pid/window pair for the action.
|
|
3. Act with `click`, `double-click`, `right-click`, `type`, `press-key`, or `scroll`. Re-run `screenshot` after actions that change the UI because coordinates can shift.
|
|
4. `move-cursor` is scope-less. Its coordinates are agent-cursor screen points, not pixels from a window or desktop screenshot.
|
|
|
|
Common forms:
|
|
|
|
- `{{command "computer.screenshot" (args "pid" "<pid>" "window-id" "<window-id>")}}`
|
|
- `{{command "computer.click" (args "pid" "<pid>" "window-id" "<window-id>" "x" "<n>" "y" "<n>")}}`
|
|
- `{{command "computer.type" (args "pid" "<pid>" "window-id" "<window-id>" "text" "<text>")}}`
|
|
- `{{command "computer.press-key" (args "pid" "<pid>" "window-id" "<window-id>" "key" "<key>")}}`
|
|
- `{{command "computer.scroll" (args "pid" "<pid>" "window-id" "<window-id>" "x" "<n>" "y" "<n>" "direction" "<up|down|left|right>" "amount" "<n>")}}`
|
|
- `{{command "computer.move-cursor" (args "x" "<screen-point-x>" "y" "<screen-point-y>")}}`
|
|
|
|
## Click Reliability
|
|
|
|
Pixel clicks are posted as background CGEvents and are never driver-verified: a `✅ Posted click` result only confirms dispatch, not effect. Electron-based apps (Discord, Feishu, VS Code) frequently drop background synthetic clicks entirely.
|
|
|
|
1. Prefer element actions over pixel coordinates. Structured screenshot content can list interactive elements with an `element_token`; click through the native tool with that token:
|
|
|
|
Send the JSON object through standard input so the shell never reparses it as a native-command argument. After defining the PowerShell helper below:
|
|
|
|
`Invoke-TuttiComputerTool -Name "click" -Arguments @{ pid = 9092; element_token = "example" }`
|
|
|
|
2. Verify every click with a fresh `screenshot` of the target window. The blue agent cursor is a visual overlay on a separate channel — after a display-configuration change it can render at a fixed offset from the true event point, so its on-screen position is not evidence of where a click landed.
|
|
|
|
3. If a background pixel click produced no visible change, do not re-click the same coordinates. Escalate delivery through the native `click` schema with `"delivery_mode":"foreground"` (briefly fronts the target window), or switch to the element-token path above.
|
|
|
|
## Native Driver Capabilities
|
|
|
|
The native tool surface is the complete entry point for authorized cua-driver capabilities that have no stable alias. Discover the live contract instead of guessing arguments or assuming that different tools share a scope model:
|
|
|
|
1. Run `{{command "computer.tool.list"}}`.
|
|
2. Run `{{command "computer.tool.describe" (args "name" "<tool>")}}` and follow its returned `inputSchema`.
|
|
3. Serialize the argument object as JSON and pipe it to `{{command "computer.tool.call" (args "name" "<tool>")}} --arguments-json -`. The legacy `--arguments-json` form remains available for simple existing scripts.
|
|
|
|
On PowerShell, use this helper instead of launching a nested `powershell.exe -Command` process. Windows PowerShell 5.1 defaults native-command pipelines to ASCII, so the helper sets `$OutputEncoding` to UTF-8 for each call and restores it afterwards:
|
|
|
|
```powershell
|
|
function Invoke-TuttiComputerTool {
|
|
param([Parameter(Mandatory)][string]$Name, [Parameter(Mandatory)][object]$Arguments)
|
|
$previousOutputEncoding = $OutputEncoding
|
|
try {
|
|
$OutputEncoding = [Text.UTF8Encoding]::new($false)
|
|
$Arguments | ConvertTo-Json -Compress -Depth 20 | {{command "computer.tool.call"}} --name $Name --arguments-json -
|
|
} finally {
|
|
$OutputEncoding = $previousOutputEncoding
|
|
}
|
|
}
|
|
|
|
Invoke-TuttiComputerTool -Name "click" -Arguments @{ pid = 9092; element_token = "example" }
|
|
```
|
|
|
|
### Desktop workflow
|
|
|
|
Desktop capture and input are not one uniform scope:
|
|
|
|
1. When the user asks to inspect the screen, desktop, or entire display rather than a specific app window, use this desktop workflow instead of the stable `screenshot` command. Inspect `get_config`, then read the persisted driver configuration:
|
|
|
|
`{{command "computer.tool.describe" (args "name" "get_config")}}`
|
|
|
|
`Invoke-TuttiComputerTool -Name "get_config" -Arguments @{}`
|
|
|
|
2. `get_desktop_state` works only when the host-global persisted `capture_scope` is `desktop`. If it is not enabled, describe the allowed `set_config` tool and change it through the managed native surface:
|
|
|
|
`{{command "computer.tool.describe" (args "name" "set_config")}}`
|
|
|
|
`Invoke-TuttiComputerTool -Name "set_config" -Arguments @{ capture_scope = "desktop" }`
|
|
|
|
This setting persists globally in cua-driver. Do not hide the mutation inside a set/call/restore sequence, and do not restore it automatically after capture. Re-read `get_config` if the result does not clearly confirm the new value.
|
|
|
|
3. Describe and call `get_desktop_state` using its live schema:
|
|
|
|
`{{command "computer.tool.describe" (args "name" "get_desktop_state")}}`
|
|
|
|
Pass the live-schema object to `Invoke-TuttiComputerTool -Name "get_desktop_state" -Arguments $arguments`.
|
|
|
|
4. The native `click` tool has its own per-call desktop contract. Confirm its live schema, then call it directly:
|
|
|
|
`{{command "computer.tool.describe" (args "name" "click")}}`
|
|
|
|
`Invoke-TuttiComputerTool -Name "click" -Arguments @{ scope = "desktop"; x = 1200; y = 700 }`
|
|
|
|
Do not infer that contract for other tools.
|
|
|
|
5. Desktop scrolling is not supported by the current driver contract: native `scroll` requires a PID and has no true desktop-coordinate mode. Use the stable window workflow when scrolling.
|
|
|
|
## Guardrails
|
|
|
|
- The Tutti computer session is shared per workspace and reused across commands. cua-driver's persisted `capture_scope` is broader global state and is never a hidden per-call switch. Native `set_config` may change it explicitly when the requested workflow requires desktop capture.
|
|
- Prefer explicit `--pid <pid> --window-id <window-id>` for stable window commands. Omit both only when automatic visible-window selection is intentional.
|
|
- Native `tool list` and `tool describe` preserve the live catalog and show Tutti's `allowed` and `denialReason` decision. `tool call` enforces that decision; never try to invoke an entry with `allowed: false`.
|
|
- If cua-driver is missing or Screen Recording/Accessibility permission is denied, report that error instead of falling back to AppleScript or shell automation.
|