maka/docs/archive/computer-use-real-ax-model-e2e.md

Real Model To AppKit AX E2E

Scope

This harness connects a live provider model to the production Maka model and tool path, the pinned cua-driver, and the owned Codex CUA Lab.app.

live provider
  -> getAIModel
  -> AiSdkBackend / streamText
  -> ToolRuntime
  -> maka_computer
  -> cua-driver
  -> AppKit Accessibility element action

The fixture launches in the background and never uses explicit activation. The launcher owns caffeinate, lock/frontmost monitoring, temporary files, fixture shutdown, and child termination.

The action policy allows only scenario-specific subsets of:

list_apps
observe
click_element
set_value
wait

Coordinate click, scroll, drag, type, key input, and compatibility pixel fallback are not available. A read-only HID age probe fences native semantic dispatch while the user is actively providing physical input.

Real Results

OpenAI through Azure bridge

gpt-5.6-sol completed:

• observe-only: two tools, zero dispatch;
• AX set value: one verified AX dispatch, zero pixel dispatch;
• physical intervention recovery: first mutation returned reobserve_required, then the model observed and succeeded;
• process restart recovery: the old semantic action returned target_missing, then the model rediscovered the new process and succeeded;
• AX click: one click_element dispatch and the external AppKit button oracle changed exactly once.
• safe multi-step task: set_value followed by click_element, each using the fresh observation from the prior step; one dispatch per semantic action and zero pixel dispatch.

Anthropic through coproxy

claude-sonnet-4-6 completed:

• AX set value through the full AiSdkBackend product path;
• physical intervention recovery with the same re-observe sequence;
• one verified AX dispatch and zero pixel dispatch in both mutation runs.
• the same safe multi-step set_value then click_element task through the full product loop.

Kimi and MiniMax protocol paths

No live Kimi or MiniMax credential is configured on this machine. Their product paths were therefore validated as hermetic-protocol, not real-runtime.

Both kimi-coding-plan and minimax-coding-plan completed:

getAIModel
  -> Anthropic-compatible streaming protocol
  -> AiSdkBackend
  -> ToolRuntime
  -> maka_computer
  -> list_apps -> observe -> set_value -> final response

The server fixtures verify each provider's exact URL prefix, API-key header, model ID, streaming tool_use, tool-result reinjection, and final semantic state. They also verify that a failed semantic action is reinjected as tool_result.is_error:true; Runtime records the failed result once, while the AI SDK adapter exposes it to the provider as a recoverable tool execution error.

Dynamic Structure Finding

The model observed one stale target. An independent AX setup action inserted a second target with the same semantic identity before the model action.

The real executor returned target_changed, with zero model dispatch and zero target mutation. The content fingerprint gate currently runs before strict element identity refetch. This is safe, but stricter than the recovered Codex behavior where semantic element actions may continue to a unique/missing/ ambiguous refetch decision after an AX structure change.

The finding is reported on executor hardening PR #910. The model-runner treats either strict refetch rejection or conservative target_changed as a fail-closed result and records the exact result code.

Runtime Finding

Real model runs exposed that returned Computer Use results with a top-level error were being recorded as successful tool invocations. A model could also retry the same ambiguity-rejected semantic target after a new observation because the generic loop signature included the changing observation ID.

The independent Runtime fix:

• classifies returned top-level errors consistently for persisted results, events, telemetry, and loop control;
• gives explicit ambiguity failures a stable semantic signature that ignores observation ID;
• blocks the next identical ambiguity retry before native execution;
• does not cache recoverable user_intervened or target_missing outcomes.

The real evidence sanitizer also previously hard-coded an OpenAI producer. Reports now preserve an explicit producer and provider while retaining the same privacy projection.

Evidence Finding

The Desktop runner previously inferred fixture windows from action output, so an incorrect action window could add itself to the ownership allowlist. The launcher now discovers the exact fixture PID/window set independently before the model runs, binds actions by tool-call dispatch traces or observation lineage, and waits for trace flush before qualification.

Evidence Privacy

Reports are launcher-owned temporary files with mode 0600. The sanitized projection retains provider/model, action types, result codes, aggregate state, and allowlisted trace fields. It drops coordinates, typed values, AX/UI text, credentials, URLs beyond origin, and raw provider payloads.