2026-09-02 01:05:41 +08:00
commit 04bac1f6e2
2200 changed files with 489415 additions and 0 deletions
@@ -0,0 +1,38 @@
# OpenWork Computer Use
Native macOS computer-use runtime for OpenWork.
Packaged OpenWork builds wrap this runtime in a bundled `Computer Use.app`
helper so macOS Accessibility and Screen Recording permissions belong to the
helper app instead of a transient Node or Swift process.
This package focuses on the reusable control layer:
- Semantic AX snapshots with compact refs like `{e1}`.
- Strict background mode that avoids foreground cursor/HID fallbacks.
- Target-window screenshots via `CGWindowListCreateImage(.optionIncludingWindow)`.
- Background input through `CGEvent.postToPid` with window-addressing fields.
- Background activation using per-process event taps plus AppKit and center-click primers.
- Non-UI orchestration modules from the original Electron prototype: realtime tool schemas/instructions and the GPT computer-use loop.
Computer Use renders a lightweight second cursor overlay from the helper app so
users can see where the agent is acting. The overlay is visual only: strict mode
does not move the real system cursor. Actual control uses the same
non-interrupting mechanism described in Bridge-style background computer use: AX
first, then `CGEvent.postToPid` addressed to the target process/window, with
per-process focus-message taps and activation primers so the user's frontmost app
can remain frontmost while the target app accepts events. Foreground HID fallback
is only used when strict mode is disabled.
Build the native stdio server:
```bash
pnpm --filter @openwork/handsfree check:native
```
Run it as an MCP-compatible adapter:
```bash
pnpm --filter @openwork/handsfree exec openwork-handsfree-computer-use mcp
```
The core runtime is intentionally MCP-independent. `ComputerUseRuntime` exposes a small direct surface (`snapshot`, `click`, `typeText`, `pressKey`, `scroll`, `wait`, `setValue`, `performAction`); `MCPServer` is only a thin stdio wrapper.