fix(vom): bug fix

CHANGELOG.md

@@ -1,92 +0,0 @@
-# Changelog
-
-All notable changes to BrowserSkill will be documented in this file.
-The format is based on [Keep a Changelog](https://keepachangelog.com/).
-
-Starting from 0.2.0, CLI / Extension / DSH Plugin share the same version number.
-
-## [Unreleased]
-
-## [0.2.0] - 2026-09-02
-
-### Added
-- File transfer: upload and download support across CLI, Extension, and DSH Plugin
-- File transfer: drag-and-drop upload (`drop-to-upload`)
-- VOM semantic graph, name enrichment, and hover perception modules
-- VOM hover probing (opt-in via `observe` parameter)
-- DSH Plugin: browser tool parity with CLI commands
-- Edge Add-ons automated publishing in CI
-- Protocol upgrade reminder when CLI / Extension protocol versions differ
-- Browser evaluation harness (`evals/browser/`)
-- Unified release script (`scripts/release.mjs`)
-
-### Changed
-- **Version scheme**: all three components now share the same semver
-- VOM rendering algorithm optimizations
-- Leaner SKILL.md agent instructions
-- DSH Plugin: simplified browser commands
-- Borrow confirmation UX — proactive focus and longer timeout
-- PiP window now has a close button
-
-### Fixed
-- Screenshot media type detection (was hard-coded to `image/png`)
-- DSH Plugin session lifecycle stability
-- DSH Plugin Cordis package ID mismatch
-- Observation thumbnail media type sniffing
-- Upload/download race conditions and layout bypass issues
-- VOM repeated name and safety policy issues
-
----
-
-*Previous releases used independent version numbers per component.*
-
-## CLI 0.1.11 / Extension 0.1.7 / DSH Plugin 0.1.2 — 2026-08-26 ~ 2026-08-29
-
-### Added
-- DSH Plugin sidebar integration, session lifecycle, and archive cleanup
-- Recorder iframe and OOPIF support
-
-### Changed
-- VOM functional refactor
-
-### Fixed
-- Recorder safety policy and bug fixes
-
-## CLI 0.1.10 / Extension 0.1.6 — 2026-08-08
-
-### Added
-- VOM observation recording and settled-state detection
-- Record overlay timer
-
-### Fixed
-- Browser keepalive disconnect handling
-
-## CLI 0.1.9 / Extension 0.1.5 — 2026-07-29
-
-### Added
-- CLI auto-update mechanism
-- Trace v3 protocol and recorder
-
-### Fixed
-- MV3 keepalive disconnect
-
-## CLI 0.1.8 / Extension 0.1.4 — 2026-07-22
-
-### Added
-- More browser interaction actions
-
-### Fixed
-- Windows named-pipe hash-only path issue
-
-## CLI 0.1.7 / Extension 0.1.3 — 2026-07-07
-
-Initial public release pair.
-
-## CLI 0.1.6 — 2026-06-30
-
-### Fixed
-- Minor CLI fixes
-
-## CLI 0.1.5 / Extension 0.1.2 — 2026-06-22
-
-First tagged releases.

Cargo.lock

@@ -134,7 +134,7 @@ dependencies = [
 
 [[package]]
 name = "bsk"
-version = "0.2.0"
+version = "0.1.11"
 dependencies = [
  "anyhow",
  "base64",
@@ -169,7 +169,7 @@ dependencies = [
 
 [[package]]
 name = "bsk-protocol"
-version = "0.2.0"
+version = "0.1.11"
 dependencies = [
  "schemars",
  "semver",

Cargo.toml

@@ -7,7 +7,7 @@ members = [
 default-members = ["crates/bsk-cli"]
 
 [workspace.package]
-version = "0.2.0"
+version = "0.1.11"
 edition = "2024"
 rust-version = "1.85"
 license = "MIT"
@@ -43,7 +43,7 @@ rand = "0.8"
 uuid = { version = "1", features = ["v4", "serde"] }
 
 # Workspace crates (version required for `cargo publish` of dependents)
-bsk-protocol = { version = "0.2.0", path = "crates/bsk-protocol" }
+bsk-protocol = { version = "0.1.9", path = "crates/bsk-protocol" }
 
 [profile.release]
 strip = "symbols"

apps/extension/e2e/fixtures/file-transfer.html

@@ -1,81 +0,0 @@
-<!doctype html>
-<html lang="en">
-  <head>
-    <meta charset="utf-8" />
-    <title>BrowserSkill file-transfer fixture</title>
-    <style>
-      body { font: 16px system-ui; max-width: 760px; margin: 48px auto; }
-      .actions { display: flex; gap: 12px; align-items: center; margin: 24px 0; }
-      button, a { padding: 10px 16px; border: 1px solid #777; border-radius: 8px; }
-      a[aria-disabled="true"] { opacity: .45; pointer-events: none; }
-      #drop { padding: 36px; border: 3px dashed #777; border-radius: 12px; text-align: center; }
-      #drop.dragging { border-color: #2563eb; background: #eff6ff; }
-      img { display: block; max-width: 480px; max-height: 360px; margin-top: 20px; }
-    </style>
-  </head>
-  <body>
-    <h1>Upload then download the same image</h1>
-    <p id="status">No image selected</p>
-    <div class="actions">
-      <button id="upload" type="button">Upload through file input</button>
-      <input id="file" type="file" accept="image/png" hidden />
-      <a id="download" aria-disabled="true">Download uploaded image</a>
-    </div>
-    <div id="drop" role="button" tabindex="0">Drop an image here</div>
-    <img id="preview" alt="Uploaded image preview" hidden />
-    <script>
-      const input = document.querySelector('#file');
-      const upload = document.querySelector('#upload');
-      const drop = document.querySelector('#drop');
-      const download = document.querySelector('#download');
-      const preview = document.querySelector('#preview');
-      const status = document.querySelector('#status');
-      let objectUrl;
-
-      window.__bskTransferFixture = { events: [], result: null };
-
-      function accept(file, mechanism) {
-        if (!file) return;
-        if (objectUrl) URL.revokeObjectURL(objectUrl);
-        objectUrl = URL.createObjectURL(file);
-        preview.src = objectUrl;
-        preview.hidden = false;
-        download.href = objectUrl;
-        download.download = file.name;
-        download.removeAttribute('aria-disabled');
-        status.textContent = `Ready via ${mechanism}: ${file.name} (${file.size} bytes)`;
-        window.__bskTransferFixture.result = {
-          mechanism,
-          name: file.name,
-          size: file.size,
-          type: file.type,
-          hasPathProperty: 'path' in file,
-        };
-      }
-
-      upload.addEventListener('click', () => input.click());
-      input.addEventListener('change', () => accept(input.files[0], 'input'));
-
-      for (const type of ['dragenter', 'dragover']) {
-        drop.addEventListener(type, event => {
-          const types = [...event.dataTransfer.types];
-          window.__bskTransferFixture.events.push({ type, types });
-          if (!types.includes('Files')) return;
-          event.preventDefault();
-          drop.classList.add('dragging');
-        });
-      }
-      drop.addEventListener('dragleave', () => drop.classList.remove('dragging'));
-      drop.addEventListener('drop', event => {
-        window.__bskTransferFixture.events.push({
-          type: 'drop',
-          types: [...event.dataTransfer.types],
-          fileCount: event.dataTransfer.files.length,
-        });
-        event.preventDefault();
-        drop.classList.remove('dragging');
-        accept(event.dataTransfer.files[0], 'drop');
-      });
-    </script>
-  </body>
-</html>

apps/extension/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@browser-skill/extension",
-  "version": "0.2.0",
+  "version": "0.1.7",
   "private": true,
   "type": "module",
   "scripts": {

apps/extension/src/entrypoints/popup/App.test.tsx

@@ -122,8 +122,8 @@ describe("App", () => {
 
     expect(screen.queryByText(/^扩展 v/)).toBeNull();
     expect(screen.queryByText(/^daemon v/)).toBeNull();
-    expect(screen.getByTitle("扩展版本").textContent).toBe(`Ext ${EXTENSION_VERSION}`);
-    expect(screen.getByTitle("bsk 版本").textContent).toBe(`CLI ${mockDaemonVersion}`);
+    expect(screen.getByTitle("扩展版本").textContent).toBe(EXTENSION_VERSION);
+    expect(screen.getByTitle("bsk 版本").textContent).toBe(mockDaemonVersion);
     expect(screen.getByText("03c3e47f")).toBeTruthy();
 
     const copyButton = screen.getByRole("button", { name: "复制实例 ID" });

apps/extension/src/entrypoints/popup/App.tsx

@@ -267,9 +267,9 @@ export function App() {
             data-slot="popup-meta"
           >
             <div className="flex shrink-0 items-center gap-1">
-              <span title={t("popup.daemonVersionHint")}>CLI {daemonVersion}</span>
+              <span title={t("popup.extensionVersionHint")}>{extensionVersion}</span>
               <span aria-hidden>/</span>
-              <span title={t("popup.extensionVersionHint")}>Ext {extensionVersion}</span>
+              <span title={t("popup.daemonVersionHint")}>{daemonVersion}</span>
             </div>
             <div className="flex min-w-0 items-center justify-end gap-1">
               <span className="shrink-0">{t("popup.instanceTitle")}</span>

apps/extension/src/lib/__tests__/capture-suppress-bridge.test.ts

@@ -2,7 +2,7 @@ import { describe, expect, it, vi } from "vitest";
 import {
   CAPTURE_SUPPRESS,
   type CaptureSuppressMessage,
-  withExtensionOverlayHidden,
+  withOverlaysHiddenForCapture,
 } from "../capture-suppress-bridge";
 
 function recordingSendToTab(events: string[]) {
@@ -12,12 +12,12 @@ function recordingSendToTab(events: string[]) {
   });
 }
 
-describe("withExtensionOverlayHidden", () => {
+describe("withOverlaysHiddenForCapture", () => {
   it("sends begin → fn → end and returns fn's result", async () => {
     const events: string[] = [];
     const sendToTab = recordingSendToTab(events);
 
-    const result = await withExtensionOverlayHidden(
+    const result = await withOverlaysHiddenForCapture(
       7,
       async () => {
         events.push("capture");
@@ -37,7 +37,7 @@ describe("withExtensionOverlayHidden", () => {
     const sendToTab = recordingSendToTab(events);
 
     await expect(
-      withExtensionOverlayHidden(
+      withOverlaysHiddenForCapture(
         7,
         async () => {
           events.push("capture");
@@ -55,7 +55,7 @@ describe("withExtensionOverlayHidden", () => {
     });
     const fn = vi.fn(async () => "shot");
 
-    const result = await withExtensionOverlayHidden(7, fn, sendToTab);
+    const result = await withOverlaysHiddenForCapture(7, fn, sendToTab);
 
     expect(result).toBe("shot");
     expect(fn).toHaveBeenCalledTimes(1);
@@ -69,7 +69,7 @@ describe("withExtensionOverlayHidden", () => {
       return { type: CAPTURE_SUPPRESS, ok: true };
     });
 
-    const result = await withExtensionOverlayHidden(7, async () => "shot", sendToTab);
+    const result = await withOverlaysHiddenForCapture(7, async () => "shot", sendToTab);
 
     expect(result).toBe("shot");
     expect(sendToTab).toHaveBeenCalledTimes(2);

apps/extension/src/lib/capture-suppress-bridge.ts

@@ -1,9 +1,9 @@
 /**
- * Wire protocol and background-side helper for temporarily excluding the
- * extension's in-page overlay. Screenshots use it to avoid captured chrome;
- * coordinate-driven native input uses it so Chrome hit-tests the page.
+ * Wire protocol and background-side helper for hiding the extension's
+ * in-page overlay while a screenshot is captured, so captured frames only
+ * contain the page itself.
  *
- * The background wraps each bounded operation in `withExtensionOverlayHidden`,
+ * The background wraps every capture in `withOverlaysHiddenForCapture`,
  * which sends `begin` (the content script hides the overlay host and only
  * acks once the compositor has produced an overlay-free frame) and always
  * follows up with `end`, even when the capture throws. Tabs without the
@@ -46,7 +46,7 @@ const defaultSendToTab: CaptureSuppressSendToTab = (tabId, message) =>
  * (or any other `begin` failure) simply skips suppression — there is no
  * overlay to hide in that tab.
  */
-export async function withExtensionOverlayHidden<T>(
+export async function withOverlaysHiddenForCapture<T>(
   tabId: number,
   fn: () => Promise<T>,
   sendToTab: CaptureSuppressSendToTab = defaultSendToTab,

apps/extension/src/session-manager/__tests__/ref-store.test.ts

@@ -13,6 +13,8 @@ describe("RefStore", () => {
       backendNodeId: 42,
       tabId: 7,
       generation: 0,
+      kind: "dom",
+      capabilities: ["interact", "screenshot"],
     });
     expect(s.size()).toBe(1);
     expect(s.isEmpty()).toBe(false);
@@ -48,4 +50,31 @@ describe("RefStore", () => {
       cdpSessionId: "child-session",
     });
   });
+
+  it("preserves explicit capabilities for visual surface refs", () => {
+    const s = new RefStore();
+    s.set("e1", 42, {
+      tabId: 7,
+      kind: "surface",
+      capabilities: ["screenshot"],
+    });
+
+    expect(s.resolveEntry("e1")).toMatchObject({
+      kind: "surface",
+      capabilities: ["screenshot"],
+    });
+  });
+
+  it("never grants ordinary interaction to a surface through defaults or caller input", () => {
+    const s = new RefStore();
+    s.set("e1", 42, { tabId: 7, kind: "surface" });
+    s.set("e2", 43, {
+      tabId: 7,
+      kind: "surface",
+      capabilities: ["interact", "screenshot"],
+    });
+
+    expect(s.resolveEntry("e1")?.capabilities).toEqual(["screenshot"]);
+    expect(s.resolveEntry("e2")?.capabilities).toEqual(["screenshot"]);
+  });
 });

apps/extension/src/session-manager/ref-store.ts

@@ -1,3 +1,5 @@
+import type { Rect } from "@browser-skill/vom";
+
 /**
  * Per-session map from `@e<N>` snapshot refs to a CDP node address.
  *
@@ -12,12 +14,17 @@
  * inside the `SessionContext`.
  */
 export type BackendNodeId = number;
+export type RefCapability = "interact" | "screenshot";
+export type RefTargetKind = "dom" | "surface";
 
 export interface RefEntry {
   backendNodeId: BackendNodeId;
   tabId: number | null;
   frameId?: string;
   cdpSessionId?: string;
+  visibleRect?: Rect;
+  kind: RefTargetKind;
+  capabilities: RefCapability[];
   generation: number;
 }
 
@@ -28,8 +35,19 @@ export type RefInput =
       tabId: number;
       frameId?: string;
       cdpSessionId?: string;
+      visibleRect?: Rect;
+      kind?: RefTargetKind;
+      capabilities?: RefCapability[];
     };
 
+function capabilitiesFor(
+  kind: RefTargetKind,
+  capabilities: RefCapability[] | undefined,
+): RefCapability[] {
+  if (kind === "surface") return ["screenshot"];
+  return capabilities ?? ["interact", "screenshot"];
+}
+
 export class RefStore {
   private readonly map = new Map<string, RefEntry>();
   private generation = 0;
@@ -70,13 +88,20 @@ export class RefStore {
       tabId?: number;
       frameId?: string;
       cdpSessionId?: string;
+      visibleRect?: Rect;
+      kind?: RefTargetKind;
+      capabilities?: RefCapability[];
     } = {},
   ): void {
+    const kind = opts.kind ?? "dom";
     this.map.set(normaliseRef(ref), {
       backendNodeId: id,
       tabId: opts.tabId ?? null,
       ...(opts.frameId ? { frameId: opts.frameId } : {}),
       ...(opts.cdpSessionId ? { cdpSessionId: opts.cdpSessionId } : {}),
+      ...(opts.visibleRect ? { visibleRect: opts.visibleRect } : {}),
+      kind,
+      capabilities: capabilitiesFor(kind, opts.capabilities),
       generation: this.generation,
     });
   }
@@ -94,14 +119,20 @@ export class RefStore {
       return {
         backendNodeId: input,
         tabId: null,
+        kind: "dom",
+        capabilities: ["interact", "screenshot"],
         generation: this.generation,
       };
     }
+    const kind = input.kind ?? "dom";
     return {
       backendNodeId: input.backendNodeId,
       tabId: input.tabId,
       ...(input.frameId ? { frameId: input.frameId } : {}),
       ...(input.cdpSessionId ? { cdpSessionId: input.cdpSessionId } : {}),
+      ...(input.visibleRect ? { visibleRect: input.visibleRect } : {}),
+      kind,
+      capabilities: capabilitiesFor(kind, input.capabilities),
       generation: this.generation,
     };
   }

apps/extension/src/tools/__tests__/file-transfer.test.ts

@@ -2,7 +2,6 @@ import { describe, expect, it, vi } from "vitest";
 import { SessionManager } from "@/session-manager/manager";
 import { type DownloadsApi, handleDownload } from "../download";
 import { captureBrowserDownload } from "../download-capture";
-import { uploadThroughFileDrop } from "../file-drop-transaction";
 import { uploadThroughActivatedFileInput } from "../file-input-transaction";
 import type { ResolvedActionTarget } from "../interaction";
 import type { CdpRunner } from "../shared";
@@ -38,20 +37,16 @@ function fakeEvent<T extends (...args: never[]) => unknown>() {
   };
 }
 
-function actionTarget(frameId?: string, sessionId?: string): ResolvedActionTarget {
+function actionTarget(frameId?: string): ResolvedActionTarget {
   return {
     tab: { tabId: 4, windowId: 100, active: true },
     backendNodeId: 123,
-    cdpTarget: { tabId: 4, ...(sessionId ? { sessionId } : {}) },
+    cdpTarget: { tabId: 4 },
     ...(frameId ? { frameId } : {}),
     usedRef: "e3",
   };
 }
 
-function withoutOverlay<T>(operation: () => Promise<T>): Promise<T> {
-  return operation();
-}
-
 function uploadCdp(
   options: {
     inputCount?: number;
@@ -64,7 +59,6 @@ function uploadCdp(
   let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
   const send = vi.fn(async (_tabId: number, method: string, params?: object) => {
     calls.push({ method, params });
-    if (method === "DOM.scrollIntoViewIfNeeded") return {};
     if (method === "Page.setInterceptFileChooserDialog") return {};
     if (method === "Page.getLayoutMetrics")
       return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
@@ -116,50 +110,6 @@ function uploadCdp(
   };
 }
 
-function dropCdp(
-  options: {
-    targetContainsPoint?: boolean;
-    dispatchError?: (type: string) => Error | Promise<never> | undefined;
-    onCall?: (method: string) => void;
-  } = {},
-) {
-  const calls: Array<{ method: string; params?: object }> = [];
-  const detach = vi.fn(async () => {});
-  const send = vi.fn(async (_tabId: number, method: string, params?: object) => {
-    calls.push({ method, params });
-    options.onCall?.(method);
-    if (method === "DOM.scrollIntoViewIfNeeded") return {};
-    if (method === "Page.getLayoutMetrics") {
-      return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
-    }
-    if (method === "DOM.getContentQuads") {
-      return { quads: [[10, 20, 210, 20, 210, 120, 10, 120]] };
-    }
-    if (method === "DOM.getNodeForLocation") return { backendNodeId: 124 };
-    if (method === "DOM.resolveNode") {
-      const id = (params as { backendNodeId?: number }).backendNodeId;
-      return { object: { objectId: id === 123 ? "drop-target" : "hit-target" } };
-    }
-    if (method === "Runtime.callFunctionOn") {
-      return { result: { value: options.targetContainsPoint ?? true } };
-    }
-    if (method === "Input.dispatchDragEvent") {
-      const type = (params as { type?: string }).type ?? "";
-      const error = options.dispatchError?.(type);
-      if (error instanceof Promise) return error;
-      if (error) throw error;
-      return {};
-    }
-    if (method === "Runtime.releaseObjectGroup") return {};
-    throw new Error(`unexpected drop CDP command ${method}`);
-  });
-  const cdp: CdpRunner = {
-    send: send as unknown as CdpRunner["send"],
-    detach,
-  };
-  return { calls, cdp, detach };
-}
-
 describe("file transfer tools", () => {
   it("captures the file input activated by the requested click and injects only staged paths", async () => {
     const manager = sessions();
@@ -296,6 +246,104 @@ describe("file transfer tools", () => {
     });
   });
 
+  it("does not dispatch browser work when upload is already cancelled", async () => {
+    const controller = new AbortController();
+    controller.abort();
+    const { cdp, calls } = uploadCdp();
+
+    const result = await uploadThroughActivatedFileInput({
+      cdp,
+      actionTarget: actionTarget(),
+      files: ["/private/stage/one"],
+      timeoutMs: 100,
+      signal: controller.signal,
+      trigger: vi.fn(),
+    });
+
+    expect(result).toMatchObject({
+      code: "cancelled",
+      data: { effect_state: "none", phase: "arm_interception" },
+    });
+    expect(calls).not.toContainEqual({
+      method: "Page.setInterceptFileChooserDialog",
+      params: { enabled: true },
+    });
+    expect(calls.some((call) => call.method === "DOM.setFileInputFiles")).toBe(false);
+  });
+
+  it("stops before file assignment when cancellation interrupts input resolution", async () => {
+    const controller = new AbortController();
+    const fixture = uploadCdp();
+    const originalSend = fixture.cdp.send;
+    let markProbeStarted!: () => void;
+    const probeStarted = new Promise<void>((resolve) => {
+      markProbeStarted = resolve;
+    });
+    fixture.cdp.send = vi.fn((tabId: number, method: string, params?: object) => {
+      const declaration = (params as { functionDeclaration?: string } | undefined)
+        ?.functionDeclaration;
+      if (
+        method === "Runtime.callFunctionOn" &&
+        declaration?.includes("count: state.inputs.length")
+      ) {
+        markProbeStarted();
+        return new Promise<never>(() => {});
+      }
+      return originalSend(tabId, method, params);
+    }) as CdpRunner["send"];
+
+    const transaction = uploadThroughActivatedFileInput({
+      cdp: fixture.cdp,
+      actionTarget: actionTarget(),
+      files: ["/private/stage/one"],
+      timeoutMs: 1_000,
+      signal: controller.signal,
+      trigger: async () => ({ tab_id: 4, x: 10, y: 10 }),
+    });
+    await probeStarted;
+    controller.abort();
+    const result = await transaction;
+
+    expect(result).toMatchObject({
+      code: "cancelled",
+      data: { effect_state: "none", phase: "resolve_input" },
+    });
+    expect(fixture.calls.some((call) => call.method === "DOM.setFileInputFiles")).toBe(false);
+  });
+
+  it("disarms chooser interception when its enable response times out", async () => {
+    const calls: Array<{ method: string; params?: object }> = [];
+    const cdp: CdpRunner = {
+      send: vi.fn(async (_tabId: number, method: string, params?: object) => {
+        calls.push({ method, params });
+        if (
+          method === "Page.setInterceptFileChooserDialog" &&
+          (params as { enabled?: boolean }).enabled
+        ) {
+          return new Promise<never>(() => {});
+        }
+        return {};
+      }) as CdpRunner["send"],
+    };
+
+    const result = await uploadThroughActivatedFileInput({
+      cdp,
+      actionTarget: actionTarget(),
+      files: ["/private/stage/one"],
+      timeoutMs: 5,
+      trigger: vi.fn(),
+    });
+
+    expect(result).toMatchObject({
+      code: "timeout",
+      data: { effect_state: "none", phase: "arm_interception" },
+    });
+    expect(calls).toContainEqual({
+      method: "Page.setInterceptFileChooserDialog",
+      params: { enabled: false, cancel: true },
+    });
+  });
+
   it("marks a timed-out file assignment unknown and detaches browser state", async () => {
     const fixture = uploadCdp();
     const originalSend = fixture.cdp.send;
@@ -321,236 +369,6 @@ describe("file transfer tools", () => {
     expect(detach).toHaveBeenCalledWith(4);
   });
 
-  it("delivers staged files through one explicit native drop without clicking", async () => {
-    const manager = sessions();
-    const ctx = await manager.start("s1");
-    ctx.refStore.set("e3", 123, { tabId: 4 });
-    const lifecycle: string[] = [];
-    const { cdp, calls } = dropCdp({ onCall: (method) => lifecycle.push(method) });
-    const overlayPhases: string[] = [];
-
-    const result = await handleUpload(
-      manager,
-      {
-        session_id: "s1",
-        ref: "@e3",
-        mode: "drop",
-        files: [{ transfer_id: "tr_1", name: "one.png", staged_path: "/stage/one.png" }],
-      },
-      {
-        cdp,
-        tabsApi: tabsApi(),
-        sendToTab: vi.fn(async (_tabId, message) => {
-          overlayPhases.push(message.phase);
-          lifecycle.push(`overlay:${message.phase}`);
-        }),
-      },
-    );
-
-    expect(result).toMatchObject({ tab_id: 4, used_ref: "e3", file_names: ["one.png"] });
-    expect(calls.filter((call) => call.method === "Input.dispatchDragEvent")).toEqual([
-      {
-        method: "Input.dispatchDragEvent",
-        params: {
-          type: "dragEnter",
-          x: 110,
-          y: 70,
-          data: { items: [], files: ["/stage/one.png"], dragOperationsMask: 1 },
-        },
-      },
-      {
-        method: "Input.dispatchDragEvent",
-        params: {
-          type: "dragOver",
-          x: 110,
-          y: 70,
-          data: { items: [], files: ["/stage/one.png"], dragOperationsMask: 1 },
-        },
-      },
-      {
-        method: "Input.dispatchDragEvent",
-        params: {
-          type: "drop",
-          x: 110,
-          y: 70,
-          data: { items: [], files: ["/stage/one.png"], dragOperationsMask: 1 },
-        },
-      },
-    ]);
-    expect(calls.some((call) => call.method === "Input.dispatchMouseEvent")).toBe(false);
-    expect(overlayPhases).toEqual(["begin", "end"]);
-    expect(lifecycle.indexOf("DOM.getContentQuads")).toBeLessThan(
-      lifecycle.indexOf("overlay:begin"),
-    );
-    expect(lifecycle.indexOf("overlay:begin")).toBeLessThan(
-      lifecycle.indexOf("DOM.getNodeForLocation"),
-    );
-    expect(lifecycle.lastIndexOf("Input.dispatchDragEvent")).toBeLessThan(
-      lifecycle.indexOf("overlay:end"),
-    );
-    expect(lifecycle.indexOf("overlay:end")).toBeLessThan(
-      lifecycle.indexOf("Runtime.releaseObjectGroup"),
-    );
-  });
-
-  it("fails closed before dragging when another element owns the action point", async () => {
-    const { cdp, calls } = dropCdp({ targetContainsPoint: false });
-    const result = await uploadThroughFileDrop({
-      cdp,
-      actionTarget: actionTarget(),
-      files: ["/stage/one.png"],
-      timeoutMs: 100,
-      withOverlayHidden: withoutOverlay,
-    });
-
-    expect(result).toMatchObject({
-      code: "permission_denied",
-      data: {
-        reason: "file_drop_target_unavailable",
-        mechanism: "drop",
-        effect_state: "none",
-        phase: "resolve_target",
-      },
-    });
-    expect(calls.some((call) => call.method === "Input.dispatchDragEvent")).toBe(false);
-  });
-
-  it("dispatches an OOPIF file drop in the child target's local coordinates", async () => {
-    const rootCalls: Array<{ method: string; params?: object }> = [];
-    const childCalls: Array<{ method: string; params?: object }> = [];
-    const cdp: CdpRunner = {
-      send: vi.fn(async (_tabId, method, params) => {
-        rootCalls.push({ method, params });
-        if (method === "DOM.scrollIntoViewIfNeeded") return {};
-        if (method === "DOM.getBoxModel") {
-          return { model: { content: [100, 100, 300, 100, 300, 200, 100, 200] } };
-        }
-        if (method === "Page.getLayoutMetrics") {
-          return { cssLayoutViewport: { clientWidth: 800, clientHeight: 600 } };
-        }
-        throw new Error(`unexpected root command ${method}`);
-      }) as CdpRunner["send"],
-      sendToTarget: vi.fn(async (_target, method, params) => {
-        childCalls.push({ method, params });
-        if (method === "DOM.scrollIntoViewIfNeeded") return {};
-        if (method === "DOM.getContentQuads") {
-          return { quads: [[10, 20, 210, 20, 210, 120, 10, 120]] };
-        }
-        if (method === "Page.getLayoutMetrics") {
-          return { cssLayoutViewport: { clientWidth: 400, clientHeight: 300 } };
-        }
-        if (method === "DOM.getNodeForLocation") {
-          return { backendNodeId: 124, frameId: "child" };
-        }
-        if (method === "DOM.resolveNode") {
-          const id = (params as { backendNodeId?: number }).backendNodeId;
-          return { object: { objectId: id === 123 ? "drop-target" : "hit-target" } };
-        }
-        if (method === "Runtime.callFunctionOn") return { result: { value: true } };
-        if (method === "Input.dispatchDragEvent" || method === "Runtime.releaseObjectGroup") {
-          return {};
-        }
-        throw new Error(`unexpected child command ${method}`);
-      }) as CdpRunner["sendToTarget"],
-      getFrameGraph: vi.fn(async () => ({
-        rootFrameId: "main",
-        frames: [
-          { frameId: "main", target: { tabId: 4 } },
-          {
-            frameId: "child",
-            parentFrameId: "main",
-            ownerBackendNodeId: 10,
-            target: { tabId: 4, sessionId: "child-session" },
-          },
-        ],
-      })),
-    };
-
-    const result = await uploadThroughFileDrop({
-      cdp,
-      actionTarget: actionTarget("child", "child-session"),
-      files: ["/stage/one.png"],
-      timeoutMs: 100,
-      withOverlayHidden: withoutOverlay,
-    });
-
-    expect(result).toMatchObject({ x: 110, y: 70 });
-    expect(
-      childCalls
-        .filter((call) => call.method === "Input.dispatchDragEvent")
-        .map((call) => call.params),
-    ).toEqual([
-      expect.objectContaining({ type: "dragEnter", x: 110, y: 70 }),
-      expect.objectContaining({ type: "dragOver", x: 110, y: 70 }),
-      expect.objectContaining({ type: "drop", x: 110, y: 70 }),
-    ]);
-    expect(rootCalls.some((call) => call.method === "Input.dispatchDragEvent")).toBe(false);
-  });
-
-  it("reports a missing native drag command as unsupported before commit", async () => {
-    const { cdp } = dropCdp({
-      dispatchError: (type) =>
-        type === "dragEnter" ? new Error("'Input.dispatchDragEvent' wasn't found") : undefined,
-    });
-    const result = await uploadThroughFileDrop({
-      cdp,
-      actionTarget: actionTarget(),
-      files: ["/stage/one.png"],
-      timeoutMs: 100,
-      withOverlayHidden: withoutOverlay,
-    });
-
-    expect(result).toMatchObject({
-      code: "unsupported",
-      data: {
-        reason: "upload_mechanism_unsupported",
-        effect_state: "none",
-        phase: "drag_enter",
-      },
-    });
-  });
-
-  it("cancels an unresolved drag before returning a pre-commit timeout", async () => {
-    const { cdp, calls } = dropCdp({
-      dispatchError: (type) => (type === "dragEnter" ? new Promise<never>(() => {}) : undefined),
-    });
-    const result = await uploadThroughFileDrop({
-      cdp,
-      actionTarget: actionTarget(),
-      files: ["/stage/one.png"],
-      timeoutMs: 10,
-      withOverlayHidden: withoutOverlay,
-    });
-
-    expect(result).toMatchObject({
-      code: "timeout",
-      data: { reason: "file_drop_failed", effect_state: "none", phase: "drag_enter" },
-    });
-    expect(calls).toContainEqual({
-      method: "Input.dispatchDragEvent",
-      params: expect.objectContaining({ type: "dragCancel" }),
-    });
-  });
-
-  it("marks an unresolved drop unknown and detaches browser state", async () => {
-    const { cdp, detach } = dropCdp({
-      dispatchError: (type) => (type === "drop" ? new Promise<never>(() => {}) : undefined),
-    });
-    const result = await uploadThroughFileDrop({
-      cdp,
-      actionTarget: actionTarget(),
-      files: ["/stage/one.png"],
-      timeoutMs: 10,
-      withOverlayHidden: withoutOverlay,
-    });
-
-    expect(result).toMatchObject({
-      code: "timeout",
-      data: { reason: "file_drop_failed", effect_state: "unknown", phase: "drop" },
-    });
-    expect(detach).toHaveBeenCalledWith(4);
-  });
-
   it("routes one exact-target download through a browser-relative capability", async () => {
     const manager = sessions();
     const ctx = await manager.start("s1");
@@ -847,7 +665,10 @@ describe("file transfer tools", () => {
     expect(downloads.cancel).not.toHaveBeenCalled();
   });
 
-  it("reconciles a download that completes while cancellation is being requested", async () => {
+  it.each([
+    ["cancel resolves", false],
+    ["cancel reports that completion won", true],
+  ])("reconciles a download that completes while %s", async (_label, cancelThrows) => {
     const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
     const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
     const onDeterminingFilename =
@@ -881,7 +702,7 @@ describe("file transfer tools", () => {
       onDeterminingFilename,
       search: vi.fn().mockResolvedValueOnce([initial]).mockResolvedValueOnce([complete]),
       cancel: vi.fn(async () => {
-        throw new Error("download already complete");
+        if (cancelThrows) throw new Error("download already complete");
       }),
       removeFile: vi.fn(async () => {}),
     };

apps/extension/src/tools/__tests__/frame-geometry.test.ts

@@ -169,7 +169,6 @@ describe("frame geometry projection", () => {
     expect(geometry).toMatchObject({
       topBounds: { x: 100, y: 100, width: 200, height: 100 },
       actionPoint: { x: 200, y: 150 },
-      targetActionPoint: { x: 100, y: 50 },
     });
   });
 
@@ -262,7 +261,6 @@ describe("frame geometry projection", () => {
     expect(geometry).toMatchObject({
       topBounds: { x: 100, y: 100, width: 200, height: 100 },
       actionPoint: { x: 200, y: 150 },
-      targetActionPoint: { x: 200, y: 150 },
     });
   });
 

apps/extension/src/tools/__tests__/interaction.test.ts

@@ -123,6 +123,29 @@ describe("handleClick", () => {
     expect(fake.cdp.send).not.toHaveBeenCalled();
   });
 
+  it("rejects screenshot-only visual surface refs before issuing CDP calls", async () => {
+    const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
+    const ctx = await sm.start("aa11");
+    ctx.refStore.set("e3", 1234, {
+      tabId: 4,
+      kind: "surface",
+      capabilities: ["screenshot"],
+    });
+    const fake = makeFakeCdp({});
+
+    const res = await handleClick(
+      sm,
+      { session_id: "aa11", ref: "@e3" },
+      { cdp: fake.cdp, tabsApi: fake.tabsApi },
+    );
+
+    expect(res).toMatchObject({
+      code: "permission_denied",
+      data: { reason: "ref_capability_denied", required_capability: "interact" },
+    });
+    expect(fake.cdp.send).not.toHaveBeenCalled();
+  });
+
   it("clicks by ref, computes the quad centre, dispatches three mouse events", async () => {
     const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
     const ctx = await sm.start("aa11");

apps/extension/src/tools/__tests__/observation.test.ts

@@ -36,6 +36,14 @@ function fakeAgentWindow(ids: number[]) {
 const TINY_PNG =
   "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR4nGNgYGBgAAAABQABh6FO1AAAAABJRU5ErkJggg==";
 
+function pngWithIhdrDimensions(width: number, height: number): string {
+  const bytes = Uint8Array.from(atob(TINY_PNG), (character) => character.charCodeAt(0));
+  const view = new DataView(bytes.buffer);
+  view.setUint32(16, width);
+  view.setUint32(20, height);
+  return btoa(String.fromCharCode(...bytes));
+}
+
 function makeScreenshotDeps(
   opts: {
     cdp?: CdpRunner;
@@ -86,6 +94,7 @@ describe("stripDataUrlPrefix", () => {
 describe("parsePngDimensions", () => {
   it("parses width/height from the IHDR chunk", () => {
     expect(parsePngDimensions(TINY_PNG)).toEqual({ width: 1, height: 1 });
+    expect(parsePngDimensions(pngWithIhdrDimensions(100, 80))).toEqual({ width: 100, height: 80 });
   });
   it("returns null on non-PNG input", () => {
     expect(parsePngDimensions("not-a-png-payload-just-random-base64-text-zzzzzzzzz")).toBeNull();
@@ -120,6 +129,20 @@ describe("handleScreenshot", () => {
     expect(get).not.toHaveBeenCalled();
   });
 
+  it("fails closed when a full-tab capture is not a valid PNG", async () => {
+    const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
+    await sm.start("aa11");
+    const res = await handleScreenshot(
+      sm,
+      { session_id: "aa11" },
+      makeScreenshotDeps({
+        captureVisibleTab: vi.fn(async () => "data:image/png;base64,bm90LWEtcG5n"),
+      }),
+    );
+
+    expect(res).toMatchObject({ code: "cdp_failed", message: /invalid PNG data/ });
+  });
+
   it("returns not_found when Agent Window has no active tab", async () => {
     const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
     await sm.start("aa11");
@@ -300,6 +323,165 @@ describe("handleScreenshot", () => {
     expect(clip).toMatchObject({ x: 10, y: 20, width: 100, height: 40 });
   });
 
+  it("captures a bounded visible crop for a visual surface without scrolling it", async () => {
+    const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
+    const ctx = await sm.start("aa11");
+    ctx.refStore.set("e5", 999, {
+      tabId: 7,
+      kind: "surface",
+      capabilities: ["screenshot"],
+    });
+    const { cdp, sent } = makeFakeCdp({
+      "Page.getLayoutMetrics": () => ({
+        cssLayoutViewport: { clientWidth: 4096, clientHeight: 4096 },
+      }),
+      "DOM.getContentQuads": () => ({ quads: [[0, 0, 4096, 0, 4096, 4096, 0, 4096]] }),
+      "Page.captureScreenshot": () => ({ data: TINY_PNG }),
+    });
+
+    const res = await handleScreenshot(
+      sm,
+      { session_id: "aa11", ref: "@e5", tab_id: 7 },
+      makeScreenshotDeps({
+        cdp,
+        get: vi.fn(async () => ({ id: 7, windowId: 100, active: false }) as chrome.tabs.Tab),
+        query: vi.fn(),
+        captureVisibleTab: vi.fn(),
+      }),
+    );
+
+    if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
+    expect(sent.some((call) => call.method === "DOM.scrollIntoViewIfNeeded")).toBe(false);
+    const clip = sent.find((call) => call.method === "Page.captureScreenshot")?.params as {
+      clip?: { width: number; height: number; scale: number };
+    };
+    expect(clip.clip).toMatchObject({ width: 4096, height: 4096 });
+    expect(clip.clip?.scale).toBeCloseTo(Math.sqrt(4_000_000 / (4096 * 4096)));
+  });
+
+  it("crops a visual surface screenshot to its observation-time visible region", async () => {
+    const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
+    const ctx = await sm.start("aa11");
+    ctx.refStore.set("e5", 999, {
+      tabId: 7,
+      kind: "surface",
+      visibleRect: { x: 25, y: 30, w: 50, h: 40 },
+    });
+    const { cdp, sent } = makeFakeCdp({
+      "Page.getLayoutMetrics": () => ({
+        cssLayoutViewport: { clientWidth: 800, clientHeight: 600 },
+      }),
+      "DOM.getContentQuads": () => ({ quads: [[0, 0, 200, 0, 200, 100, 0, 100]] }),
+      "Page.captureScreenshot": () => ({ data: pngWithIhdrDimensions(100, 80) }),
+    });
+
+    const res = await handleScreenshot(
+      sm,
+      { session_id: "aa11", ref: "@e5", tab_id: 7 },
+      makeScreenshotDeps({
+        cdp,
+        get: vi.fn(async () => ({ id: 7, windowId: 100, active: false }) as chrome.tabs.Tab),
+        query: vi.fn(),
+        captureVisibleTab: vi.fn(),
+      }),
+    );
+
+    if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
+    const clip = sent.find((call) => call.method === "Page.captureScreenshot")?.params as {
+      clip?: { x: number; y: number; width: number; height: number };
+    };
+    expect(clip.clip).toMatchObject({ x: 25, y: 30, width: 50, height: 40 });
+    expect(res).toMatchObject({ width: 100, height: 80 });
+  });
+
+  it("routes OOPIF surface geometry through its child session and captures top coordinates", async () => {
+    const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
+    const ctx = await sm.start("aa11");
+    ctx.refStore.set("e5", 999, {
+      tabId: 7,
+      frameId: "child-frame",
+      cdpSessionId: "child-session",
+      kind: "surface",
+      visibleRect: { x: 224, y: 346, w: 200, h: 80 },
+    });
+    const { cdp, sent } = makeFakeCdp({
+      "DOM.getBoxModel": () => ({
+        model: { content: [204, 306, 604, 306, 604, 506, 204, 506] },
+      }),
+      "Page.getLayoutMetrics": () => ({
+        cssLayoutViewport: { clientWidth: 1000, clientHeight: 800 },
+      }),
+      "Page.captureScreenshot": () => ({ data: TINY_PNG }),
+    });
+    cdp.getFrameGraph = vi.fn(async () => ({
+      rootFrameId: "main",
+      frames: [
+        { frameId: "main", target: { tabId: 7 } },
+        {
+          frameId: "child-frame",
+          parentFrameId: "main",
+          ownerBackendNodeId: 99,
+          target: { tabId: 7, sessionId: "child-session" },
+        },
+      ],
+    }));
+    const targetCalls: Array<{ sessionId?: string; method: string }> = [];
+    cdp.sendToTarget = vi.fn(async (target, method) => {
+      targetCalls.push({ sessionId: target.sessionId, method });
+      if (method === "DOM.getContentQuads") {
+        return { quads: [[10, 20, 110, 20, 110, 60, 10, 60]] };
+      }
+      if (method === "Page.getLayoutMetrics") {
+        return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } };
+      }
+      throw new Error(`unexpected child CDP call ${method}`);
+    }) as CdpRunner["sendToTarget"];
+
+    const res = await handleScreenshot(
+      sm,
+      { session_id: "aa11", ref: "@e5", tab_id: 7 },
+      makeScreenshotDeps({
+        cdp,
+        get: vi.fn(async () => ({ id: 7, windowId: 100, active: false }) as chrome.tabs.Tab),
+        query: vi.fn(),
+        captureVisibleTab: vi.fn(),
+      }),
+    );
+
+    if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
+    expect(targetCalls).toEqual([
+      { sessionId: "child-session", method: "DOM.getContentQuads" },
+      { sessionId: "child-session", method: "Page.getLayoutMetrics" },
+    ]);
+    const clip = sent.find((call) => call.method === "Page.captureScreenshot")?.params as {
+      clip?: { x: number; y: number; width: number; height: number };
+    };
+    expect(clip.clip).toMatchObject({ x: 224, y: 346, width: 200, height: 80 });
+  });
+
+  it("fails closed when a ref capture is not a valid PNG", async () => {
+    const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
+    const ctx = await sm.start("aa11");
+    ctx.refStore.set("e5", 999, { tabId: 7, kind: "surface" });
+    const { cdp } = makeFakeCdp({
+      "DOM.getContentQuads": () => ({ quads: [[0, 0, 50, 0, 50, 50, 0, 50]] }),
+      "Page.captureScreenshot": () => ({ data: "bm90LWEtcG5n" }),
+    });
+
+    const res = await handleScreenshot(
+      sm,
+      { session_id: "aa11", ref: "@e5", tab_id: 7 },
+      makeScreenshotDeps({
+        cdp,
+        get: vi.fn(async () => ({ id: 7, windowId: 100, active: false }) as chrome.tabs.Tab),
+        query: vi.fn(),
+        captureVisibleTab: vi.fn(),
+      }),
+    );
+
+    expect(res).toMatchObject({ code: "cdp_failed", message: /invalid PNG data/ });
+  });
+
   it("returns not_found for unknown ref", async () => {
     const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
     await sm.start("aa11");
@@ -2833,6 +3015,85 @@ describe("handleSnapshot", () => {
     );
   });
 
+  it("discovers rendered canvases only in observe and stores a screenshot-only ref", async () => {
+    const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
+    const ctx = await sm.start("aa11");
+    const root: CdpAxNode = {
+      nodeId: "1",
+      role: { type: "role", value: "RootWebArea" },
+      name: { type: "computedString", value: "Canvas app" },
+      backendDOMNodeId: 100,
+    };
+    const strings = ["body", "canvas", "static", "auto"];
+    const i = (value: string) => strings.indexOf(value);
+    const send = vi.fn(async (_tabId: number, method: string) => {
+      if (method === "Accessibility.enable" || method === "DOMSnapshot.enable") return {};
+      if (method === "Accessibility.getFullAXTree") return { nodes: [root] };
+      if (method === "Page.getLayoutMetrics") {
+        return { cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 } };
+      }
+      if (method === "DOMSnapshot.captureSnapshot") {
+        return {
+          strings,
+          documents: [
+            {
+              nodes: {
+                parentIndex: [-1, 0],
+                nodeName: [i("body"), i("canvas")],
+                backendNodeId: [100, 200],
+                attributes: [[], []],
+              },
+              layout: {
+                nodeIndex: [0, 1],
+                styles: [
+                  [i("static"), i("auto"), i("auto")],
+                  [i("static"), i("auto"), i("auto")],
+                ],
+                bounds: [
+                  [0, 0, 1000, 800],
+                  [20, 30, 800, 500],
+                ],
+                paintOrders: [0, 1],
+              },
+            },
+          ],
+        };
+      }
+      throw new Error(`unexpected CDP method ${method}`);
+    });
+    const deps = {
+      cdp: {
+        send: send as unknown as <T = unknown>(
+          tabId: number,
+          method: string,
+          params?: object,
+        ) => Promise<T>,
+        trackSessionTab: vi.fn(),
+      },
+      tabsApi: {
+        get: vi.fn(
+          async (tabId: number) => ({ id: tabId, windowId: 100, active: true }) as chrome.tabs.Tab,
+        ),
+        query: vi.fn(async () => [{ id: 4, windowId: 100, active: true } as chrome.tabs.Tab]),
+      },
+      conditionalSurfaceProbe: false,
+    };
+
+    const observed = await handleObserve(sm, { session_id: "aa11" }, deps);
+    if ("code" in observed) throw new Error(`unexpected error: ${JSON.stringify(observed)}`);
+    expect(observed.text).toContain('@e1 surface "canvas visual surface"');
+    expect(ctx.refStore.resolveEntry("e1")).toMatchObject({
+      backendNodeId: 200,
+      kind: "surface",
+      capabilities: ["screenshot"],
+    });
+
+    const snapshot = await handleSnapshot(sm, { session_id: "aa11" }, deps);
+    if ("code" in snapshot) throw new Error(`unexpected error: ${JSON.stringify(snapshot)}`);
+    expect(snapshot.text).not.toContain("visual surface");
+    expect(snapshot.ref_count).toBe(0);
+  });
+
   it("allows passive snapshots of explicit user-window tabs", async () => {
     const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
     await sm.start("aa11");

apps/extension/src/tools/__tests__/snapshot-ref.test.ts

@@ -24,10 +24,14 @@ describe("lookupSnapshotRef", () => {
     expect(lookupSnapshotRef(ctx, "@e3", 4)).toEqual({
       backendNodeId: 1234,
       refKey: "e3",
+      kind: "dom",
+      capabilities: ["interact", "screenshot"],
     });
     expect(lookupSnapshotRef(ctx, "e3", 4)).toEqual({
       backendNodeId: 1234,
       refKey: "e3",
+      kind: "dom",
+      capabilities: ["interact", "screenshot"],
     });
   });
 
@@ -53,6 +57,8 @@ describe("resolveSnapshotRef", () => {
       tabId: 4,
       frameId: "child-frame",
       cdpSessionId: "child-session",
+      kind: "dom",
+      capabilities: ["interact", "screenshot"],
     });
 
     const expected = {
@@ -60,6 +66,8 @@ describe("resolveSnapshotRef", () => {
       refKey: "e3",
       frameId: "child-frame",
       cdpSessionId: "child-session",
+      kind: "dom" as const,
+      capabilities: ["interact", "screenshot"] as const,
     };
     expect(lookupSnapshotRef(ctx, "@e3", 4)).toEqual(expected);
     expect(resolveSnapshotRef(ctx, "@e3", 4)).toEqual(expected);
@@ -105,6 +113,28 @@ describe("resolveSnapshotRef", () => {
     expect(resolveSnapshotRef(ctx, "@e3", 4)).toEqual({
       backendNodeId: 1234,
       refKey: "e3",
+      kind: "dom",
+      capabilities: ["interact", "screenshot"],
+    });
+  });
+
+  it("rejects an operation outside the ref's declared capabilities", async () => {
+    const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
+    const ctx = await sm.start("aa11");
+    ctx.refStore.set("e3", 1234, {
+      tabId: 4,
+      kind: "surface",
+      capabilities: ["screenshot"],
+    });
+
+    expect(resolveSnapshotRef(ctx, "@e3", 4, "interact")).toMatchObject({
+      code: "permission_denied",
+      message: "ref @e3 does not support interact",
+      data: {
+        reason: "ref_capability_denied",
+        required_capability: "interact",
+        kind: "surface",
+      },
     });
   });
 });

apps/extension/src/tools/download-capture.ts

@@ -9,6 +9,8 @@ import { type CdpRunner, isRpcError } from "./shared";
 
 const CORRELATION_GRACE_MS = 750;
 const UNIQUE_SETTLE_MS = 50;
+const CLEANUP_RECONCILE_MS = 1_000;
+const CLEANUP_POLL_MS = 25;
 const SIZE_POLL_MS = 250;
 
 type DeterminingFilenameListener = (
@@ -110,26 +112,35 @@ function captureError(
 
 async function cleanupClaimedDownload(downloads: DownloadsApi, downloadId: number): Promise<void> {
   const lookup = async () => (await downloads.search({ id: downloadId }))[0];
-  const item = await lookup();
+  let item = await lookup();
   if (!item || item.state === "interrupted") return;
   if (item.state === "complete") {
     await downloads.removeFile(downloadId);
     return;
   }
 
+  let cancelError: unknown;
   try {
     await downloads.cancel(downloadId);
-  } catch (cancelError) {
-    // Completion can win the race after the lookup but before cancellation.
-    // Reconcile against Chrome's authoritative state before declaring cleanup
-    // failed so every terminal state has one explicit cleanup path.
-    const reconciled = await lookup();
-    if (!reconciled || reconciled.state === "interrupted") return;
-    if (reconciled.state === "complete") {
+  } catch (error) {
+    cancelError = error;
+  }
+
+  const deadline = Date.now() + CLEANUP_RECONCILE_MS;
+  while (true) {
+    item = await lookup();
+    if (!item || item.state === "interrupted") return;
+    if (item.state === "complete") {
       await downloads.removeFile(downloadId);
       return;
     }
-    throw cancelError;
+    if (Date.now() >= deadline) {
+      if (cancelError instanceof Error) throw cancelError;
+      throw new Error("download remained in progress after cancellation");
+    }
+    await new Promise((resolve) =>
+      setTimeout(resolve, Math.min(CLEANUP_POLL_MS, Math.max(1, deadline - Date.now()))),
+    );
   }
 }
 

apps/extension/src/tools/file-drop-transaction.ts

@@ -1,319 +0,0 @@
-// One native file-drop transaction bound to an explicit page target. CDP drag
-// events are sent to the target that owns the node: root targets use top-level
-// coordinates, while OOPIF targets require coordinates local to their session.
-
-import type { RpcError, TransferEffectState } from "@/transport/types";
-import { transferError } from "./errors";
-import { resolveNodeGeometry } from "./frame-geometry";
-import type { ResolvedActionTarget } from "./interaction";
-import { type CdpRunner, isRpcError, sendToCdpTarget } from "./shared";
-import { BoundedWaitError, waitBounded } from "./transfer-transaction";
-
-const CLEANUP_TIMEOUT_MS = 1_000;
-
-type FileDropPhase = "resolve_target" | "drag_enter" | "drag_over" | "drop" | "cleanup";
-
-interface RuntimeReply {
-  result?: { value?: unknown; objectId?: string };
-  exceptionDetails?: { text?: string; exception?: { description?: string } };
-}
-
-interface DragData {
-  items: [];
-  files: string[];
-  dragOperationsMask: number;
-}
-
-export interface FileDropTransactionOptions {
-  cdp: CdpRunner;
-  actionTarget: ResolvedActionTarget;
-  files: string[];
-  timeoutMs: number;
-  signal?: AbortSignal;
-  withOverlayHidden: <T>(operation: () => Promise<T>) => Promise<T>;
-}
-
-export interface FileDropTransactionResult {
-  x: number;
-  y: number;
-}
-
-function failure(
-  code: RpcError["code"],
-  reason: "upload_mechanism_unsupported" | "file_drop_target_unavailable" | "file_drop_failed",
-  message: string,
-  effectState: TransferEffectState,
-  phase: FileDropPhase,
-): RpcError {
-  const error = transferError(code, reason, message, { effectState, phase });
-  error.data = { ...error.data, mechanism: "drop" };
-  return error;
-}
-
-function enrichFailure(
-  error: RpcError,
-  effectState: TransferEffectState,
-  phase: FileDropPhase,
-): RpcError {
-  return {
-    ...error,
-    data: {
-      ...error.data,
-      mechanism: "drop",
-      effect_state: effectState,
-      phase,
-    },
-  };
-}
-
-function messageOf(error: unknown): string {
-  return error instanceof Error ? error.message : String(error);
-}
-
-function methodUnavailable(error: unknown): boolean {
-  return /(?:method|command).*(?:not found|wasn't found|unsupported)|wasn't found|-32601/i.test(
-    messageOf(error),
-  );
-}
-
-function runtimeError(reply: RuntimeReply, fallback: string): Error | null {
-  if (!reply.exceptionDetails) return null;
-  return new Error(
-    reply.exceptionDetails.exception?.description ?? reply.exceptionDetails.text ?? fallback,
-  );
-}
-
-async function pointBelongsToTarget(
-  options: FileDropTransactionOptions,
-  x: number,
-  y: number,
-  objectGroup: string,
-): Promise<boolean> {
-  const target = options.actionTarget.cdpTarget;
-  const hit = await sendToCdpTarget<{ backendNodeId?: number; frameId?: string }>(
-    options.cdp,
-    target,
-    "DOM.getNodeForLocation",
-    { x: Math.round(x), y: Math.round(y), includeUserAgentShadowDOM: true },
-  );
-  if (typeof hit.backendNodeId !== "number") return false;
-  if (options.actionTarget.frameId && hit.frameId !== options.actionTarget.frameId) return false;
-
-  const [resolvedTarget, resolvedHit] = await Promise.all([
-    sendToCdpTarget<{ object?: { objectId?: string } }>(options.cdp, target, "DOM.resolveNode", {
-      backendNodeId: options.actionTarget.backendNodeId,
-      objectGroup,
-    }),
-    sendToCdpTarget<{ object?: { objectId?: string } }>(options.cdp, target, "DOM.resolveNode", {
-      backendNodeId: hit.backendNodeId,
-      objectGroup,
-    }),
-  ]);
-  const targetObjectId = resolvedTarget.object?.objectId;
-  const hitObjectId = resolvedHit.object?.objectId;
-  if (!targetObjectId || !hitObjectId) return false;
-
-  const reply = await sendToCdpTarget<RuntimeReply>(options.cdp, target, "Runtime.callFunctionOn", {
-    objectId: targetObjectId,
-    functionDeclaration: `function(hit) {
-        for (let node = hit; node; ) {
-          if (node === this) return true;
-          if (this.contains?.(node)) return true;
-          const root = node.getRootNode?.();
-          node = node.parentNode || (root && root !== node ? root.host : null);
-        }
-        return false;
-      }`,
-    arguments: [{ objectId: hitObjectId }],
-    returnByValue: true,
-  });
-  const error = runtimeError(reply, "failed to verify file-drop target");
-  if (error) throw error;
-  return reply.result?.value === true;
-}
-
-export async function uploadThroughFileDrop(
-  options: FileDropTransactionOptions,
-): Promise<FileDropTransactionResult | RpcError> {
-  const deadline = Date.now() + options.timeoutMs;
-  const target = options.actionTarget.cdpTarget;
-  const objectGroup = `bsk-file-drop-${crypto.randomUUID()}`;
-  const dragData: DragData = {
-    items: [],
-    files: options.files,
-    dragOperationsMask: 1,
-  };
-  let dragStarted = false;
-  let dropSent = false;
-  let dragPoint = { x: 0, y: 0 };
-  let cleanupFailed = false;
-  let outcome: FileDropTransactionResult | RpcError = failure(
-    "protocol_error",
-    "file_drop_failed",
-    "file-drop transaction ended without an outcome",
-    "none",
-    "cleanup",
-  );
-
-  try {
-    const geometry = await waitBounded(
-      resolveNodeGeometry(
-        options.cdp,
-        options.actionTarget.tab.tabId,
-        {
-          target,
-          backendNodeId: options.actionTarget.backendNodeId,
-          ...(options.actionTarget.frameId ? { frameId: options.actionTarget.frameId } : {}),
-        },
-        { scrollIntoView: true },
-      ),
-      deadline,
-      options.signal,
-      "resolving file-drop target timed out",
-    );
-    if (isRpcError(geometry)) {
-      outcome = enrichFailure(geometry, "none", "resolve_target");
-      return outcome;
-    }
-    const point = geometry.targetActionPoint;
-    dragPoint = point;
-
-    outcome = await options.withOverlayHidden(async () => {
-      try {
-        try {
-          const belongs = await waitBounded(
-            pointBelongsToTarget(options, point.x, point.y, objectGroup),
-            deadline,
-            options.signal,
-            "verifying file-drop target timed out",
-          );
-          if (!belongs) {
-            return failure(
-              "permission_denied",
-              "file_drop_target_unavailable",
-              "file-drop target is not the topmost element at its action point",
-              "none",
-              "resolve_target",
-            );
-          }
-        } catch (error) {
-          return failure(
-            error instanceof BoundedWaitError ? "timeout" : "cdp_failed",
-            "file_drop_target_unavailable",
-            messageOf(error),
-            "none",
-            "resolve_target",
-          );
-        }
-
-        for (const [type, phase] of [
-          ["dragEnter", "drag_enter"],
-          ["dragOver", "drag_over"],
-        ] as const) {
-          try {
-            if (type === "dragEnter") dragStarted = true;
-            await waitBounded(
-              sendToCdpTarget(options.cdp, target, "Input.dispatchDragEvent", {
-                type,
-                x: point.x,
-                y: point.y,
-                data: dragData,
-              }),
-              deadline,
-              options.signal,
-              `${type} timed out`,
-            );
-          } catch (error) {
-            const unsupported = methodUnavailable(error);
-            if (type === "dragEnter" && unsupported) dragStarted = false;
-            return failure(
-              unsupported
-                ? "unsupported"
-                : error instanceof BoundedWaitError
-                  ? "timeout"
-                  : "cdp_failed",
-              unsupported ? "upload_mechanism_unsupported" : "file_drop_failed",
-              messageOf(error),
-              "none",
-              phase,
-            );
-          }
-        }
-
-        try {
-          dropSent = true;
-          await waitBounded(
-            sendToCdpTarget(options.cdp, target, "Input.dispatchDragEvent", {
-              type: "drop",
-              x: point.x,
-              y: point.y,
-              data: dragData,
-            }),
-            deadline,
-            options.signal,
-            "file drop timed out",
-          );
-          return { x: point.x, y: point.y };
-        } catch (error) {
-          outcome = failure(
-            error instanceof BoundedWaitError ? "timeout" : "cdp_failed",
-            "file_drop_failed",
-            messageOf(error),
-            "unknown",
-            "drop",
-          );
-          return outcome;
-        }
-      } finally {
-        if (dragStarted && !dropSent) {
-          try {
-            await waitBounded(
-              sendToCdpTarget(options.cdp, target, "Input.dispatchDragEvent", {
-                type: "dragCancel",
-                x: dragPoint.x,
-                y: dragPoint.y,
-                data: dragData,
-              }),
-              Date.now() + CLEANUP_TIMEOUT_MS,
-              undefined,
-              "cancelling file drag timed out",
-            );
-          } catch {
-            cleanupFailed = true;
-          }
-        }
-      }
-    });
-    return outcome;
-  } catch (error) {
-    outcome = failure(
-      error instanceof BoundedWaitError ? "timeout" : "cdp_failed",
-      "file_drop_target_unavailable",
-      messageOf(error),
-      "none",
-      "resolve_target",
-    );
-    return outcome;
-  } finally {
-    try {
-      await waitBounded(
-        sendToCdpTarget(options.cdp, target, "Runtime.releaseObjectGroup", { objectGroup }),
-        Date.now() + CLEANUP_TIMEOUT_MS,
-        undefined,
-        "releasing file-drop object group timed out",
-      );
-    } catch {
-      cleanupFailed = true;
-    }
-
-    const effect = isRpcError(outcome)
-      ? (outcome.data?.effect_state as TransferEffectState | undefined)
-      : "committed";
-    if (effect === "unknown" || cleanupFailed) {
-      await options.cdp.detach?.(target.tabId);
-    }
-    if (cleanupFailed && isRpcError(outcome)) {
-      outcome.data = { ...outcome.data, cleanup_state: "failed" };
-    }
-  }
-}

apps/extension/src/tools/file-input-transaction.ts

@@ -7,9 +7,9 @@ import type { ClickResult, RpcError, TransferEffectState } from "@/transport/typ
 import { transferError } from "./errors";
 import type { ResolvedActionTarget } from "./interaction";
 import { type CdpRunner, isRpcError, sendToCdpTarget } from "./shared";
-import { BoundedWaitError, remainingMs, waitBounded } from "./transfer-transaction";
 
 const CLEANUP_TIMEOUT_MS = 1_000;
+const DETACH_TIMEOUT_MS = 500;
 const ACTIVATION_GRACE_MS = 1_000;
 const PROBE_INTERVAL_MS = 20;
 
@@ -46,6 +46,51 @@ export interface FileInputTransactionResult {
   multiple: boolean;
 }
 
+class BoundedWaitError extends Error {
+  constructor(
+    readonly kind: "timeout" | "aborted",
+    message: string,
+  ) {
+    super(message);
+  }
+}
+
+function remainingMs(deadline: number): number {
+  return Math.max(0, deadline - Date.now());
+}
+
+async function waitBounded<T>(
+  operation: () => Promise<T>,
+  deadline: number,
+  signal: AbortSignal | undefined,
+  timeoutMessage: string,
+): Promise<T> {
+  const remaining = remainingMs(deadline);
+  if (signal?.aborted) throw new BoundedWaitError("aborted", "upload transaction aborted");
+  if (remaining === 0) throw new BoundedWaitError("timeout", timeoutMessage);
+
+  let timer: ReturnType<typeof setTimeout> | undefined;
+  let onAbort: (() => void) | undefined;
+  const boundary = new Promise<never>((_resolve, reject) => {
+    timer = setTimeout(() => reject(new BoundedWaitError("timeout", timeoutMessage)), remaining);
+    if (signal) {
+      onAbort = () => reject(new BoundedWaitError("aborted", "upload transaction aborted"));
+      signal.addEventListener("abort", onAbort, { once: true });
+    }
+  });
+  try {
+    const work = Promise.resolve().then(() => {
+      if (signal?.aborted) throw new BoundedWaitError("aborted", "upload transaction aborted");
+      if (remainingMs(deadline) === 0) throw new BoundedWaitError("timeout", timeoutMessage);
+      return operation();
+    });
+    return await Promise.race([work, boundary]);
+  } finally {
+    if (timer) clearTimeout(timer);
+    if (signal && onAbort) signal.removeEventListener("abort", onAbort);
+  }
+}
+
 function runtimeError(reply: RuntimeReply, fallback: string): Error | null {
   if (!reply.exceptionDetails) return null;
   return new Error(
@@ -83,13 +128,18 @@ function sameTarget(
 
 async function delay(ms: number, signal?: AbortSignal): Promise<void> {
   await waitBounded(
-    new Promise((resolve) => setTimeout(resolve, ms)),
+    () => new Promise((resolve) => setTimeout(resolve, ms)),
     Date.now() + ms + 1,
     signal,
     "upload activation probe timed out",
   );
 }
 
+function waitErrorCode(error: unknown): RpcError["code"] {
+  if (!(error instanceof BoundedWaitError)) return "cdp_failed";
+  return error.kind === "aborted" ? "cancelled" : "timeout";
+}
+
 async function callOnTrigger<T = RuntimeReply>(
   options: FileInputTransactionOptions,
   objectId: string,
@@ -114,8 +164,8 @@ export async function uploadThroughActivatedFileInput(
   const objectGroup = `bsk-upload-${crypto.randomUUID()}`;
   const stateKey = `__bskUpload_${crypto.randomUUID().replaceAll("-", "")}`;
   const chooserEvents: ChooserEvent[] = [];
-  let interceptionArmed = false;
-  let probeArmed = false;
+  let interceptionState: "not_started" | "possibly_active" | "active" = "not_started";
+  let probeState: "not_started" | "possibly_active" | "active" = "not_started";
   let triggerObjectId: string | undefined;
   let outcome: FileInputTransactionResult | RpcError = transferFailure(
     "protocol_error",
@@ -133,18 +183,20 @@ export async function uploadThroughActivatedFileInput(
 
   try {
     try {
+      interceptionState = "possibly_active";
       await waitBounded(
-        sendToCdpTarget(options.cdp, target, "Page.setInterceptFileChooserDialog", {
-          enabled: true,
-        }),
+        () =>
+          sendToCdpTarget(options.cdp, target, "Page.setInterceptFileChooserDialog", {
+            enabled: true,
+          }),
         deadline,
         options.signal,
         "arming native file chooser interception timed out",
       );
-      interceptionArmed = true;
+      interceptionState = "active";
     } catch (err) {
       outcome = transferFailure(
-        err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
+        waitErrorCode(err),
         "file_input_probe_failed",
         err instanceof Error ? err.message : String(err),
         "none",
@@ -155,15 +207,16 @@ export async function uploadThroughActivatedFileInput(
 
     try {
       const resolved = await waitBounded(
-        sendToCdpTarget<{ object?: { objectId?: string } }>(
-          options.cdp,
-          target,
-          "DOM.resolveNode",
-          {
-            backendNodeId: options.actionTarget.backendNodeId,
-            objectGroup,
-          },
-        ),
+        () =>
+          sendToCdpTarget<{ object?: { objectId?: string } }>(
+            options.cdp,
+            target,
+            "DOM.resolveNode",
+            {
+              backendNodeId: options.actionTarget.backendNodeId,
+              objectGroup,
+            },
+          ),
         deadline,
         options.signal,
         "resolving upload trigger timed out",
@@ -171,11 +224,13 @@ export async function uploadThroughActivatedFileInput(
       triggerObjectId = resolved.object?.objectId;
       if (!triggerObjectId) throw new Error("DOM.resolveNode returned no trigger objectId");
 
+      probeState = "possibly_active";
       const armed = await waitBounded(
-        callOnTrigger<RuntimeReply>(
-          options,
-          triggerObjectId,
-          `function(key) {
+        () =>
+          callOnTrigger<RuntimeReply>(
+            options,
+            triggerObjectId as string,
+            `function(key) {
             const doc = this.ownerDocument;
             const owner = doc.defaultView;
             if (!owner) return false;
@@ -193,9 +248,9 @@ export async function uploadThroughActivatedFileInput(
             doc.addEventListener("click", state.listener, true);
             return true;
           }`,
-          [stateKey],
-          true,
-        ),
+            [stateKey],
+            true,
+          ),
         deadline,
         options.signal,
         "arming frame-scoped file input probe timed out",
@@ -204,10 +259,10 @@ export async function uploadThroughActivatedFileInput(
       if (armError || armed.result?.value !== true) {
         throw armError ?? new Error("file input probe did not arm");
       }
-      probeArmed = true;
+      probeState = "active";
     } catch (err) {
       outcome = transferFailure(
-        err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
+        waitErrorCode(err),
         "file_input_probe_failed",
         err instanceof Error ? err.message : String(err),
         "none",
@@ -219,14 +274,14 @@ export async function uploadThroughActivatedFileInput(
     let click: ClickResult | RpcError;
     try {
       click = await waitBounded(
-        options.trigger(),
+        options.trigger,
         deadline,
         options.signal,
         "upload trigger timed out",
       );
     } catch (err) {
       outcome = transferFailure(
-        err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
+        waitErrorCode(err),
         "file_input_probe_failed",
         err instanceof Error ? err.message : String(err),
         "none",
@@ -244,17 +299,23 @@ export async function uploadThroughActivatedFileInput(
       let summary: { count: number; multiple: boolean } = { count: 0, multiple: false };
       while (Date.now() < activationDeadline) {
         if (chooserEvents.length > 0) break;
-        const reply = await callOnTrigger<RuntimeReply>(
-          options,
-          triggerObjectId,
-          `function(key) {
+        const reply = await waitBounded(
+          () =>
+            callOnTrigger<RuntimeReply>(
+              options,
+              triggerObjectId as string,
+              `function(key) {
             const state = this.ownerDocument.defaultView?.[key];
             return state
               ? { count: state.inputs.length, multiple: state.inputs[0]?.multiple === true }
               : { count: 0, multiple: false };
           }`,
-          [stateKey],
-          true,
+              [stateKey],
+              true,
+            ),
+          activationDeadline,
+          options.signal,
+          "inspecting activated file input timed out",
         );
         const summaryError = runtimeError(reply, "failed to inspect activated file input");
         if (summaryError) throw summaryError;
@@ -317,21 +378,33 @@ export async function uploadThroughActivatedFileInput(
           );
           return outcome;
         }
-        const input = await callOnTrigger<RuntimeReply>(
-          options,
-          triggerObjectId,
-          `function(key) { return this.ownerDocument.defaultView?.[key]?.inputs[0]; }`,
-          [stateKey],
-          false,
+        const input = await waitBounded(
+          () =>
+            callOnTrigger<RuntimeReply>(
+              options,
+              triggerObjectId as string,
+              `function(key) { return this.ownerDocument.defaultView?.[key]?.inputs[0]; }`,
+              [stateKey],
+              false,
+            ),
+          deadline,
+          options.signal,
+          "resolving activated file input timed out",
         );
         const inputError = runtimeError(input, "failed to resolve activated file input object");
         if (inputError) throw inputError;
         if (!input.result?.objectId) throw new Error("activated file input returned no objectId");
-        const described = await sendToCdpTarget<{ node?: { backendNodeId?: number } }>(
-          options.cdp,
-          target,
-          "DOM.describeNode",
-          { objectId: input.result.objectId },
+        const described = await waitBounded(
+          () =>
+            sendToCdpTarget<{ node?: { backendNodeId?: number } }>(
+              options.cdp,
+              target,
+              "DOM.describeNode",
+              { objectId: input.result?.objectId },
+            ),
+          deadline,
+          options.signal,
+          "describing activated file input timed out",
         );
         backendNodeId = described.node?.backendNodeId;
         if (typeof backendNodeId !== "number") {
@@ -350,17 +423,18 @@ export async function uploadThroughActivatedFileInput(
 
       try {
         await waitBounded(
-          sendToCdpTarget(options.cdp, target, "DOM.setFileInputFiles", {
-            files: options.files,
-            backendNodeId,
-          }),
+          () =>
+            sendToCdpTarget(options.cdp, target, "DOM.setFileInputFiles", {
+              files: options.files,
+              backendNodeId,
+            }),
           deadline,
           options.signal,
           "setting file input files timed out",
         );
       } catch (err) {
         outcome = transferFailure(
-          err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
+          waitErrorCode(err),
           "set_file_input_failed",
           err instanceof Error ? err.message : String(err),
           "unknown",
@@ -372,7 +446,7 @@ export async function uploadThroughActivatedFileInput(
       return outcome;
     } catch (err) {
       outcome = transferFailure(
-        err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
+        waitErrorCode(err),
         "file_input_probe_failed",
         err instanceof Error ? err.message : String(err),
         "none",
@@ -382,61 +456,70 @@ export async function uploadThroughActivatedFileInput(
     }
   } finally {
     chooserSubscription?.dispose();
-    let cleanupFailed = false;
-    if (probeArmed && triggerObjectId) {
+    const cleanupDeadline = Date.now() + CLEANUP_TIMEOUT_MS;
+    const cleanupTasks: Promise<boolean>[] = [];
+    const cleanup = async (operation: () => Promise<unknown>, message: string) => {
       try {
-        await waitBounded(
-          callOnTrigger(
-            options,
-            triggerObjectId,
-            `function(key) {
+        await waitBounded(operation, cleanupDeadline, undefined, message);
+        return true;
+      } catch {
+        return false;
+      }
+    };
+    if (probeState !== "not_started" && triggerObjectId) {
+      cleanupTasks.push(
+        cleanup(
+          () =>
+            callOnTrigger(
+              options,
+              triggerObjectId as string,
+              `function(key) {
               const owner = this.ownerDocument.defaultView;
               const state = owner?.[key];
               if (state?.listener) this.ownerDocument.removeEventListener("click", state.listener, true);
               if (owner) delete owner[key];
             }`,
-            [stateKey],
-            true,
-          ),
-          Date.now() + CLEANUP_TIMEOUT_MS,
-          undefined,
+              [stateKey],
+              true,
+            ),
           "cleaning file input probe timed out",
-        );
-      } catch {
-        cleanupFailed = true;
-      }
-    }
-    if (interceptionArmed) {
-      try {
-        await waitBounded(
-          sendToCdpTarget(options.cdp, target, "Page.setInterceptFileChooserDialog", {
-            enabled: false,
-            cancel: true,
-          }),
-          Date.now() + CLEANUP_TIMEOUT_MS,
-          undefined,
-          "disabling file chooser interception timed out",
-        );
-      } catch {
-        cleanupFailed = true;
-      }
-    }
-    try {
-      await waitBounded(
-        sendToCdpTarget(options.cdp, target, "Runtime.releaseObjectGroup", { objectGroup }),
-        Date.now() + CLEANUP_TIMEOUT_MS,
-        undefined,
-        "releasing upload object group timed out",
+        ),
       );
-    } catch {
-      cleanupFailed = true;
+    }
+    if (interceptionState !== "not_started") {
+      cleanupTasks.push(
+        cleanup(
+          () =>
+            sendToCdpTarget(options.cdp, target, "Page.setInterceptFileChooserDialog", {
+              enabled: false,
+              cancel: true,
+            }),
+          "disabling file chooser interception timed out",
+        ),
+      );
+    }
+    let cleanupFailed = (await Promise.all(cleanupTasks)).some((completed) => !completed);
+    if (!cleanupFailed) {
+      cleanupFailed = !(await cleanup(
+        () => sendToCdpTarget(options.cdp, target, "Runtime.releaseObjectGroup", { objectGroup }),
+        "releasing upload object group timed out",
+      ));
     }
 
     const effect = isRpcError(outcome)
       ? (outcome.data?.effect_state as TransferEffectState | undefined)
       : "committed";
     if (effect === "unknown" || cleanupFailed) {
-      await options.cdp.detach?.(target.tabId);
+      try {
+        await waitBounded(
+          () => options.cdp.detach?.(target.tabId) ?? Promise.resolve(),
+          Date.now() + DETACH_TIMEOUT_MS,
+          undefined,
+          "detaching upload target timed out",
+        );
+      } catch {
+        cleanupFailed = true;
+      }
     }
     if (cleanupFailed && isRpcError(outcome)) {
       outcome.data = { ...outcome.data, cleanup_state: "failed" };

apps/extension/src/tools/frame-geometry.ts

@@ -29,10 +29,7 @@ export interface NodeAddress {
 export interface ResolvedNodeGeometry {
   topVisibleRegions: Region;
   topBounds: ViewportRect;
-  /** Point in the top-level tab viewport, used by root-target input events. */
   actionPoint: Point;
-  /** Point in the addressed CDP target's viewport, used by OOPIF-local input events. */
-  targetActionPoint: Point;
 }
 
 function geometryError(message: string): RpcError {
@@ -265,30 +262,11 @@ export async function resolveNodeGeometry(
     if (isRpcError(localRegion)) return localRegion;
 
     let topVisibleRegions: Region;
-    let targetActionPoint: Point | null = null;
     if (address.frameId && graph) {
       const projection = await resolveFrameProjection(cdp, graph, address.frameId);
       if (!projection)
         return geometryError(`could not resolve frame geometry for ${address.frameId}`);
       topVisibleRegions = projectRegionToViewport(localRegion, projection);
-      if (address.target.sessionId) {
-        const localViewport = projection.edges[0]?.sourceViewport ?? projection.topViewport;
-        const localVisibleRegions = localRegion
-          .map((polygon) =>
-            clipPolygon(
-              polygon,
-              rectPolygon({
-                x: 0,
-                y: 0,
-                w: localViewport.width,
-                h: localViewport.height,
-              }),
-            ),
-          )
-          .filter((polygon) => polygon.length >= 3);
-        const localActionRegion = largestRegion(localVisibleRegions);
-        targetActionPoint = localActionRegion ? polygonCentroid(localActionRegion) : null;
-      }
     } else {
       const viewport = await targetViewport(cdp, address.target);
       if (!viewport) return geometryError("could not resolve top viewport geometry");
@@ -305,11 +283,7 @@ export async function resolveNodeGeometry(
     if (!topBounds || !actionPoint) {
       return { code: "permission_denied", message: "element not visible" };
     }
-    if (address.target.sessionId && !targetActionPoint) {
-      return { code: "permission_denied", message: "element not visible in its target" };
-    }
-    targetActionPoint ??= actionPoint;
-    return { topVisibleRegions, topBounds, actionPoint, targetActionPoint };
+    return { topVisibleRegions, topBounds, actionPoint };
   } catch (error) {
     return geometryError(error instanceof Error ? error.message : String(error));
   }

apps/extension/src/tools/interaction.ts

@@ -169,7 +169,7 @@ export async function resolveBackendNode(
     };
   }
   if (hasRef) {
-    const resolved = resolveSnapshotRef(ctx, params.ref as string, target.tabId);
+    const resolved = resolveSnapshotRef(ctx, params.ref as string, target.tabId, "interact");
     if (isRpcError(resolved)) return resolved;
     return {
       backendNodeId: resolved.backendNodeId,

apps/extension/src/tools/observation.ts

@@ -8,6 +8,7 @@ import {
   applyVomInteractionRecovery,
   type CondSurface,
   isVomReferenceNode,
+  type Rect,
   renderVom,
   type VomNode,
   type VomOptions,
@@ -17,7 +18,7 @@ import { ChromiumCdp } from "@/browser-driver/chromium-cdp";
 import type { CdpTarget } from "@/browser-driver/frame-graph";
 import {
   type CaptureSuppressSendToTab,
-  withExtensionOverlayHidden,
+  withOverlaysHiddenForCapture,
 } from "@/lib/capture-suppress-bridge";
 import type { SessionContext, SessionManager } from "@/session-manager/manager";
 import type {
@@ -62,6 +63,11 @@ import {
   type CaptureVomObservationResult,
   projectRecordSafeObservation,
 } from "./vom/record-safe-observation";
+import {
+  clusterRenderedSurfaces,
+  discoverRenderedSurfaces,
+  projectRenderedSurfaces,
+} from "./vom/rendered-surfaces";
 import {
   buildSemanticGraph,
   buildSemanticVomScene,
@@ -93,6 +99,9 @@ export const normaliseRef = sharedNormaliseRef;
 // screenshot — `tool.screenshot`
 // ---------------------------------------------------------------------------
 
+const VISUAL_SURFACE_SCREENSHOT_MAX_EDGE = 2_048;
+const VISUAL_SURFACE_SCREENSHOT_MAX_PIXELS = 4_000_000;
+
 /**
  * Strip the `data:image/...;base64,` prefix from a Chrome
  * `captureVisibleTab` dataURL and return the raw base64 payload.
@@ -107,8 +116,8 @@ export function stripDataUrlPrefix(dataUrl: string): string {
 
 /**
  * Parse a PNG's IHDR chunk and return `(width, height)`. Returns
- * `null` on any malformed input so callers fall back to `0/0` instead
- * of throwing.
+ * `null` on malformed input so callers can reject dimensions that are
+ * not authoritative instead of guessing from a CSS-space rectangle.
  *
  * PNG layout: 8-byte signature, then a 4-byte length, 4-byte type
  * ("IHDR"), then the chunk data — width is bytes 16-19 BE, height is
@@ -120,9 +129,9 @@ export function parsePngDimensions(base64: string): { width: number; height: num
     const head = base64.length > 64 ? base64.slice(0, 64) : base64;
     const bin = atob(head);
     if (bin.length < 24) return null;
-    if (bin.charCodeAt(0) !== 0x89 || bin.charCodeAt(1) !== 0x50 || bin.charCodeAt(2) !== 0x4e) {
-      return null;
-    }
+    const expectedSignature = [0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a];
+    if (expectedSignature.some((byte, index) => bin.charCodeAt(index) !== byte)) return null;
+    if (bin.slice(12, 16) !== "IHDR") return null;
     const u32 = (off: number) =>
       (bin.charCodeAt(off) << 24) |
       (bin.charCodeAt(off + 1) << 16) |
@@ -185,6 +194,8 @@ async function captureElementScreenshot(
   target: CdpTarget,
   backendNodeId: number,
   frameId?: string,
+  visualSurface = false,
+  observedVisibleRect?: Rect,
   signal?: AbortSignal,
 ): Promise<{ image_base64: string; width: number; height: number } | RpcError> {
   if (signal?.aborted) return cancelled("screenshot");
@@ -192,11 +203,36 @@ async function captureElementScreenshot(
     cdp,
     tabId,
     { target, backendNodeId, ...(frameId ? { frameId } : {}) },
-    { scrollIntoView: true },
+    { scrollIntoView: !visualSurface },
   );
   if (isRpcError(geometry)) return geometry;
   if (signal?.aborted) return cancelled("screenshot");
-  const rect = geometry.topBounds;
+  const liveRect = geometry.topBounds;
+  const rect =
+    visualSurface && observedVisibleRect
+      ? (() => {
+          const x = Math.max(liveRect.x, observedVisibleRect.x);
+          const y = Math.max(liveRect.y, observedVisibleRect.y);
+          const right = Math.min(
+            liveRect.x + liveRect.width,
+            observedVisibleRect.x + observedVisibleRect.w,
+          );
+          const bottom = Math.min(
+            liveRect.y + liveRect.height,
+            observedVisibleRect.y + observedVisibleRect.h,
+          );
+          return right > x && bottom > y ? { x, y, width: right - x, height: bottom - y } : null;
+        })()
+      : liveRect;
+  if (!rect) return { code: "permission_denied", message: "surface is no longer visible" };
+  const scale = visualSurface
+    ? Math.min(
+        1,
+        VISUAL_SURFACE_SCREENSHOT_MAX_EDGE / rect.width,
+        VISUAL_SURFACE_SCREENSHOT_MAX_EDGE / rect.height,
+        Math.sqrt(VISUAL_SURFACE_SCREENSHOT_MAX_PIXELS / (rect.width * rect.height)),
+      )
+    : 1;
 
   try {
     const shot = await cdp.send<{ data?: string }>(tabId, "Page.captureScreenshot", {
@@ -206,7 +242,7 @@ async function captureElementScreenshot(
         y: rect.y,
         width: rect.width,
         height: rect.height,
-        scale: 1,
+        scale,
       },
     });
     if (signal?.aborted) return cancelled("screenshot");
@@ -214,10 +250,10 @@ async function captureElementScreenshot(
     if (!image_base64) {
       return { code: "cdp_failed", message: "Page.captureScreenshot returned no data" };
     }
-    const dims = parsePngDimensions(image_base64) ?? {
-      width: Math.round(rect.width),
-      height: Math.round(rect.height),
-    };
+    const dims = parsePngDimensions(image_base64);
+    if (!dims) {
+      return { code: "cdp_failed", message: "Page.captureScreenshot returned invalid PNG data" };
+    }
     return { image_base64, width: dims.width, height: dims.height };
   } catch (err) {
     return {
@@ -313,7 +349,7 @@ export async function handleScreenshot(
     if (!deps.cdp) {
       return { code: "cdp_failed", message: "screenshot ref capture requires CDP" };
     }
-    const node = resolveSnapshotRef(ctx, ref, target.tabId);
+    const node = resolveSnapshotRef(ctx, ref, target.tabId, "screenshot");
     if (isRpcError(node)) return node;
     if (signal?.aborted) return cancelled("screenshot");
     deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
@@ -324,7 +360,7 @@ export async function handleScreenshot(
       tabId: target.tabId,
       ...(node.cdpSessionId ? { sessionId: node.cdpSessionId } : {}),
     };
-    const captured = await withExtensionOverlayHidden(
+    const captured = await withOverlaysHiddenForCapture(
       target.tabId,
       () =>
         captureElementScreenshot(
@@ -333,6 +369,8 @@ export async function handleScreenshot(
           nodeTarget,
           node.backendNodeId,
           node.frameId,
+          node.kind === "surface",
+          node.visibleRect,
           signal,
         ),
       deps.sendToTab,
@@ -356,7 +394,7 @@ export async function handleScreenshot(
     );
   }
 
-  const captured = await withExtensionOverlayHidden(
+  const captured = await withOverlaysHiddenForCapture(
     target.tabId,
     () => captureFullTabPng(deps, ctx, target, signal),
     deps.sendToTab,
@@ -364,7 +402,10 @@ export async function handleScreenshot(
   if (isRpcError(captured)) return captured;
   if (signal?.aborted) return cancelled("screenshot");
   const image_base64 = captured;
-  const dims = parsePngDimensions(image_base64) ?? { width: 0, height: 0 };
+  const dims = parsePngDimensions(image_base64);
+  if (!dims) {
+    return { code: "cdp_failed", message: "screenshot capture returned invalid PNG data" };
+  }
   return withShotDialogs({
     image_base64,
     width: dims.width,
@@ -1083,6 +1124,7 @@ async function runHoverProbes(
 
 export interface CaptureVomObservationOptions extends VomOptions {
   conditionalSurfaceProbe?: boolean;
+  renderedSurfaceDiscovery?: boolean;
   hoverProbeBypassOverlay?: (tabId: number, enabled: boolean) => Promise<void>;
   signal?: AbortSignal;
 }
@@ -1131,7 +1173,20 @@ export async function captureVomObservation(
     captured,
     hoverProbes.surfaceProbes,
   );
-  const rendered = renderVom(decoratedScene, {
+  const observedScene = options.renderedSurfaceDiscovery
+    ? projectRenderedSurfaces(
+        decoratedScene,
+        normalizedDocuments,
+        clusterRenderedSurfaces(
+          discoverRenderedSurfaces(
+            normalizedDocuments,
+            captured.viewport,
+            captured.excludedBackendNodeIds,
+          ),
+        ),
+      )
+    : decoratedScene;
+  const rendered = renderVom(observedScene, {
     maxDepth: options.maxDepth,
     maxTokens: options.maxTokens,
     redactValues: options.redactValues,
@@ -1186,6 +1241,7 @@ async function handleVomObservation(
       maxTokens: params.max_tokens,
       activeRegionPolicy: true,
       conditionalSurfaceProbe: effectiveConditionalSurfaceProbe,
+      renderedSurfaceDiscovery: toolName === "observe",
       hoverProbeBypassOverlay: deps.hoverProbeBypassOverlay,
       signal,
     });
@@ -1203,6 +1259,9 @@ async function handleVomObservation(
             tabId: target.tabId,
             ...(ref.frameId ? { frameId: ref.frameId } : {}),
             ...(refTarget?.sessionId ? { cdpSessionId: refTarget.sessionId } : {}),
+            ...(ref.visibleRect ? { visibleRect: ref.visibleRect } : {}),
+            ...(ref.kind ? { kind: ref.kind } : {}),
+            ...(ref.capabilities ? { capabilities: ref.capabilities } : {}),
           },
         ] as const;
       }),

apps/extension/src/tools/snapshot-ref.ts

@@ -2,8 +2,9 @@
 // compound CDP node identity via the session RefStore, and build stable
 // `ref_not_found` errors for hard-failure tool paths.
 
+import type { Rect } from "@browser-skill/vom";
 import type { SessionContext } from "@/session-manager/manager";
-import { normaliseRef } from "@/session-manager/ref-store";
+import { normaliseRef, type RefCapability, type RefTargetKind } from "@/session-manager/ref-store";
 import type { RpcError } from "@/transport/types";
 import { rpcError } from "./errors";
 
@@ -12,6 +13,9 @@ export interface SnapshotRefLookup {
   refKey: string;
   frameId?: string;
   cdpSessionId?: string;
+  visibleRect?: Rect;
+  kind: RefTargetKind;
+  capabilities: RefCapability[];
 }
 
 function refEntryForTab(ctx: SessionContext, refKey: string, tabId: number) {
@@ -37,6 +41,9 @@ export function lookupSnapshotRef(
     refKey,
     ...(entry.frameId ? { frameId: entry.frameId } : {}),
     ...(entry.cdpSessionId ? { cdpSessionId: entry.cdpSessionId } : {}),
+    ...(entry.visibleRect ? { visibleRect: entry.visibleRect } : {}),
+    kind: entry.kind,
+    capabilities: entry.capabilities,
   };
 }
 
@@ -49,6 +56,7 @@ export function resolveSnapshotRef(
   ctx: SessionContext,
   ref: string,
   tabId: number,
+  requiredCapability?: RefCapability,
 ): SnapshotRefLookup | RpcError {
   const looked = lookupSnapshotRef(ctx, ref, tabId);
   if (looked === null) {
@@ -58,5 +66,13 @@ export function resolveSnapshotRef(
       `ref ${ref} unknown for tab ${tabId} in session ${ctx.sessionId}`,
     );
   }
+  if (requiredCapability && !looked.capabilities.includes(requiredCapability)) {
+    return rpcError(
+      "permission_denied",
+      "ref_capability_denied",
+      `ref ${ref} does not support ${requiredCapability}`,
+      { ref: looked.refKey, required_capability: requiredCapability, kind: looked.kind },
+    );
+  }
   return looked;
 }

apps/extension/src/tools/transfer-transaction.ts

@@ -1,42 +0,0 @@
-// Shared time boundary for browser-side file-transfer transactions. Concrete
-// upload/download mechanisms own their state machines and effect semantics.
-
-export class BoundedWaitError extends Error {
-  constructor(
-    readonly kind: "timeout" | "aborted",
-    message: string,
-  ) {
-    super(message);
-  }
-}
-
-export function remainingMs(deadline: number): number {
-  return Math.max(0, deadline - Date.now());
-}
-
-export async function waitBounded<T>(
-  promise: Promise<T>,
-  deadline: number,
-  signal: AbortSignal | undefined,
-  timeoutMessage: string,
-): Promise<T> {
-  const remaining = remainingMs(deadline);
-  if (signal?.aborted) throw new BoundedWaitError("aborted", "file transfer aborted");
-  if (remaining === 0) throw new BoundedWaitError("timeout", timeoutMessage);
-
-  let timer: ReturnType<typeof setTimeout> | undefined;
-  let onAbort: (() => void) | undefined;
-  const boundary = new Promise<never>((_resolve, reject) => {
-    timer = setTimeout(() => reject(new BoundedWaitError("timeout", timeoutMessage)), remaining);
-    if (signal) {
-      onAbort = () => reject(new BoundedWaitError("aborted", "file transfer aborted"));
-      signal.addEventListener("abort", onAbort, { once: true });
-    }
-  });
-  try {
-    return await Promise.race([promise, boundary]);
-  } finally {
-    if (timer) clearTimeout(timer);
-    if (signal && onAbort) signal.removeEventListener("abort", onAbort);
-  }
-}

apps/extension/src/tools/upload.ts

@@ -2,13 +2,8 @@
 // target, then delegate the browser protocol transaction to the file-input
 // transaction module.
 
-import {
-  type CaptureSuppressSendToTab,
-  withExtensionOverlayHidden,
-} from "@/lib/capture-suppress-bridge";
 import type { SessionManager } from "@/session-manager/manager";
 import type { RpcError, UploadParams, UploadResult } from "@/transport/types";
-import { uploadThroughFileDrop } from "./file-drop-transaction";
 import { uploadThroughActivatedFileInput } from "./file-input-transaction";
 import { clickResolvedTarget, type InteractionDeps, resolveActionTarget } from "./interaction";
 import {
@@ -23,7 +18,6 @@ const DEFAULT_TIMEOUT_MS = 120_000;
 
 export interface UploadDeps extends InteractionDeps {
   cdp: CdpRunner;
-  sendToTab?: CaptureSuppressSendToTab;
 }
 
 export async function handleUpload(
@@ -47,39 +41,19 @@ export async function handleUpload(
   const address = await resolveActionTarget(deps.cdp, ctx, target, params, "upload");
   if (isRpcError(address)) return address;
   const timeoutMs = params.timeout_ms ?? DEFAULT_TIMEOUT_MS;
-  const files = params.files.map((file) => file.staged_path as string);
-  const mode = params.mode ?? "input";
-  let usedRef = address.usedRef;
-  let usedSelector = address.usedSelector;
-  if (mode === "drop") {
-    deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
-    const transaction = await uploadThroughFileDrop({
-      cdp: deps.cdp,
-      actionTarget: address,
-      files,
-      timeoutMs,
-      signal: deps.signal,
-      withOverlayHidden: (operation) =>
-        withExtensionOverlayHidden(target.tabId, operation, deps.sendToTab),
-    });
-    if (isRpcError(transaction)) return transaction;
-  } else {
-    const transaction = await uploadThroughActivatedFileInput({
-      cdp: deps.cdp,
-      actionTarget: address,
-      files,
-      timeoutMs,
-      signal: deps.signal,
-      trigger: () => clickResolvedTarget(ctx, address, {}, deps),
-    });
-    if (isRpcError(transaction)) return transaction;
-    usedRef = transaction.click.used_ref;
-    usedSelector = transaction.click.used_selector;
-  }
+  const transaction = await uploadThroughActivatedFileInput({
+    cdp: deps.cdp,
+    actionTarget: address,
+    files: params.files.map((file) => file.staged_path as string),
+    timeoutMs,
+    signal: deps.signal,
+    trigger: () => clickResolvedTarget(ctx, address, {}, deps),
+  });
+  if (isRpcError(transaction)) return transaction;
   return {
     tab_id: target.tabId,
-    used_ref: usedRef,
-    used_selector: usedSelector,
+    used_ref: transaction.click.used_ref,
+    used_selector: transaction.click.used_selector,
     file_names: params.files.map((file) => file.name),
   };
 }

apps/extension/src/tools/vom/__tests__/capture.test.ts

@@ -1,6 +1,7 @@
 import { describe, expect, it, vi } from "vitest";
 import { OVERLAY_HOST_MARKER_ATTR, OVERLAY_HOST_NAME } from "../../../lib/overlay-bridge";
 import { captureViewModel, collectOverlayExcludedBackendIds, probeHoverSurfaces } from "../capture";
+import { discoverRenderedSurfaces } from "../rendered-surfaces";
 
 // Minimal but format-accurate captureSnapshot reply: a body with one
 // fixed full-screen overlay div carrying a password input.
@@ -724,6 +725,44 @@ describe("captureViewModel", () => {
     expect(nodes.find((node) => node.backendNodeId === 14)?.rendered).toBe(false);
   });
 
+  it("propagates fully transparent ancestor suppression to painted descendants", async () => {
+    const S = ["html", "body", "div", "canvas", "static", "auto", "visible", "1", "0"];
+    const i = (value: string) => S.indexOf(value);
+    const snapshot = {
+      strings: S,
+      documents: [
+        {
+          nodes: {
+            parentIndex: [-1, 0, 1, 2],
+            nodeName: [i("html"), i("body"), i("div"), i("canvas")],
+            backendNodeId: [10, 11, 12, 13],
+            attributes: [[], [], [], []],
+          },
+          layout: {
+            nodeIndex: [1, 2, 3],
+            styles: [
+              [i("static"), i("auto"), i("auto"), i("visible"), i("1")],
+              [i("static"), i("auto"), i("auto"), i("visible"), i("0")],
+              [i("static"), i("auto"), i("auto"), i("visible"), i("1")],
+            ],
+            bounds: [
+              [0, 0, 1000, 800],
+              [10, 10, 200, 100],
+              [10, 10, 200, 100],
+            ],
+            paintOrders: [0, 1, 2],
+          },
+        },
+      ],
+    };
+
+    const { nodes } = await captureViewModel(makeCdp(snapshot), 4);
+    const canvas = nodes.find((node) => node.backendNodeId === 13);
+
+    expect(canvas?.rendered).toBe(true);
+    expect(canvas?.visuallySuppressed).toBe(true);
+  });
+
   it("subtracts scroll offset to make rects viewport-relative", async () => {
     const cdp = {
       send: vi.fn(async (_t: number, method: string) => {
@@ -1139,6 +1178,97 @@ describe("captureViewModel", () => {
     expect(input?.rect).toEqual({ x: 16, y: 28, w: 40, h: 20 });
   });
 
+  it("keeps same-origin iframe canvas discovery in top-level CSS coordinates at dpr 1 and 2", async () => {
+    const captureAtDpr = async (dpr: number) => {
+      const S = ["html", "body", "iframe", "canvas", "static", "auto"];
+      const i = (s: string) => S.indexOf(s);
+      const scale = (values: number[]) => values.map((value) => value * dpr);
+      const snapshot = {
+        strings: S,
+        documents: [
+          {
+            frameId: "main",
+            nodes: {
+              parentIndex: [-1, 0, 1],
+              nodeName: [i("html"), i("body"), i("iframe")],
+              backendNodeId: [10, 11, 13],
+              attributes: [[], [], []],
+              contentDocumentIndex: { index: [2], value: [1] },
+            },
+            layout: {
+              nodeIndex: [1, 2],
+              styles: [
+                [i("static"), i("auto")],
+                [i("static"), i("auto")],
+              ],
+              bounds: [scale([0, 0, 1000, 800]), scale([100, 150, 300, 200])],
+              paintOrders: [0, 1],
+            },
+          },
+          {
+            frameId: "child",
+            nodes: {
+              parentIndex: [-1, 0],
+              nodeName: [i("body"), i("canvas")],
+              backendNodeId: [20, 21],
+              attributes: [[], []],
+            },
+            layout: {
+              nodeIndex: [0, 1],
+              styles: [
+                [i("static"), i("auto")],
+                [i("static"), i("auto")],
+              ],
+              bounds: [scale([0, 0, 300, 200]), scale([20, 30, 120, 60])],
+              paintOrders: [0, 1],
+            },
+          },
+        ],
+      };
+      const cdp = {
+        send: vi.fn(async (_tabId: number, method: string) => {
+          if (method === "DOMSnapshot.enable") return {};
+          if (method === "DOMSnapshot.captureSnapshot") return snapshot;
+          if (method === "Page.getLayoutMetrics") {
+            return {
+              cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
+              layoutViewport: { clientWidth: 1000 * dpr, clientHeight: 800 * dpr },
+            };
+          }
+          throw new Error(method);
+        }) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
+      };
+      const captured = await captureViewModel(cdp, 4);
+      const childNodes = captured.frameNodes?.get("child") ?? [];
+      return discoverRenderedSurfaces(
+        [
+          {
+            frameId: "child",
+            contextScopeId: "child",
+            parentFrameId: "main",
+            ownerBackendNodeId: 13,
+            target: { tabId: 4 },
+            domNodes: childNodes,
+            axNodes: [],
+          },
+        ],
+        captured.viewport,
+      );
+    };
+
+    const atDpr1 = await captureAtDpr(1);
+    const atDpr2 = await captureAtDpr(2);
+
+    expect(atDpr1).toEqual([
+      expect.objectContaining({
+        backendNodeId: 21,
+        frameId: "child",
+        visibleRect: { x: 120, y: 180, w: 120, h: 60 },
+      }),
+    ]);
+    expect(atDpr2).toEqual(atDpr1);
+  });
+
   it("normalizes bounds then subtracts CSS scroll at dpr>1", async () => {
     const S = ["html", "body", "div", "position", "fixed", "static", "pointer-events", "auto"];
     const i = (s: string) => S.indexOf(s);

apps/extension/src/tools/vom/__tests__/rendered-surfaces.test.ts

@@ -0,0 +1,256 @@
+import type { VomScene } from "@browser-skill/vom";
+import { describe, expect, it } from "vitest";
+import type { CapturedNode } from "../capture";
+import type { CapturedFrameDocument } from "../frame-capture";
+import {
+  clusterRenderedSurfaces,
+  discoverRenderedSurfaces,
+  projectRenderedSurfaces,
+} from "../rendered-surfaces";
+import type { SemanticAxNode } from "../semantic-graph";
+
+function domNode(backendNodeId: number, overrides: Partial<CapturedNode> = {}): CapturedNode {
+  return {
+    backendNodeId,
+    parentBackendNodeId: null,
+    frameId: "main",
+    ownerFrameBackendNodeId: null,
+    tag: "canvas",
+    attrs: {},
+    rect: { x: 10, y: 20, w: 800, h: 500 },
+    localRect: { x: 10, y: 20, w: 800, h: 500 },
+    paintOrder: 1,
+    position: "static",
+    pointerEvents: "auto",
+    rendered: true,
+    ...overrides,
+  };
+}
+
+function document(
+  domNodes: CapturedNode[],
+  axNodes: SemanticAxNode[] = [],
+): CapturedFrameDocument<SemanticAxNode> {
+  return {
+    frameId: domNodes[0]?.frameId ?? "main",
+    contextScopeId: domNodes[0]?.frameId ?? "main",
+    target: { tabId: 4 },
+    domNodes,
+    axNodes,
+  };
+}
+
+describe("rendered surface discovery", () => {
+  it("finds prominent rendered canvases and excludes hidden descendants", () => {
+    const visible = domNode(2, { parentBackendNodeId: 1 });
+    const hidden = domNode(3, { parentBackendNodeId: 4 });
+    const surfaces = discoverRenderedSurfaces(
+      [
+        document([
+          domNode(1, { tag: "main", rect: null, localRect: null }),
+          visible,
+          domNode(4, { tag: "section", attrs: { "aria-hidden": "true" }, rect: null }),
+          hidden,
+        ]),
+      ],
+      { width: 1280, height: 720 },
+    );
+
+    expect(surfaces).toHaveLength(1);
+    expect(surfaces[0]).toMatchObject({ backendNodeId: 2 });
+  });
+
+  it("keeps tiny canvases as individually addressable exact discoveries", () => {
+    const surfaces = discoverRenderedSurfaces(
+      [document([domNode(2, { rect: { x: 1, y: 1, w: 8, h: 8 } })])],
+      { width: 1280, height: 720 },
+    );
+    expect(surfaces).toEqual([expect.objectContaining({ backendNodeId: 2 })]);
+  });
+
+  it("keeps a surface when the canvas only has partial native accessibility content", () => {
+    const axNodes = [
+      {
+        nodeId: "canvas",
+        backendDOMNodeId: 2,
+        role: { type: "role", value: "presentation" },
+        childIds: ["button"],
+        frameId: "main",
+      },
+      {
+        nodeId: "button",
+        backendDOMNodeId: 3,
+        role: { type: "role", value: "button" },
+        frameId: "main",
+      },
+    ] as SemanticAxNode[];
+    expect(
+      discoverRenderedSurfaces([document([domNode(2)], axNodes)], { width: 1280, height: 720 }),
+    ).toEqual([expect.objectContaining({ backendNodeId: 2 })]);
+  });
+
+  it("discovers unnamed canvases without guessing labels from engineering attributes", () => {
+    const canvas = domNode(2, {
+      attrs: {
+        id: "must-not-be-a-label",
+        class: "must-not-be-a-label",
+        "data-testid": "must-not-be-a-label",
+      },
+    });
+
+    expect(discoverRenderedSurfaces([document([canvas])], { width: 1280, height: 720 })).toEqual([
+      expect.not.objectContaining({
+        label: expect.anything(),
+      }),
+    ]);
+  });
+
+  it("treats blank explicit Canvas names as unnamed", () => {
+    expect(
+      discoverRenderedSurfaces(
+        [document([domNode(2, { attrs: { "aria-label": "   ", title: "\n\t" } })])],
+        { width: 1280, height: 720 },
+      ),
+    ).toEqual([expect.not.objectContaining({ label: expect.anything() })]);
+  });
+
+  it("excludes canvases inside BrowserSkill-owned DOM subtrees", () => {
+    const owner = domNode(1, { tag: "div", rect: null, localRect: null });
+    const canvas = domNode(2, { parentBackendNodeId: 1 });
+
+    expect(
+      discoverRenderedSurfaces(
+        [document([owner, canvas])],
+        { width: 1280, height: 720 },
+        new Set([1]),
+      ),
+    ).toEqual([]);
+  });
+
+  it("excludes canvases suppressed by a fully transparent ancestor", () => {
+    const owner = domNode(1, {
+      tag: "div",
+      rect: { x: 0, y: 0, w: 900, h: 600 },
+      visuallySuppressed: true,
+    });
+    const canvas = domNode(2, { parentBackendNodeId: 1 });
+
+    expect(
+      discoverRenderedSurfaces([document([owner, canvas])], { width: 1280, height: 720 }),
+    ).toEqual([]);
+  });
+
+  it("clips canvases to overflow ancestors and drops fully clipped canvases", () => {
+    const owner = domNode(1, {
+      tag: "div",
+      rect: { x: 10, y: 20, w: 100, h: 70 },
+      overflowX: "hidden",
+      overflowY: "hidden",
+    });
+    const partial = domNode(2, {
+      parentBackendNodeId: 1,
+      rect: { x: 55, y: 30, w: 140, h: 60 },
+    });
+    const fullyClipped = domNode(3, {
+      parentBackendNodeId: 1,
+      rect: { x: 140, y: 30, w: 80, h: 50 },
+    });
+
+    expect(
+      discoverRenderedSurfaces([document([owner, partial, fullyClipped])], {
+        width: 1280,
+        height: 720,
+      }),
+    ).toEqual([
+      expect.objectContaining({
+        backendNodeId: 2,
+        visibleRect: { x: 55, y: 30, w: 55, h: 60 },
+      }),
+    ]);
+  });
+});
+
+describe("rendered surface clustering and projection", () => {
+  it("collapses overlapping canvas layers and attaches them to the nearest retained parent", () => {
+    const body = domNode(1, { tag: "body", rect: null, localRect: null });
+    const surfaces = discoverRenderedSurfaces(
+      [
+        document([
+          body,
+          domNode(2, { parentBackendNodeId: 1, paintOrder: 1 }),
+          domNode(3, { parentBackendNodeId: 1, paintOrder: 2 }),
+        ]),
+      ],
+      { width: 1280, height: 720 },
+    );
+    const groups = clusterRenderedSurfaces(surfaces);
+    const scene: VomScene = {
+      viewport: { width: 1280, height: 720 },
+      nodes: [
+        {
+          id: 10,
+          parentId: null,
+          backendNodeId: 1,
+          frameId: "main",
+          tag: "body",
+          role: "RootWebArea",
+          rect: null,
+          paintOrder: 0,
+          position: "static",
+          pointerEvents: "auto",
+        },
+      ],
+    };
+
+    expect(groups).toHaveLength(1);
+    expect(groups[0]?.members).toHaveLength(2);
+    expect(groups[0]?.label).toBeUndefined();
+    expect(
+      projectRenderedSurfaces(
+        scene,
+        [document([body, ...surfaces.map((s) => domNode(s.backendNodeId))])],
+        groups,
+      ),
+    ).toMatchObject({
+      visualSurfaces: [expect.objectContaining({ parentId: 10, backendNodeId: 3, memberCount: 2 })],
+    });
+  });
+
+  it("does not merge a contained canvas with a larger independent canvas", () => {
+    const groups = clusterRenderedSurfaces([
+      ...discoverRenderedSurfaces(
+        [
+          document([
+            domNode(1, { tag: "main", rect: null, localRect: null }),
+            domNode(2, { parentBackendNodeId: 1 }),
+            domNode(3, {
+              parentBackendNodeId: 1,
+              rect: { x: 30, y: 40, w: 200, h: 100 },
+            }),
+          ]),
+        ],
+        { width: 1280, height: 720 },
+      ),
+    ]);
+
+    expect(groups).toHaveLength(2);
+  });
+
+  it("does not merge coincident canvases owned by different containers", () => {
+    const groups = clusterRenderedSurfaces(
+      discoverRenderedSurfaces(
+        [
+          document([
+            domNode(1, { tag: "section", rect: null, localRect: null }),
+            domNode(2, { tag: "section", rect: null, localRect: null }),
+            domNode(3, { parentBackendNodeId: 1 }),
+            domNode(4, { parentBackendNodeId: 2 }),
+          ]),
+        ],
+        { width: 1280, height: 720 },
+      ),
+    );
+
+    expect(groups).toHaveLength(2);
+  });
+});

apps/extension/src/tools/vom/capture.ts

@@ -11,7 +11,15 @@ import { childFrameProjection, type GeometryProjection, projectRectToViewport }
 import type { CdpRunner } from "../shared";
 import { clearHover, ProbeBudget, waitForHover } from "./hover-perception";
 
-const REQUESTED_STYLES = ["position", "pointer-events", "cursor", "visibility", "opacity"] as const;
+const REQUESTED_STYLES = [
+  "position",
+  "pointer-events",
+  "cursor",
+  "visibility",
+  "opacity",
+  "overflow-x",
+  "overflow-y",
+] as const;
 const STYLE_COL = Object.fromEntries(
   REQUESTED_STYLES.map((name, index) => [name, index]),
 ) as Record<(typeof REQUESTED_STYLES)[number], number>;
@@ -39,6 +47,10 @@ export interface CapturedNode {
    * `textContent`: the live parser always sets it, hand-built fixtures may not.
    */
   cursor?: string;
+  overflowX?: string;
+  overflowY?: string;
+  /** True when this node or one of its DOM ancestors makes the subtree fully transparent/hidden. */
+  visuallySuppressed?: boolean;
   /**
    * Whether the live DOM snapshot provides a painted, non-hidden box for this
    * node. Semantic resolution uses this only for DOM fallback nodes; AX-backed
@@ -818,6 +830,8 @@ function parseDocumentNodes(
   const cursorCol = STYLE_COL.cursor;
   const visibilityCol = STYLE_COL.visibility;
   const opacityCol = STYLE_COL.opacity;
+  const overflowXCol = STYLE_COL["overflow-x"];
+  const overflowYCol = STYLE_COL["overflow-y"];
   const inputValues = sparseIndexMap(dn.inputValue);
   const textValues = sparseIndexMap(dn.textValue);
   const checkedInputs = new Set(dn.inputChecked?.index ?? []);
@@ -843,6 +857,7 @@ function parseDocumentNodes(
 
   // Build CapturedNodes.
   const nodes: CapturedNode[] = [];
+  const visuallySuppressedByNodeIndex = new Array<boolean>(count).fill(false);
   for (let n = 0; n < count; n++) {
     if (excluded[n]) continue;
     const backendNodeId = dn.backendNodeId[n];
@@ -862,7 +877,11 @@ function parseDocumentNodes(
     let position = "static";
     let pointerEvents = "auto";
     let cursor = "auto";
+    let overflowX = "visible";
+    let overflowY = "visible";
     let rendered = false;
+    let visuallySuppressed =
+      parentIdx >= 0 ? visuallySuppressedByNodeIndex[parentIdx] === true : false;
     const li = layoutByNode.get(n);
     if (li !== undefined) {
       const b = dl?.bounds?.[li];
@@ -885,12 +904,19 @@ function parseDocumentNodes(
       cursor = str(strings, styleRow[cursorCol]) || "auto";
       const visibility = str(strings, styleRow[visibilityCol]) || "visible";
       const opacity = str(strings, styleRow[opacityCol]) || "1";
+      overflowX = str(strings, styleRow[overflowXCol]) || "visible";
+      overflowY = str(strings, styleRow[overflowYCol]) || "visible";
+      visuallySuppressed ||=
+        visibility === "hidden" ||
+        visibility === "collapse" ||
+        (Number.parseFloat(opacity) || 0) <= 0;
       rendered =
         localRect !== null &&
         visibility !== "hidden" &&
         visibility !== "collapse" &&
         (Number.parseFloat(opacity) || 0) > 0;
     }
+    visuallySuppressedByNodeIndex[n] = visuallySuppressed;
 
     // Skip non-element nodes (#text, #cdata-section, etc.) — they carry no
     // geometry and are never queried by callers.
@@ -925,6 +951,9 @@ function parseDocumentNodes(
       position,
       pointerEvents,
       cursor,
+      overflowX,
+      overflowY,
+      visuallySuppressed,
       rendered,
       textContent,
       ...(formValue !== undefined ? { formValue } : {}),

apps/extension/src/tools/vom/rendered-surfaces/cluster.ts

@@ -0,0 +1,65 @@
+import type { Rect } from "@browser-skill/vom";
+import type { RenderedSurface, RenderedSurfaceGroup } from "./types";
+
+const STACK_IOU_RATIO = 0.9;
+const STACK_SIZE_RATIO = 0.9;
+
+function area(rect: Rect): number {
+  return Math.max(0, rect.w) * Math.max(0, rect.h);
+}
+
+function intersectionArea(a: Rect, b: Rect): number {
+  const width = Math.max(0, Math.min(a.x + a.w, b.x + b.w) - Math.max(a.x, b.x));
+  const height = Math.max(0, Math.min(a.y + a.h, b.y + b.h) - Math.max(a.y, b.y));
+  return width * height;
+}
+
+function sameVisualStack(a: RenderedSurface, b: RenderedSurface): boolean {
+  if (a.frameId !== b.frameId) return false;
+  if (a.parentBackendNodeId !== b.parentBackendNodeId) return false;
+  const aArea = area(a.visibleRect);
+  const bArea = area(b.visibleRect);
+  const intersection = intersectionArea(a.visibleRect, b.visibleRect);
+  const union = aArea + bArea - intersection;
+  const widthRatio =
+    Math.min(a.visibleRect.w, b.visibleRect.w) / Math.max(a.visibleRect.w, b.visibleRect.w);
+  const heightRatio =
+    Math.min(a.visibleRect.h, b.visibleRect.h) / Math.max(a.visibleRect.h, b.visibleRect.h);
+  return (
+    union > 0 &&
+    intersection / union >= STACK_IOU_RATIO &&
+    widthRatio >= STACK_SIZE_RATIO &&
+    heightRatio >= STACK_SIZE_RATIO
+  );
+}
+
+function toGroup(members: RenderedSurface[]): RenderedSurfaceGroup {
+  const representative = [...members].sort(
+    (a, b) => area(b.visibleRect) - area(a.visibleRect) || b.paintOrder - a.paintOrder,
+  )[0];
+  return {
+    frameId: representative.frameId,
+    members,
+    representative,
+    label: members.map((surface) => surface.label).find((label) => label !== undefined),
+  };
+}
+
+export function clusterRenderedSurfaces(surfaces: RenderedSurface[]): RenderedSurfaceGroup[] {
+  const groups: RenderedSurface[][] = [];
+  for (const surface of surfaces) {
+    const matching = groups.find((group) =>
+      group.some((member) => sameVisualStack(member, surface)),
+    );
+    if (matching) matching.push(surface);
+    else groups.push([surface]);
+  }
+  return groups
+    .map(toGroup)
+    .sort(
+      (a, b) =>
+        a.frameId.localeCompare(b.frameId) ||
+        a.representative.visibleRect.y - b.representative.visibleRect.y ||
+        a.representative.visibleRect.x - b.representative.visibleRect.x,
+    );
+}

apps/extension/src/tools/vom/rendered-surfaces/discovery.ts

@@ -0,0 +1,113 @@
+import type { Rect, Viewport } from "@browser-skill/vom";
+import type { CapturedNode } from "../capture";
+import type { CapturedFrameDocument } from "../frame-capture";
+import type { SemanticAxNode } from "../semantic-graph";
+import type { RenderedSurface } from "./types";
+
+function clean(value: string | undefined): string | undefined {
+  const normalized = value?.replace(/\s+/g, " ").trim();
+  return normalized || undefined;
+}
+
+function intersectViewport(rect: Rect, viewport: Viewport): Rect | null {
+  return intersectRect(rect, { x: 0, y: 0, w: viewport.width, h: viewport.height });
+}
+
+function intersectRect(a: Rect, b: Rect): Rect | null {
+  const x = Math.max(a.x, b.x);
+  const y = Math.max(a.y, b.y);
+  const right = Math.min(a.x + a.w, b.x + b.w);
+  const bottom = Math.min(a.y + a.h, b.y + b.h);
+  if (right <= x || bottom <= y) return null;
+  return { x, y, w: right - x, h: bottom - y };
+}
+
+const CLIPPING_OVERFLOW = new Set(["auto", "clip", "hidden", "scroll"]);
+
+function clipToOverflowAncestors(
+  node: CapturedNode,
+  rect: Rect,
+  nodesByBackend: ReadonlyMap<number, CapturedNode>,
+): Rect | null {
+  let clipped: Rect | null = rect;
+  let parentBackendNodeId = node.parentBackendNodeId;
+  let guard = 0;
+  while (clipped && parentBackendNodeId !== null && guard <= nodesByBackend.size) {
+    const parent = nodesByBackend.get(parentBackendNodeId);
+    if (!parent) break;
+    if (parent.rect) {
+      const clipX = CLIPPING_OVERFLOW.has(parent.overflowX ?? "visible");
+      const clipY = CLIPPING_OVERFLOW.has(parent.overflowY ?? "visible");
+      if (clipX || clipY) {
+        const clipRect = {
+          x: clipX ? parent.rect.x : clipped.x,
+          y: clipY ? parent.rect.y : clipped.y,
+          w: clipX ? parent.rect.w : clipped.w,
+          h: clipY ? parent.rect.h : clipped.h,
+        };
+        clipped = intersectRect(clipped, clipRect);
+      }
+    }
+    parentBackendNodeId = parent.parentBackendNodeId;
+    guard += 1;
+  }
+  return clipped;
+}
+
+function excludedByDomPolicy(
+  node: CapturedNode,
+  nodesByBackend: Map<number, CapturedNode>,
+  excludedBackendNodeIds: ReadonlySet<number>,
+): boolean {
+  let current: CapturedNode | undefined = node;
+  let guard = 0;
+  while (current && guard <= nodesByBackend.size) {
+    if (excludedBackendNodeIds.has(current.backendNodeId)) return true;
+    if (current.visuallySuppressed === true) return true;
+    const attrs = current.attrs;
+    if (
+      Object.prototype.hasOwnProperty.call(attrs, "hidden") ||
+      Object.prototype.hasOwnProperty.call(attrs, "inert") ||
+      (attrs["aria-hidden"] ?? "").toLowerCase() === "true"
+    ) {
+      return true;
+    }
+    current =
+      current.parentBackendNodeId === null
+        ? undefined
+        : nodesByBackend.get(current.parentBackendNodeId);
+    guard += 1;
+  }
+  return false;
+}
+
+export function discoverRenderedSurfaces(
+  documents: CapturedFrameDocument<SemanticAxNode>[],
+  viewport: Viewport,
+  excludedBackendNodeIds: ReadonlySet<number> = new Set(),
+): RenderedSurface[] {
+  const surfaces: RenderedSurface[] = [];
+  for (const document of documents) {
+    const nodesByBackend = new Map(document.domNodes.map((node) => [node.backendNodeId, node]));
+    for (const node of document.domNodes) {
+      if (node.tag.toLowerCase() !== "canvas" || node.rendered !== true || !node.rect) continue;
+      if (excludedByDomPolicy(node, nodesByBackend, excludedBackendNodeIds)) continue;
+      const viewportRect = intersectViewport(node.rect, viewport);
+      const visibleRect = viewportRect
+        ? clipToOverflowAncestors(node, viewportRect, nodesByBackend)
+        : null;
+      if (!visibleRect) continue;
+      const label = clean(node.attrs["aria-label"] ?? node.attrs.title);
+      surfaces.push({
+        renderingKind: "canvas",
+        frameId: document.frameId,
+        backendNodeId: node.backendNodeId,
+        parentBackendNodeId: node.parentBackendNodeId,
+        visibleRect,
+        paintOrder: node.paintOrder,
+        ...(label ? { label } : {}),
+      });
+    }
+  }
+  return surfaces;
+}

apps/extension/src/tools/vom/rendered-surfaces/index.ts

@@ -0,0 +1,4 @@
+export { clusterRenderedSurfaces } from "./cluster";
+export { discoverRenderedSurfaces } from "./discovery";
+export { projectRenderedSurfaces } from "./project";
+export type { RenderedSurface, RenderedSurfaceGroup } from "./types";

apps/extension/src/tools/vom/rendered-surfaces/project.ts

@@ -0,0 +1,100 @@
+import type { VomScene, VomVisualSurface } from "@browser-skill/vom";
+import type { CapturedFrameDocument } from "../frame-capture";
+import type { SemanticAxNode } from "../semantic-graph";
+import type { RenderedSurfaceGroup } from "./types";
+
+function frameBackendKey(frameId: string, backendNodeId: number): string {
+  return `${frameId}\u0000${backendNodeId}`;
+}
+
+interface ProjectionIndex {
+  sceneByBackend: ReadonlyMap<string, number>;
+  documentByFrame: ReadonlyMap<string, CapturedFrameDocument<SemanticAxNode>>;
+  domByFrame: ReadonlyMap<
+    string,
+    ReadonlyMap<number, CapturedFrameDocument<SemanticAxNode>["domNodes"][number]>
+  >;
+  rootSceneIdByFrame: ReadonlyMap<string, number>;
+}
+
+function nearestSceneParent(index: ProjectionIndex, group: RenderedSurfaceGroup): number | null {
+  const document = index.documentByFrame.get(group.frameId);
+  const domByBackend = index.domByFrame.get(group.frameId) ?? new Map();
+  let parentBackendNodeId = group.representative.parentBackendNodeId;
+  let guard = 0;
+  while (parentBackendNodeId !== null && guard <= domByBackend.size) {
+    const sceneId = index.sceneByBackend.get(frameBackendKey(group.frameId, parentBackendNodeId));
+    if (sceneId !== undefined) return sceneId;
+    parentBackendNodeId = domByBackend.get(parentBackendNodeId)?.parentBackendNodeId ?? null;
+    guard += 1;
+  }
+
+  if (document?.parentFrameId && document.ownerBackendNodeId !== undefined) {
+    const owner = index.sceneByBackend.get(
+      frameBackendKey(document.parentFrameId, document.ownerBackendNodeId),
+    );
+    if (owner !== undefined) return owner;
+  }
+
+  return index.rootSceneIdByFrame.get(group.frameId) ?? null;
+}
+
+function buildProjectionIndex(
+  scene: VomScene,
+  documents: CapturedFrameDocument<SemanticAxNode>[],
+): ProjectionIndex {
+  const rootSceneIdByFrame = new Map<string, number>();
+  for (const node of scene.nodes) {
+    if (
+      node.frameId &&
+      !rootSceneIdByFrame.has(node.frameId) &&
+      ["rootwebarea", "webarea"].includes(node.role?.toLowerCase() ?? "")
+    ) {
+      rootSceneIdByFrame.set(node.frameId, node.id);
+    }
+  }
+  return {
+    sceneByBackend: new Map(
+      scene.nodes.flatMap((node) =>
+        node.backendNodeId === undefined
+          ? []
+          : [[frameBackendKey(node.frameId ?? "", node.backendNodeId), node.id] as const],
+      ),
+    ),
+    documentByFrame: new Map(documents.map((document) => [document.frameId, document])),
+    domByFrame: new Map(
+      documents.map((document) => [
+        document.frameId,
+        new Map(document.domNodes.map((node) => [node.backendNodeId, node])),
+      ]),
+    ),
+    rootSceneIdByFrame,
+  };
+}
+
+export function projectRenderedSurfaces(
+  scene: VomScene,
+  documents: CapturedFrameDocument<SemanticAxNode>[],
+  groups: RenderedSurfaceGroup[],
+): VomScene {
+  const index = buildProjectionIndex(scene, documents);
+  const visualSurfaces: VomVisualSurface[] = [];
+
+  for (const group of groups) {
+    const parentId = nearestSceneParent(index, group);
+    visualSurfaces.push({
+      parentId,
+      backendNodeId: group.representative.backendNodeId,
+      frameId: group.frameId,
+      renderingKind: "canvas",
+      visibleRect: group.representative.visibleRect,
+      ...(group.label ? { label: group.label } : {}),
+      memberCount: group.members.length,
+    });
+  }
+
+  return {
+    ...scene,
+    ...(visualSurfaces.length > 0 ? { visualSurfaces } : {}),
+  };
+}

apps/extension/src/tools/vom/rendered-surfaces/types.ts

@@ -0,0 +1,18 @@
+import type { Rect } from "@browser-skill/vom";
+
+export interface RenderedSurface {
+  renderingKind: "canvas";
+  frameId: string;
+  backendNodeId: number;
+  parentBackendNodeId: number | null;
+  visibleRect: Rect;
+  paintOrder: number;
+  label?: string;
+}
+
+export interface RenderedSurfaceGroup {
+  frameId: string;
+  members: RenderedSurface[];
+  representative: RenderedSurface;
+  label?: string;
+}

apps/extension/src/transport/types.ts

@@ -23,6 +23,7 @@ export type ErrorCode =
 export type RpcErrorReason =
   | "agent_window_scope"
   | "element_not_visible"
+  | "ref_capability_denied"
   | "ref_not_found"
   | "selector_not_found"
   | "target_not_fillable"
@@ -36,9 +37,6 @@ export type RpcErrorReason =
   | "file_input_probe_failed"
   | "file_input_not_activated"
   | "set_file_input_failed"
-  | "upload_mechanism_unsupported"
-  | "file_drop_target_unavailable"
-  | "file_drop_failed"
   | "download_capture_failed"
   | "transfer_outcome_unknown"
   | "transfer_timeout"
@@ -526,15 +524,12 @@ export interface UploadFile {
   staged_path?: string;
 }
 
-export type UploadMode = "input" | "drop";
-
 export interface UploadParams {
   session_id: string;
   ref?: string;
   selector?: string;
   tab_id?: number;
   files: UploadFile[];
-  mode?: UploadMode;
   timeout_ms?: number;
 }
 

apps/extension/test-fixtures/rendered-surfaces/README.md

@@ -0,0 +1,105 @@
+# Rendered Surface browser fixtures
+
+These pages exercise Canvas discovery through a real browser DOMSnapshot and AX tree. They are deliberately framework- and site-independent.
+
+## Serve
+
+From the repository root:
+
+```bash
+python3 -m http.server 4173 --bind 127.0.0.1 \
+  --directory apps/extension/test-fixtures/rendered-surfaces
+```
+
+Use an Agent Window large enough to keep the primary matrix visible:
+
+```bash
+bsk session start --width 1440 --height 1100
+bsk navigate http://127.0.0.1:4173/index.html --session <id>
+bsk observe --session <id>
+```
+
+Always stop the session after the run.
+
+## Pages
+
+### `index.html`
+
+Expected individual Surface labels:
+
+- `fixture-basic-large`
+- `fixture-tiny-8px`
+- `fixture-pointer-events-none`
+- `fixture-partial-ax`
+- `fixture-contained-outer`
+- `fixture-contained-inner`
+- `fixture-different-parent-a`
+- `fixture-different-parent-b`
+
+Expected one grouped Surface:
+
+- `fixture-layered-group`, with `layers=2`
+
+Expected absent:
+
+- `fixture-display-none`
+- `fixture-visibility-hidden`
+- `fixture-opacity-zero`
+- `fixture-parent-opacity-zero`
+- `fixture-hidden-attribute`
+- `fixture-inert-parent`
+- `fixture-aria-hidden-parent`
+- `fixture-zero-size`
+- `fixture-offscreen-right`
+- `fixture-fully-overflow-clipped`
+
+`fixture-partially-overflow-clipped` must remain present because part of it is visible. Its screenshot must contain only the browser's final composited view.
+
+### `unlabeled.html`
+
+This page contains eight Canvas elements but must render seven generic Surface markers because
+the two coincident layers form one group.
+
+- Every marker uses the stable fallback label `canvas visual surface`.
+- Exactly one marker has `layers=2`.
+- The two contained canvases remain separate markers.
+- `id`, `class`, `data-testid`, nearby prose, and fallback AX content never become a Surface label.
+- Blank `aria-label` and `title` values behave as no label.
+- Fallback AX content does not suppress the corresponding Surface.
+
+### `frames.html`
+
+Expected Surface labels:
+
+- `fixture-same-origin-frame`
+- `fixture-cross-origin-frame`
+- `fixture-nested-frame`
+
+The cross-origin frame uses `http://localhost:4173` while the parent uses `http://127.0.0.1:4173`. With Chromium site isolation this exercises the OOPIF path without an external service.
+
+### `viewport.html`
+
+Run with a `900x700` Agent Window.
+
+- `fixture-partial-viewport` is partly visible and must be represented.
+- `fixture-below-fold` is absent initially.
+- After scrolling it into view and observing again, `fixture-below-fold` must be represented with a fresh ref.
+
+### `dynamic.html`
+
+Use the normal DOM buttons and re-observe after each action:
+
+- Initial state: `fixture-dynamic` absent.
+- After **Insert canvas**: present.
+- After **Resize to zero**: absent.
+- After **Restore size**: present with a fresh ref.
+- After **Remove canvas**: absent.
+
+## Invariants
+
+- `snapshot` must not emit any of these Surface markers.
+- `observe` must emit only currently visible Canvas surfaces.
+- Existing DOM/AX controls remain readable alongside Surface markers.
+- Missing labels never prevent discovery; naming and discovery are independent.
+- A Surface ref allows `screenshot --ref` and rejects ordinary click/fill/hover/select.
+- No fixture URL, id, label, or layout constant may appear in production discovery code.

apps/extension/test-fixtures/rendered-surfaces/dynamic.html

@@ -0,0 +1,46 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <title>Rendered Surface dynamic matrix</title>
+    <style>
+      html, body { margin: 0; font: 14px system-ui, sans-serif; }
+      body { padding: 16px; }
+      .controls { display: flex; gap: 8px; margin-bottom: 16px; }
+      button { padding: 8px 12px; }
+      canvas { display: block; width: 260px; height: 120px; background: #e8daef; border: 2px solid #7d3c98; }
+    </style>
+  </head>
+  <body>
+    <h1>Dynamic Canvas lifecycle</h1>
+    <div class="controls">
+      <button id="insert" type="button">Insert canvas</button>
+      <button id="zero" type="button">Resize to zero</button>
+      <button id="restore" type="button">Restore size</button>
+      <button id="remove" type="button">Remove canvas</button>
+    </div>
+    <div id="mount"></div>
+    <script>
+      const mount = document.querySelector("#mount");
+      const getCanvas = () => mount.querySelector("canvas");
+      document.querySelector("#insert").addEventListener("click", () => {
+        if (getCanvas()) return;
+        const canvas = document.createElement("canvas");
+        canvas.width = 260;
+        canvas.height = 120;
+        canvas.setAttribute("aria-label", "fixture-dynamic");
+        mount.append(canvas);
+      });
+      document.querySelector("#zero").addEventListener("click", () => {
+        const canvas = getCanvas();
+        if (canvas) canvas.style.cssText = "width:0;height:0;border:0";
+      });
+      document.querySelector("#restore").addEventListener("click", () => {
+        const canvas = getCanvas();
+        if (canvas) canvas.removeAttribute("style");
+      });
+      document.querySelector("#remove").addEventListener("click", () => getCanvas()?.remove());
+    </script>
+  </body>
+</html>

apps/extension/test-fixtures/rendered-surfaces/frame.html

@@ -0,0 +1,28 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <style>
+      html, body { margin: 0; font: 14px system-ui, sans-serif; }
+      body { padding: 12px; background: #fdfefe; }
+      canvas { display: block; width: 220px; height: 90px; background: #d5f5e3; border: 2px solid #1e8449; }
+      iframe { width: 280px; height: 150px; margin-top: 10px; border: 2px solid #85929e; }
+    </style>
+  </head>
+  <body>
+    <canvas id="frame-canvas" width="220" height="90"></canvas>
+    <script>
+      const params = new URLSearchParams(location.search);
+      const canvas = document.querySelector("#frame-canvas");
+      if (params.get("canvas") === "0") canvas.remove();
+      else canvas.setAttribute("aria-label", params.get("label") || "fixture-frame-default");
+      if (params.get("nested") === "1") {
+        const iframe = document.createElement("iframe");
+        iframe.title = "nested fixture frame";
+        iframe.src = "frame.html?label=fixture-nested-frame";
+        document.body.append(iframe);
+      }
+    </script>
+  </body>
+</html>

apps/extension/test-fixtures/rendered-surfaces/frames.html

@@ -0,0 +1,22 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <title>Rendered Surface frame matrix</title>
+    <style>
+      html, body { margin: 0; font: 14px system-ui, sans-serif; }
+      body { padding: 16px; }
+      .frames { display: grid; grid-template-columns: repeat(3, 320px); gap: 16px; }
+      iframe { width: 300px; height: 300px; border: 2px solid #5d6d7e; }
+    </style>
+  </head>
+  <body>
+    <h1>Frame matrix</h1>
+    <div class="frames">
+      <iframe title="same-origin fixture" src="frame.html?label=fixture-same-origin-frame"></iframe>
+      <iframe title="cross-origin fixture" src="http://localhost:4173/frame.html?label=fixture-cross-origin-frame"></iframe>
+      <iframe title="nested fixture" src="frame.html?canvas=0&nested=1"></iframe>
+    </div>
+  </body>
+</html>

apps/extension/test-fixtures/rendered-surfaces/index.html

@@ -0,0 +1,118 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <title>Rendered Surface discovery matrix</title>
+    <style>
+      * { box-sizing: border-box; }
+      html, body { margin: 0; min-height: 100%; font: 14px/1.4 system-ui, sans-serif; }
+      body { padding: 16px; background: #f4f6f8; color: #17202a; }
+      h1 { margin: 0 0 12px; font-size: 20px; }
+      .matrix { display: grid; grid-template-columns: repeat(4, minmax(220px, 1fr)); gap: 12px; }
+      .case { min-height: 150px; padding: 10px; overflow: hidden; border: 1px solid #aeb6bf; border-radius: 6px; background: white; }
+      .case h2 { margin: 0 0 8px; font-size: 13px; }
+      canvas { display: block; background: #d6eaf8; border: 2px solid #2874a6; }
+      .stack { position: relative; width: 180px; height: 90px; }
+      .stack canvas { position: absolute; inset: 0; width: 180px; height: 90px; }
+      .stack canvas + canvas { background: rgb(244 208 63 / 45%); border-color: #b7950b; }
+      .contained { position: relative; width: 190px; height: 100px; }
+      .contained > canvas:first-child { width: 190px; height: 100px; }
+      .contained > canvas:last-child { position: absolute; left: 55px; top: 30px; width: 70px; height: 40px; background: #fadbd8; border-color: #c0392b; }
+      .same-place { position: relative; width: 180px; height: 90px; }
+      .same-place > div { position: absolute; inset: 0; }
+      .same-place canvas { width: 180px; height: 90px; }
+      .same-place > div:last-child canvas { border-color: #7d3c98; background: rgb(210 180 222 / 35%); }
+      .clip-box { width: 100px; height: 70px; overflow: hidden; border: 2px dashed #566573; }
+      .partially-clipped { width: 140px; height: 60px; transform: translateX(45px); }
+      .fully-clipped { width: 80px; height: 50px; transform: translateX(130px); }
+      .tiny { width: 8px; height: 8px; border-width: 1px; }
+    </style>
+  </head>
+  <body>
+    <h1>Rendered Surface discovery matrix</h1>
+    <main class="matrix">
+      <section class="case">
+        <h2>Visible named canvas</h2>
+        <canvas aria-label="fixture-basic-large" width="200" height="90"></canvas>
+      </section>
+
+      <section class="case">
+        <h2>Tiny canvas remains addressable</h2>
+        <canvas class="tiny" aria-label="fixture-tiny-8px" width="8" height="8"></canvas>
+      </section>
+
+      <section class="case">
+        <h2>Visible but pointer-events none</h2>
+        <canvas aria-label="fixture-pointer-events-none" width="180" height="80" style="pointer-events: none"></canvas>
+      </section>
+
+      <section class="case">
+        <h2>Partial AX/fallback content</h2>
+        <canvas aria-label="fixture-partial-ax" width="180" height="80">
+          <button type="button">Fallback button must not suppress the surface</button>
+        </canvas>
+      </section>
+
+      <section class="case">
+        <h2>Two real rendering layers</h2>
+        <div class="stack">
+          <canvas aria-label="fixture-layered-group" width="180" height="90"></canvas>
+          <canvas width="180" height="90"></canvas>
+        </div>
+      </section>
+
+      <section class="case">
+        <h2>Contained canvases are independent</h2>
+        <div class="contained">
+          <canvas aria-label="fixture-contained-outer" width="190" height="100"></canvas>
+          <canvas aria-label="fixture-contained-inner" width="70" height="40"></canvas>
+        </div>
+      </section>
+
+      <section class="case">
+        <h2>Same geometry, different parents</h2>
+        <div class="same-place">
+          <div><canvas aria-label="fixture-different-parent-a" width="180" height="90"></canvas></div>
+          <div><canvas aria-label="fixture-different-parent-b" width="180" height="90"></canvas></div>
+        </div>
+      </section>
+
+      <section class="case">
+        <h2>Partially overflow-clipped</h2>
+        <div class="clip-box">
+          <canvas class="partially-clipped" aria-label="fixture-partially-overflow-clipped" width="140" height="60"></canvas>
+        </div>
+      </section>
+
+      <section class="case">
+        <h2>Fully overflow-clipped</h2>
+        <div class="clip-box">
+          <canvas class="fully-clipped" aria-label="fixture-fully-overflow-clipped" width="80" height="50"></canvas>
+        </div>
+      </section>
+
+      <section class="case">
+        <h2>CSS-hidden variants</h2>
+        <canvas aria-label="fixture-display-none" width="80" height="40" style="display: none"></canvas>
+        <canvas aria-label="fixture-visibility-hidden" width="80" height="40" style="visibility: hidden"></canvas>
+        <canvas aria-label="fixture-opacity-zero" width="80" height="40" style="opacity: 0"></canvas>
+      </section>
+
+      <section class="case">
+        <h2>Hidden ancestor variants</h2>
+        <div style="opacity: 0"><canvas aria-label="fixture-parent-opacity-zero" width="80" height="40"></canvas></div>
+        <div hidden><canvas aria-label="fixture-hidden-attribute" width="80" height="40"></canvas></div>
+        <div inert><canvas aria-label="fixture-inert-parent" width="80" height="40"></canvas></div>
+        <div aria-hidden="true"><canvas aria-label="fixture-aria-hidden-parent" width="80" height="40"></canvas></div>
+      </section>
+
+      <section class="case">
+        <h2>Zero-size and offscreen</h2>
+        <canvas aria-label="fixture-zero-size" width="80" height="40" style="width: 0; height: 0; border: 0"></canvas>
+        <canvas aria-label="fixture-offscreen-right" width="80" height="40" style="position: fixed; left: 5000px; top: 0"></canvas>
+      </section>
+    </main>
+  </body>
+</html>
+

apps/extension/test-fixtures/rendered-surfaces/unlabeled.html

@@ -0,0 +1,68 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <title>Unlabeled Rendered Surface matrix</title>
+    <style>
+      * { box-sizing: border-box; }
+      html, body { margin: 0; min-height: 100%; font: 14px/1.4 system-ui, sans-serif; }
+      body { padding: 16px; background: #f4f6f8; color: #17202a; }
+      h1 { margin: 0 0 12px; font-size: 20px; }
+      .matrix { display: grid; grid-template-columns: repeat(3, minmax(240px, 1fr)); gap: 12px; }
+      .case { min-height: 150px; padding: 10px; overflow: hidden; border: 1px solid #aeb6bf; border-radius: 6px; background: white; }
+      .case h2 { margin: 0 0 8px; font-size: 13px; }
+      .case p { margin: 0 0 8px; }
+      canvas { display: block; background: #d6eaf8; border: 2px solid #2874a6; }
+      .stack { position: relative; width: 180px; height: 90px; }
+      .stack canvas { position: absolute; inset: 0; width: 180px; height: 90px; }
+      .stack canvas + canvas { background: rgb(244 208 63 / 45%); border-color: #b7950b; }
+      .contained { position: relative; width: 190px; height: 100px; }
+      .contained > canvas:first-child { width: 190px; height: 100px; }
+      .contained > canvas:last-child { position: absolute; left: 55px; top: 30px; width: 70px; height: 40px; background: #fadbd8; border-color: #c0392b; }
+    </style>
+  </head>
+  <body>
+    <h1>Unlabeled Rendered Surface matrix</h1>
+    <main class="matrix">
+      <section class="case">
+        <h2>Only engineering attributes</h2>
+        <canvas id="must-not-be-a-label" class="must-not-be-a-label" data-testid="must-not-be-a-label" width="200" height="90"></canvas>
+      </section>
+
+      <section class="case">
+        <h2>Blank explicit names</h2>
+        <canvas aria-label="   " title="   " width="180" height="80"></canvas>
+      </section>
+
+      <section class="case">
+        <h2>Nearby prose is not a Canvas name</h2>
+        <p>must-not-be-inferred-as-a-surface-label</p>
+        <canvas width="180" height="80"></canvas>
+      </section>
+
+      <section class="case">
+        <h2>Fallback AX content does not name or suppress the Canvas</h2>
+        <canvas width="180" height="80">
+          <button type="button">Fallback control remains separate</button>
+        </canvas>
+      </section>
+
+      <section class="case">
+        <h2>Unnamed rendering layers still cluster</h2>
+        <div class="stack">
+          <canvas width="180" height="90"></canvas>
+          <canvas width="180" height="90"></canvas>
+        </div>
+      </section>
+
+      <section class="case">
+        <h2>Unnamed contained canvases stay independent</h2>
+        <div class="contained">
+          <canvas width="190" height="100"></canvas>
+          <canvas width="70" height="40"></canvas>
+        </div>
+      </section>
+    </main>
+  </body>
+</html>

apps/extension/test-fixtures/rendered-surfaces/viewport.html

@@ -0,0 +1,22 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <title>Rendered Surface viewport matrix</title>
+    <style>
+      html, body { margin: 0; font: 14px system-ui, sans-serif; }
+      body { min-height: 1800px; padding: 16px; }
+      canvas { display: block; background: #fdebd0; border: 2px solid #ca6f1e; }
+      #partial { position: fixed; left: -70px; top: 120px; width: 180px; height: 100px; }
+      #below { position: absolute; left: 40px; top: 1350px; width: 240px; height: 120px; }
+    </style>
+  </head>
+  <body>
+    <h1>Viewport matrix</h1>
+    <p>The orange canvas on the left is intentionally clipped by the viewport.</p>
+    <canvas id="partial" aria-label="fixture-partial-viewport" width="180" height="100"></canvas>
+    <canvas id="below" aria-label="fixture-below-fold" width="240" height="120"></canvas>
+  </body>
+</html>
+

crates/bsk-cli/skill/SKILL.md

@@ -73,6 +73,16 @@ CSS-only hover menu looks like from here. It hovers a bounded set of likely trig
 few seconds and touches the live page; once you know which element hides the menu, `bsk hover <ref>`
 is cheaper and more precise.
 
+When `bsk observe` renders
+`@eN surface ... [visual-only; requires=image-understanding; ...]`, the ref identifies rendered
+canvas content that is not represented by the text observation, not an interactable DOM control.
+Keep using any reliable DOM/AX text and controls that appear alongside it. If you can actually
+inspect image output, use `bsk screenshot --ref @eN --session <id>` to obtain the visible crop. If
+you lack multimodal or image-reading capability, do not take a screenshot and pretend to know its
+contents, and never guess coordinates; tell the user that they need to switch to a model with
+image-understanding capability. Do not pass a visual surface ref to click, fill, hover, or select;
+those commands intentionally reject screenshot-only refs.
+
 Escalate page reading only as needed:
 
 1. `bsk observe` for normal semantic understanding, text, controls, and refs.
@@ -116,15 +126,156 @@ navigate   navigate-back   navigate-forward   reload   wait-for-navigation   wai
 observe   snapshot   get-html   screenshot   console   network
 click   hover   fill   select   press   evaluate
 tab list|create|close|select|borrow|return   window resize   emulate
-upload   download   request-help   record start|stop
+request-help   record start|stop
 ```
 
 Required flags that are easy to get wrong:
 
-```text
-bsk fill <ref> --value <text>      bsk select <ref> --value <option-value>
-bsk screenshot --out <path>        bsk emulate --device <preset-id>
-bsk upload <ref> --file <path>     bsk download <ref> --out <path>
+### Tabs (require `--session <id>`)
+
+| Command | Summary |
+|---------|---------|
+| `bsk tab list` | List tabs (`--scope user\|agent\|all`, default `all`) |
+| `bsk tab create` | New tab in Agent Window (`--url`, `--no-active`, `--index`) |
+| `bsk tab close <tab-id>` | Close an agent tab |
+| `bsk tab select <tab-id>` | Focus an agent tab |
+| `bsk tab borrow <tab-id>` | Move a user tab into the Agent Window |
+| `bsk tab return <tab-id>` | Return a borrowed tab to its original window |
+
+### Observation (require `--session` unless noted)
+
+| Command | Summary |
+|---------|---------|
+| `bsk snapshot` | First-choice static page understanding: accessibility tree with `@eN` element refs |
+| `bsk observe` | Semantic VOM observation with bounded perception probes for conditional surfaces |
+| `bsk get-html` | Raw HTML dump after snapshot is insufficient (high token cost) |
+| `bsk screenshot` | PNG capture after snapshot is insufficient: full visible tab, or `--ref @eN` to crop to one element (`--out` path optional) |
+
+### Console & network debugging (read-only; require `--session`)
+
+| Command | Summary |
+|---------|---------|
+| `bsk console` | Buffered page console messages, JS exceptions, and browser log entries (`--include-stack` for stack traces) |
+| `bsk network` | Buffered network responses (status, method, URL, MIME/resource type) and failures (`net::ERR_*` reason) |
+
+Both capture from the moment the tab is attached and read a bounded per-tab buffer: `--since <seq>` pages from a cursor (`next_since` in the result), `--limit` (default 50, max 200), `--max-text-chars` (default 1000, max 4096), `--tab-id` to target a non-active tab. Both are strictly read-only — they never intercept or modify traffic, and request/response headers, bodies, and timings are not captured.
+
+### Navigation
+
+| Command | Summary |
+|---------|---------|
+| `bsk navigate <url>` | Go to URL in agent tab (`--wait-until`, `--timeout`) |
+| `bsk navigate-back` | History back one step |
+| `bsk navigate-forward` | History forward one step |
+| `bsk reload` | Reload current tab (`--hard` bypass cache) |
+
+(`bsk navigate back` / `bsk navigate forward` are equivalent subcommands.)
+
+### Interaction
+
+| Command | Summary |
+|---------|---------|
+| `bsk click <ref-or-selector>` | Click element (`--button`, `--click-count`, `--modifiers`) |
+| `bsk hover <ref-or-selector>` | Move the mouse to an element and wait for hover UI to settle (`--settle`, `--modifiers`) |
+| `bsk fill <ref-or-selector> --value <text>` | Clear and type into input |
+| `bsk select <ref-or-selector> --value <v>` | Set `<select>` option(s) by `value` (repeat `--value` for multi-select) |
+| `bsk press <key>` | Key/combo (`Enter`, `Ctrl+A`, …; optional `--ref` to focus first) |
+
+### File transfer (require `--session`)
+
+| Command | Summary |
+|---------|---------|
+| `bsk upload <ref-or-selector> --file <path>` | Click one upload trigger and attach an agent-readable local file (`--file` is repeatable) |
+| `bsk download <ref-or-selector> --out <path>` | Click one download trigger and copy the single completed file to an exact local path (`--overwrite` is opt-in) |
+
+The agent/harness decides whether a file transfer is appropriate and which local path belongs to the task. Treat upload as disclosure of that file to the current website, and download as accepting website-controlled bytes onto the local filesystem. Use only paths that are necessary for the user's bounded goal.
+
+BrowserSkill enforces the mechanical boundary: files are staged under a session-scoped opaque transfer, only daemon-minted capabilities reach the extension, upload/download still obey Agent Window tab checks, and transfers are chunk/size bounded. Upload intercepts the native chooser for one transaction, locates the input activated in the resolved target's document, and assigns only the staged file paths. Download uniquely correlates one exact-target browser intent with one Chrome download in either event order, routes it through a daemon-minted relative directory, and lets the daemon validate and import it. Upload staging remains available for a later form submission and is removed when the session ends. Downloads cannot overwrite an existing destination unless `--overwrite` is explicit. BrowserSkill does not inspect file content or decide whether its meaning is sensitive.
+
+Do not use `request-help` merely because a native file chooser or browser download is involved; try these commands first. For transfer failures, use the structured error instead of retrying blindly:
+
+- `reason=file_input_not_activated` means the requested click did not activate exactly one `<input type="file">`; the page may use a non-input picker such as `window.showOpenFilePicker()`. Do not retry blindly. If human help is available, call `request-help` and tell the user the exact original local path to choose. The staged daemon path is internal and must not be shown to the user.
+- `reason=file_input_probe_failed` means BrowserSkill could not safely establish the browser-side upload transaction. Do not repeat the same action; use `request-help` when available.
+- `reason=set_file_input_failed` means BrowserSkill found the activated file input but Chrome rejected the staged path or assignment. Check the extension's file-URL access permission; otherwise use `request-help`.
+- `reason=download_capture_failed` means BrowserSkill could not attribute exactly one completed download to the requested target. Do not retry blindly or accept an unrelated browser download; use `request-help` when available.
+- `effect_state=none` means BrowserSkill confirmed that no file-transfer effect was committed. Follow the accompanying reason; a corrected target or explicit human fallback may be attempted.
+- `effect_state=unknown` means the browser may already have attached or created the file. Do not repeat the transfer. Observe the page if that can establish the result; otherwise stop and report the uncertainty.
+- `effect_state=committed` means the browser-side effect occurred even if later completion or cleanup failed. Do not repeat it; continue only after verifying the resulting page/download state.
+
+If `request-help` returns `outcome="disabled"`, do not retry it. Stop gracefully and report the transfer mechanism that requires human intervention.
+
+### Scripting & timing
+
+| Command | Summary |
+|---------|---------|
+| `bsk evaluate <expression>` | Run JS in agent tab (see red lines). JS throw → stderr, **exit 0** (RPC success); use `--json` and check `.ok` to detect JS errors |
+| `bsk wait-for-navigation` | Block until load/DOM idle/etc. (`--wait-until`, `--timeout`) |
+| `bsk wait-ms <duration>` | Sleep (`500ms`, `2s`, `1m`; **no** `--session`) |
+
+### Ask the human for help — `bsk request-help`
+
+When a step needs a human (captcha, login, OTP) or you want the user to
+confirm an important action, pause and ask:
+
+    bsk request-help --session <id> --prompt "Solve the captcha, then click Done only after the site accepts it" \
+      --title "Captcha required" --target @e7 --target "#submit" --timeout 5m
+
+- `--prompt` (required): what the user should do.
+- `--title` (optional): custom title for the overlay panel. When omitted,
+  the extension shows its default localized title.
+- `--target` (repeatable): a snapshot ref (`@e7`) or CSS selector
+  (`#submit`) to scroll to and flash-highlight. **Strongly recommended** —
+  whenever the prompt refers to a concrete element (a button to click, a
+  field to fill, a checkbox to toggle), pass its `@eN` ref / selector so the
+  user is guided straight to the right spot instead of hunting for it. For
+  interaction scenarios, always include the relevant target(s); reserve a
+  prompt with no `--target` for cases where there is genuinely no specific
+  element to point at (e.g. "wait for the page to finish loading").
+- `--timeout` (default `5m`): how long to wait.
+- `--completion-criteria` (optional): JSON success detector. Use it only
+  when there is a concrete post-help success signal, e.g.
+  `{"any":[{"url_contains":"/dashboard"},{"selector_exists":"[data-testid='account-menu']"}],"stable_for_ms":1000}`.
+
+The target tab is brought to the foreground; the page stays interactive
+while the agent control mask is hidden. The call blocks until the user
+explicitly acts, the timeout expires, cancellation arrives, or explicit
+completion criteria match. Page reloads, SPA route changes, and captcha
+refreshes do not return control by themselves. The result `outcome` is one of:
+
+- `continued` — the user finished and clicked Done / return control (treat as confirm).
+- `cancelled` — the user clicked Cancel (treat as reject/abort).
+- `timed_out` — nobody acted within the timeout.
+- `completed` — the explicit `--completion-criteria` matched while the user had control.
+- `navigated` — deprecated legacy outcome. Do not rely on navigation as a completion signal.
+
+`note` carries any text the user typed back. `resolved_targets` reports
+which refs/selectors matched a live element.
+
+`request-help` does not refresh the page model after the user returns
+control. After a `continued` or `completed` result, issue a separate
+observation tool call (usually `bsk snapshot --session <id>`) before using
+new refs or reasoning about the post-help page state.
+#### Disabling request-help (unattended mode)
+Set `BSK_REQUEST_HELP=off` on unattended servers: `bsk request-help` then
+returns immediately with `outcome="disabled"` (no overlay, no waiting,
+exit 0). Any other value keeps it enabled. If you get `disabled`, do not
+retry — complete the task autonomously or stop gracefully.
+
+### Recording — `bsk record`
+
+Capture the user's own actions in the Agent Window to a **trace bundle**, for later LLM-driven automation. New CLI builds request **trace v3** (page observations + action chain); older extensions may still return **trace v2** (actions only), which the CLI exports as a single `trace.json` without `states/`.
+
+```bash
+bsk record start --browser <instance-id-or-label> \
+  [--url https://…] [--purpose "publish a wiki doc"] \
+  [--max-page-tokens 3000] [--redact-values] \
+  [--output trace]
+# `--url` is optional; default https://example.com/ when omitted (must be http(s)).
+# Blocks until the user clicks Finish in the recording panel, then writes:
+#   trace/trace.json    — action chain (+ state index when v3)
+#   trace/states/       — v3 only: one `sN.txt` observe snapshot per settled page state
+
+bsk record stop [--output trace]   # terminal fallback if the browser panel is unavailable
 ```
 
 `select` matches an option's `value` attribute, not its visible label. Device preset ids are
@@ -139,34 +290,6 @@ lowercase and hyphenated, such as `iphone-14`.
 - `record` captures a user's actions for later replay. Read `bsk record start --help` before use,
   and never record banking, SSO, password-manager, or other sensitive pages.
 
-## File transfer
-
-`upload` and `download` stage files through the daemon; the agent never touches browser-internal
-paths. Treat upload as disclosure to the website, download as accepting website-controlled bytes.
-
-Upload has two independent mechanisms — choose explicitly, never rely on automatic fallback:
-
-- **Default (input mode):** for upload buttons, file-input labels, or "upload from computer"
-  actions. The command clicks the target and intercepts the native file chooser.
-- **`--mode drop`:** for reliably identified attachment-receiving areas — an explicit drop zone,
-  chat composer, email editor, or form attachment area. Do not target page whitespace, generic
-  containers, or areas whose attachment ownership is ambiguous.
-
-Decision sequence when uploading:
-
-1. Try input mode (the default).
-2. If it returns `reason=file_input_not_activated` with `effect_state=none`, re-observe. When a
-   reliable attachment target exists, try `--mode drop` once against that target.
-3. Otherwise fall back to `request-help`.
-4. **Never** switch mechanisms or repeat when `effect_state` is `unknown` or `committed` — the
-   browser may already have applied the file.
-
-A successful drop means Chrome dispatched the native file-drop event; it does not prove the site
-accepted the attachment. Observe the page once after the command.
-
-Download default-refuses to overwrite; pass `--overwrite` when replacing an existing file is
-intended. Read `bsk upload --help` and `bsk download --help` for all flags and error details.
-
 ## Recover without wandering
 
 - Stale ref: observe again and retry the intended action once.

crates/bsk-cli/src/cli/mod.rs

@@ -182,7 +182,7 @@ pub enum Command {
     /// Set `<select>` option values by `value` attribute.
     Select(SelectArgs),
 
-    /// Upload files through a page file input or explicit drop target.
+    /// Upload one or more local files through a page file chooser.
     Upload(UploadArgs),
 
     /// Capture one browser download and write it to a local path.

crates/bsk-cli/src/cli/render_error.rs

@@ -40,6 +40,7 @@ use bsk_protocol::ErrorCode;
 pub mod reason {
     pub const AGENT_WINDOW_SCOPE: &str = "agent_window_scope";
     pub const ELEMENT_NOT_VISIBLE: &str = "element_not_visible";
+    pub const REF_CAPABILITY_DENIED: &str = "ref_capability_denied";
     pub const REF_NOT_FOUND: &str = "ref_not_found";
     pub const SELECTOR_NOT_FOUND: &str = "selector_not_found";
     pub const TARGET_NOT_FILLABLE: &str = "target_not_fillable";
@@ -52,9 +53,6 @@ pub mod reason {
     pub const FILE_INPUT_PROBE_FAILED: &str = "file_input_probe_failed";
     pub const FILE_INPUT_NOT_ACTIVATED: &str = "file_input_not_activated";
     pub const SET_FILE_INPUT_FAILED: &str = "set_file_input_failed";
-    pub const UPLOAD_MECHANISM_UNSUPPORTED: &str = "upload_mechanism_unsupported";
-    pub const FILE_DROP_TARGET_UNAVAILABLE: &str = "file_drop_target_unavailable";
-    pub const FILE_DROP_FAILED: &str = "file_drop_failed";
     pub const DOWNLOAD_CAPTURE_FAILED: &str = "download_capture_failed";
     pub const TRANSFER_OUTCOME_UNKNOWN: &str = "transfer_outcome_unknown";
     pub const TRANSFER_TIMEOUT: &str = "transfer_timeout";
@@ -251,6 +249,13 @@ pub fn info_for_error(code: ErrorCode, data: Option<&serde_json::Value>) -> Rend
             ),
             exit_code: base.exit_code,
         },
+        (ErrorCode::PermissionDenied, reason::REF_CAPABILITY_DENIED) => RenderInfo {
+            summary: "snapshot ref does not support this operation",
+            hint: Some(
+                "visual surface refs are screenshot-only; inspect `bsk screenshot --ref <ref> --session <id>` with image-understanding capability",
+            ),
+            exit_code: base.exit_code,
+        },
         (ErrorCode::InvalidParams, reason::TARGET_NOT_SELECT) => RenderInfo {
             summary: "target element is not a <select>",
             hint: Some(
@@ -306,7 +311,7 @@ pub fn info_for_error(code: ErrorCode, data: Option<&serde_json::Value>) -> Rend
         (ErrorCode::Unsupported, reason::FILE_INPUT_NOT_ACTIVATED) => RenderInfo {
             summary: "the upload trigger did not activate a file input",
             hint: Some(
-                "the page may use a non-input picker such as showOpenFilePicker(); when effect_state is none, re-observe and use `--mode drop` once only for a reliably identified attachment-receiving area — otherwise use `bsk request-help` with the exact original local path",
+                "the page may use a non-input picker such as showOpenFilePicker(); do not retry blindly — use `bsk request-help` with the exact original local path, or stop if human help is disabled",
             ),
             exit_code: base.exit_code,
         },
@@ -326,30 +331,6 @@ pub fn info_for_error(code: ErrorCode, data: Option<&serde_json::Value>) -> Rend
             ),
             exit_code: base.exit_code,
         },
-        (ErrorCode::Unsupported, reason::UPLOAD_MECHANISM_UNSUPPORTED) => RenderInfo {
-            summary: "the browser does not support native file-drop upload",
-            hint: Some(
-                "do not retry the same drop; use a standard file-input target or `bsk request-help`",
-            ),
-            exit_code: base.exit_code,
-        },
-        (
-            ErrorCode::PermissionDenied | ErrorCode::Timeout | ErrorCode::CdpFailed,
-            reason::FILE_DROP_TARGET_UNAVAILABLE,
-        ) => RenderInfo {
-            summary: "the file-drop target could not be safely resolved",
-            hint: Some(
-                "rerun observe and select the attachment-receiving editor or drop zone itself; do not guess another target — no file was delivered when effect_state is none",
-            ),
-            exit_code: base.exit_code,
-        },
-        (ErrorCode::Timeout | ErrorCode::CdpFailed, reason::FILE_DROP_FAILED) => RenderInfo {
-            summary: "the browser could not complete the native file drop",
-            hint: Some(
-                "do not retry when effect_state is unknown; observe the page first, otherwise use `bsk request-help`",
-            ),
-            exit_code: base.exit_code,
-        },
         (ErrorCode::CdpFailed, reason::DOWNLOAD_CAPTURE_FAILED) => RenderInfo {
             summary: "the browser download could not be attributed or completed",
             hint: Some(
@@ -454,6 +435,18 @@ mod tests {
         assert!(!info.summary.contains("sandbox"));
     }
 
+    #[test]
+    fn ref_capability_denied_explains_visual_surface_refs() {
+        let data = serde_json::json!({ "reason": reason::REF_CAPABILITY_DENIED });
+        let info = info_for_error(ErrorCode::PermissionDenied, Some(&data));
+        assert_eq!(info.summary, "snapshot ref does not support this operation");
+        assert!(
+            info.hint.unwrap().contains("screenshot-only"),
+            "expected visual-surface-specific hint"
+        );
+        assert_eq!(info.exit_code, 1);
+    }
+
     #[test]
     fn session_busy_overrides_timeout_copy() {
         let data = serde_json::json!({ "reason": reason::SESSION_BUSY });
@@ -504,10 +497,7 @@ mod tests {
             info.summary,
             "the upload trigger did not activate a file input"
         );
-        let hint = info.hint.unwrap();
-        assert!(hint.contains("--mode drop"));
-        assert!(hint.contains("reliably identified attachment-receiving area"));
-        assert!(hint.contains("exact original local path"));
+        assert!(info.hint.unwrap().contains("exact original local path"));
 
         let control = serde_json::json!({ "reason": reason::FILE_INPUT_PROBE_FAILED });
         let info = info_for_error(ErrorCode::Timeout, Some(&control));
@@ -517,29 +507,6 @@ mod tests {
         );
         assert!(info.hint.unwrap().contains("do not retry"));
 
-        let unsupported_drop =
-            serde_json::json!({ "reason": reason::UPLOAD_MECHANISM_UNSUPPORTED });
-        let info = info_for_error(ErrorCode::Unsupported, Some(&unsupported_drop));
-        assert_eq!(
-            info.summary,
-            "the browser does not support native file-drop upload"
-        );
-        assert!(info.hint.unwrap().contains("standard file-input"));
-
-        let unavailable_target =
-            serde_json::json!({ "reason": reason::FILE_DROP_TARGET_UNAVAILABLE });
-        let info = info_for_error(ErrorCode::PermissionDenied, Some(&unavailable_target));
-        assert_eq!(
-            info.summary,
-            "the file-drop target could not be safely resolved"
-        );
-        assert!(info.hint.unwrap().contains("effect_state is none"));
-
-        let failed_drop = serde_json::json!({ "reason": reason::FILE_DROP_FAILED });
-        let info = info_for_error(ErrorCode::Timeout, Some(&failed_drop));
-        assert!(info.summary.contains("native file drop"));
-        assert!(info.hint.unwrap().contains("effect_state is unknown"));
-
         let download = serde_json::json!({ "reason": reason::DOWNLOAD_CAPTURE_FAILED });
         let info = info_for_error(ErrorCode::CdpFailed, Some(&download));
         assert!(info.summary.contains("download could not be attributed"));

crates/bsk-cli/src/cli/upload.rs

@@ -10,35 +10,19 @@ use base64::Engine;
 use bsk_protocol::Method;
 use bsk_protocol::tools::{
     TransferBeginParams, TransferBeginResult, TransferChunkParams, TransferChunkResult,
-    TransferIdParams, TransferReadyResult, TransferReleaseResult, UploadFile, UploadMode,
-    UploadParams, UploadResult,
+    TransferIdParams, TransferReadyResult, TransferReleaseResult, UploadFile, UploadParams,
+    UploadResult,
 };
-use clap::{Args, ValueEnum};
+use clap::Args;
 
 use crate::cli::ensure_daemon::ensure_daemon;
 use crate::cli::error::{CliError, Format};
 use crate::cli::interaction::split_target;
 use crate::cli::navigate::parse_timeout_ms;
 
-#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, ValueEnum)]
-pub enum UploadModeArg {
-    #[default]
-    Input,
-    Drop,
-}
-
-impl From<UploadModeArg> for UploadMode {
-    fn from(value: UploadModeArg) -> Self {
-        match value {
-            UploadModeArg::Input => Self::Input,
-            UploadModeArg::Drop => Self::Drop,
-        }
-    }
-}
-
 #[derive(Debug, Clone, Args)]
 pub struct UploadArgs {
-    /// Snapshot ref (`@e3`) or CSS selector for the chooser trigger / drop target.
+    /// Snapshot ref (`@e3`) or CSS selector for the file input / chooser trigger.
     pub target: Option<String>,
     #[arg(long = "ref")]
     pub ref_: Option<String>,
@@ -47,9 +31,6 @@ pub struct UploadArgs {
     /// Local file to upload. Repeat for a multiple-file input.
     #[arg(long = "file", required = true)]
     pub files: Vec<PathBuf>,
-    /// Browser mechanism used to deliver the staged files.
-    #[arg(long, value_enum, default_value_t)]
-    pub mode: UploadModeArg,
     #[arg(long)]
     pub session: String,
     #[arg(long = "tab-id")]
@@ -92,7 +73,6 @@ pub fn dispatch(args: UploadArgs, format: Format) -> Result<(), CliError> {
                 staged_path: None,
             })
             .collect(),
-        mode: args.mode.into(),
         timeout_ms: Some(args.timeout),
     };
     let result = crate::cli::business_rpc::call::<_, UploadResult>(

crates/bsk-cli/tests/cli_parse.rs

@@ -6,7 +6,6 @@ use bsk::cli::daemon::{DaemonCmd, parse_duration};
 use bsk::cli::navigate::NavigateCmd;
 use bsk::cli::record::{RecordCmd, RecordSub};
 use bsk::cli::session::{SessionCmd, SessionSub};
-use bsk::cli::upload::UploadModeArg;
 use bsk::{Cli, Command};
 use clap::Parser;
 
@@ -14,54 +13,6 @@ fn parse(args: &[&str]) -> Cli {
     Cli::try_parse_from(args).expect("clap parse should succeed")
 }
 
-#[test]
-fn parses_upload_modes() {
-    let cli = parse(&[
-        "bsk",
-        "upload",
-        "@e1",
-        "--file",
-        "image.png",
-        "--session",
-        "s1",
-    ]);
-    let Command::Upload(default_args) = cli.command else {
-        panic!("expected upload subcommand");
-    };
-    assert_eq!(default_args.mode, UploadModeArg::Input);
-
-    let cli = parse(&[
-        "bsk",
-        "upload",
-        "@e1",
-        "--file",
-        "image.png",
-        "--session",
-        "s1",
-        "--mode",
-        "drop",
-    ]);
-    let Command::Upload(drop_args) = cli.command else {
-        panic!("expected upload subcommand");
-    };
-    assert_eq!(drop_args.mode, UploadModeArg::Drop);
-
-    assert!(
-        Cli::try_parse_from([
-            "bsk",
-            "upload",
-            "@e1",
-            "--file",
-            "image.png",
-            "--session",
-            "s1",
-            "--mode",
-            "auto",
-        ])
-        .is_err()
-    );
-}
-
 #[test]
 fn parses_daemon_start_with_defaults() {
     let cli = parse(&["bsk", "daemon", "start"]);

crates/bsk-protocol/schema/tool_upload_params.json

@@ -13,9 +13,6 @@
         "$ref": "#/definitions/UploadFile"
       }
     },
-    "mode": {
-      "$ref": "#/definitions/UploadMode"
-    },
     "ref": {
       "type": [
         "string",
@@ -68,13 +65,6 @@
           "type": "string"
         }
       }
-    },
-    "UploadMode": {
-      "type": "string",
-      "enum": [
-        "input",
-        "drop"
-      ]
     }
   }
 }

crates/bsk-protocol/src/tools/file_transfer.rs

@@ -15,20 +15,6 @@ pub struct UploadFile {
     pub staged_path: Option<String>,
 }
 
-#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
-#[serde(rename_all = "snake_case")]
-pub enum UploadMode {
-    #[default]
-    Input,
-    Drop,
-}
-
-impl UploadMode {
-    fn is_input(&self) -> bool {
-        matches!(self, Self::Input)
-    }
-}
-
 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
 pub struct UploadParams {
     pub session_id: String,
@@ -44,8 +30,6 @@ pub struct UploadParams {
     #[serde(default, skip_serializing_if = "Option::is_none")]
     pub tab_id: Option<i64>,
     pub files: Vec<UploadFile>,
-    #[serde(default, skip_serializing_if = "UploadMode::is_input")]
-    pub mode: UploadMode,
     #[serde(default, skip_serializing_if = "Option::is_none")]
     pub timeout_ms: Option<u32>,
 }
@@ -167,32 +151,11 @@ mod tests {
                 name: "image.png".into(),
                 staged_path: None,
             }],
-            mode: UploadMode::Input,
             timeout_ms: None,
         })
         .unwrap();
         assert_eq!(value["ref"], "@e3");
         assert!(value["files"][0].get("staged_path").is_none());
-        assert!(value.get("mode").is_none());
-    }
-
-    #[test]
-    fn upload_drop_mode_is_explicit_on_the_wire() {
-        let value = serde_json::to_value(UploadParams {
-            session_id: "s1".into(),
-            ref_: Some("@e3".into()),
-            selector: None,
-            tab_id: None,
-            files: vec![UploadFile {
-                transfer_id: "tr_1".into(),
-                name: "image.png".into(),
-                staged_path: None,
-            }],
-            mode: UploadMode::Drop,
-            timeout_ms: None,
-        })
-        .unwrap();
-        assert_eq!(value["mode"], "drop");
     }
 
     #[test]

docs/architecture.md

@@ -148,7 +148,7 @@ flowchart LR
 - The invoking agent/harness decides whether a transfer is authorized and supplies the task-local source or destination path.
 - The CLI is the only component that reads an upload source or writes the final download destination. Before browser dispatch it owns rollback of partially staged uploads; after dispatch, ownership moves to the session because a transport timeout cannot prove that Chrome did not attach the file. Download output becomes visible through one atomic commit, and replacement is opt-in without a pre-delete window. The extension never receives either agent-facing path.
 - The daemon is the authority for storage capabilities and limits. It issues opaque session-scoped transfer IDs, stages bounded chunks in a private runtime directory, and injects only private staged upload paths. For download it mints one relative Chrome directory capability. Only after validating the reported path, file type, symlink boundary, and authoritative byte limit does it take ownership of browser-file cleanup and import the bytes.
-- The extension owns only the browser transaction. Every transfer resolves one `ResolvedActionTarget`. The default upload mechanism arms Chrome's chooser interception before clicking, then accepts either an exact `Page.fileChooserOpened` input node or an independent probe anchored in the trigger node's document; one verified input is committed with `DOM.setFileInputFiles`, while a non-input picker is rejected immediately. Explicit drop mode performs no click and never falls back to the chooser mechanism: after geometry resolution it temporarily excludes BrowserSkill's own overlay, verifies that the resolved drop zone still owns its local action point, and sends one native `dragEnter` / `dragOver` / `drop` transaction to that node's CDP target before restoring the overlay. OOPIF drops use target-local coordinates rather than top-level click coordinates. Download correlates exact-target CDP intent and `chrome.downloads` filename candidates in either arrival order, claims only one unique match, and never cancels an unclaimed candidate.
+- The extension owns only the browser transaction. Both tools resolve one `ResolvedActionTarget` and use that same target for protocol setup and the triggering click. Upload arms Chrome's chooser interception before clicking, then accepts either an exact `Page.fileChooserOpened` input node or an independent probe anchored in the trigger node's own document; one verified input is committed with `DOM.setFileInputFiles`. A non-input picker is rejected immediately as unsupported. Download correlates exact-target CDP intent and `chrome.downloads` filename candidates in either arrival order, claims only one unique match, and never cancels an unclaimed candidate.
 - Browser-side operations report `effect_state` (`none`, `committed`, or `unknown`), `phase`, and `cleanup_state`. Confirmed success wins over a late cancel; an unknown effect is preserved across timeout or transport loss and must not be retried blindly. A transfer deadline sends cancellation to the extension and keeps the session queue occupied for bounded compensation rather than abandoning an in-flight browser effect.
 - Download staging is released after CLI commit. Upload staging remains until session teardown because the page may read an attached file only on a later form submission. Remaining staging is released on session stop/browser disconnect and on daemon startup after a crash. BrowserSkill does not inspect content or decide whether a transfer is appropriate.
 

packages/dsh-plugin-browserskill/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@wxg-prc-cpg/browser-skill-dsh-plugin",
   "description": "DeepSeek Harness tool plugin that exposes BrowserSkill browser automation (browser_* tools) to the model",
-  "version": "0.2.0",
+  "version": "0.1.2",
   "license": "MIT",
   "publishConfig": {
     "access": "public",

packages/dsh-plugin-browserskill/src/tools.ts

@@ -489,7 +489,9 @@ function defineBrowserOperations(deps: ToolDeps, register: DefinitionRegistrar):
       ? "Capture an indented aria-tree snapshot of the session's active tab. Interactive elements " +
         "carry @eN refs for browser_interact actions. Prefer browser_inspect action=observe for a richer semantic view."
       : "Produce a semantic VOM observation of the session's active tab (roles, states, perception " +
-        "probes) with @eN refs for browser_interact actions. Read-only: never submits input.";
+        "probes) with @eN refs for browser_interact actions. Visual-only surfaces require image " +
+        "understanding and their refs are screenshot-only; keep using any DOM/AX semantics rendered " +
+        "alongside them. Read-only: never submits input.";
     register(
       defineTool({
         name,
@@ -752,12 +754,17 @@ function defineBrowserOperations(deps: ToolDeps, register: DefinitionRegistrar):
       name: "inspect.screenshot",
       description:
         "Capture a PNG screenshot of the session's active tab, or crop to a snapshot ref element. " +
+        "This is the supported operation for visual surface refs emitted by inspect.observe. Only use " +
+        "it to interpret a visual-only surface when you can inspect image output. If you lack multimodal " +
+        "or image-reading capability, tell the user to switch to a model with image understanding; do " +
+        "not guess content or coordinates. " +
         "Returns the image itself when the deployment supports image input, otherwise a file path.",
       parameters: {
         session: SESSION_PARAM,
         ref: {
           type: "string",
-          description: "Snapshot ref (@e3) from the last snapshot/observe; crops to that element.",
+          description:
+            "Ref (@e3) from the last snapshot/observe; crops to that DOM element or visual surface.",
         },
       },
       output: {

packages/vom/src/__tests__/render.test.ts

@@ -29,6 +29,139 @@ function coreRefs(refs: ReturnType<typeof renderVom>["refs"]) {
 }
 
 describe("renderVom single-layer page", () => {
+  it("renders visual surfaces as screenshot-only refs", () => {
+    const input = scene([
+      node({ id: 1, role: "RootWebArea", frameId: "main" }),
+      node({ id: 2, parentId: 1, role: "Iframe", frameId: "main", backendNodeId: 20 }),
+    ]);
+    input.visualSurfaces = [
+      {
+        parentId: 2,
+        backendNodeId: 99,
+        frameId: "child",
+        renderingKind: "canvas",
+        visibleRect: { x: 10, y: 20, w: 800, h: 500 },
+        label: "Data grid",
+        memberCount: 2,
+      },
+    ];
+    const out = renderVom(input);
+
+    expect(out.text).toContain("[visual surfaces]");
+    expect(out.text).toContain(
+      '@e1 surface "Data grid" [bounds=10,20,800,500; rendering=canvas; layers=2; visual-only; requires=image-understanding; use: bsk screenshot --ref @e1]',
+    );
+    expect(out.refs).toContainEqual(
+      expect.objectContaining({
+        ref: "e1",
+        backendNodeId: 99,
+        kind: "surface",
+        visibleRect: { x: 10, y: 20, w: 800, h: 500 },
+        capabilities: ["screenshot"],
+      }),
+    );
+  });
+
+  it("renders an unnamed visual surface with a stable generic label", () => {
+    const input = scene([node({ id: 1, role: "RootWebArea", frameId: "main" })]);
+    input.visualSurfaces = [
+      {
+        parentId: 1,
+        backendNodeId: 99,
+        frameId: "main",
+        renderingKind: "canvas",
+        visibleRect: { x: 10, y: 20, w: 800, h: 500 },
+        memberCount: 1,
+      },
+    ];
+
+    const out = renderVom(input);
+
+    expect(out.text).toContain(
+      '@e1 surface "canvas visual surface" [bounds=10,20,800,500; rendering=canvas; visual-only; requires=image-understanding; use: bsk screenshot --ref @e1]',
+    );
+    expect(out.refs).toContainEqual(
+      expect.objectContaining({
+        ref: "e1",
+        name: "canvas visual surface",
+        kind: "surface",
+        capabilities: ["screenshot"],
+      }),
+    );
+  });
+
+  it("appends visual refs after DOM refs so canvas density cannot renumber controls", () => {
+    const input = scene([
+      node({ id: 1, role: "RootWebArea", frameId: "main" }),
+      node({ id: 2, parentId: 1, role: "button", name: "Submit", tag: "button" }),
+    ]);
+    input.visualSurfaces = [
+      {
+        parentId: 1,
+        backendNodeId: 99,
+        frameId: "main",
+        renderingKind: "canvas",
+        visibleRect: { x: 10, y: 20, w: 80, h: 50 },
+        memberCount: 1,
+      },
+    ];
+
+    const withoutVisuals = renderVom({ ...input, visualSurfaces: [] });
+    const withVisuals = renderVom(input);
+
+    expect(withoutVisuals.refs[0]).toMatchObject({ ref: "e1", backendNodeId: 2, kind: "dom" });
+    expect(withVisuals.refs[0]).toMatchObject({ ref: "e1", backendNodeId: 2, kind: "dom" });
+    expect(withVisuals.refs[1]).toMatchObject({ ref: "e2", backendNodeId: 99, kind: "surface" });
+    expect(withVisuals.text.indexOf('@e1 button "Submit"')).toBeLessThan(
+      withVisuals.text.indexOf("[visual surfaces]"),
+    );
+  });
+
+  it("bounds visual output and prefers explicitly labeled surfaces", () => {
+    const input = scene([node({ id: 1, role: "RootWebArea", frameId: "main" })]);
+    input.visualSurfaces = Array.from({ length: 12 }, (_, index) => ({
+      parentId: 1,
+      backendNodeId: 100 + index,
+      frameId: "main",
+      renderingKind: "canvas" as const,
+      visibleRect: { x: index * 10, y: 20, w: 8, h: 8 },
+      ...(index === 11 ? { label: "Important status" } : {}),
+      memberCount: 1,
+    }));
+
+    const out = renderVom(input);
+    const surfaceRefs = out.refs.filter((ref) => ref.kind === "surface");
+
+    expect(surfaceRefs).toHaveLength(8);
+    expect(surfaceRefs[0]).toMatchObject({ backendNodeId: 111, name: "Important status" });
+    expect(out.text).toContain("… 4 additional visual surfaces omitted");
+    expect(out.truncated).toBe(true);
+  });
+
+  it("does not let a visual surface exhaust the budget before later DOM content", () => {
+    const input = scene([
+      node({ id: 1, role: "RootWebArea", name: "Doc", frameId: "main" }),
+      node({ id: 2, parentId: 1, role: "button", name: "First", tag: "button" }),
+      node({ id: 3, parentId: 1, role: "button", name: "Later", tag: "button" }),
+    ]);
+    input.visualSurfaces = [
+      {
+        parentId: 1,
+        backendNodeId: 99,
+        frameId: "main",
+        renderingKind: "canvas",
+        visibleRect: { x: 10, y: 20, w: 800, h: 500 },
+        memberCount: 1,
+      },
+    ];
+
+    const out = renderVom(input, { maxTokens: 30 });
+
+    expect(out.text).toContain('@e2 button "Later"');
+    expect(out.refs.filter((ref) => ref.kind === "surface")).toHaveLength(0);
+    expect(out.truncated).toBe(true);
+  });
+
   it("does not derive handle context across frame scopes", () => {
     const out = renderVom(
       scene([

packages/vom/src/index.ts

@@ -15,6 +15,8 @@ export type {
   VomNode,
   VomOptions,
   VomRef,
+  VomRefCapability,
   VomResult,
   VomScene,
+  VomVisualSurface,
 } from "./types";

packages/vom/src/render.ts

@@ -9,6 +9,7 @@ import type {
   VomOptions,
   VomResult,
   VomScene,
+  VomVisualSurface,
 } from "./types";
 
 const SKIP_ROLES = new Set(["generic", "none", "presentation", "inlinetextbox"]);
@@ -87,6 +88,7 @@ const MAX_CUSTOM_RECOVERY_AREA = 180_000;
 const MAX_RECOVERABLE_SINGLE_INTENT_AREA = 360_000;
 const MAX_HANDLE_CONTEXT_ITEMS = 3;
 const MAX_SURFACE_ITEMS = 12;
+const MAX_VISUAL_SURFACES = 8;
 const MAX_SCOPE_LINES = 40;
 const RECOVERY_HANDLER_ATTRS = ["onclick", "onmousedown", "onkeydown", "onkeyup", "onkeypress"];
 const NATIVE_TAGS = new Set(["button", "input", "select", "textarea"]);
@@ -793,6 +795,108 @@ interface RenderState {
   duplicateRefNames: Set<string>;
   surfaceMap: Map<number, CondSurface>;
   scopeMap: Map<number, ActiveScopeBlock>;
+  visualSurfaces: VomVisualSurface[];
+}
+
+function visualSurfaceArea(surface: VomVisualSurface): number {
+  return Math.max(0, surface.visibleRect.w) * Math.max(0, surface.visibleRect.h);
+}
+
+function compareVisualSurfaces(a: VomVisualSurface, b: VomVisualSurface): number {
+  const aLabeled = cleaned(a.label) !== undefined;
+  const bLabeled = cleaned(b.label) !== undefined;
+  return (
+    Number(bLabeled) - Number(aLabeled) ||
+    visualSurfaceArea(b) - visualSurfaceArea(a) ||
+    a.frameId.localeCompare(b.frameId) ||
+    a.visibleRect.y - b.visibleRect.y ||
+    a.visibleRect.x - b.visibleRect.x ||
+    a.backendNodeId - b.backendNodeId
+  );
+}
+
+function visualSurfaceLine(surface: VomVisualSurface, ref: string): string {
+  const label = cleaned(surface.label) ?? `${surface.renderingKind} visual surface`;
+  const layers = surface.memberCount > 1 ? `; layers=${surface.memberCount}` : "";
+  const { x, y, w, h } = surface.visibleRect;
+  const bounds = [x, y, w, h].map(Math.round).join(",");
+  return `  @${ref} surface ${JSON.stringify(label)} [bounds=${bounds}; rendering=${surface.renderingKind}${layers}; visual-only; requires=image-understanding; use: bsk screenshot --ref @${ref}]`;
+}
+
+function selectVisualSurfaces(
+  surfaces: readonly VomVisualSurface[],
+  nodesById: ReadonlyMap<number, VomNode>,
+): { selected: VomVisualSurface[]; eligibleCount: number } {
+  const selected: VomVisualSurface[] = [];
+  let eligibleCount = 0;
+
+  for (const surface of surfaces) {
+    if (surface.parentId !== null && !nodesById.has(surface.parentId)) continue;
+    eligibleCount += 1;
+
+    const insertionIndex = selected.findIndex(
+      (candidate) => compareVisualSurfaces(surface, candidate) < 0,
+    );
+    if (insertionIndex < 0) {
+      if (selected.length < MAX_VISUAL_SURFACES) selected.push(surface);
+      continue;
+    }
+    selected.splice(insertionIndex, 0, surface);
+    if (selected.length > MAX_VISUAL_SURFACES) selected.pop();
+  }
+
+  return { selected, eligibleCount };
+}
+
+function emitVisualAppendix(state: RenderState): void {
+  if (state.stopped) return;
+  const { selected, eligibleCount } = selectVisualSurfaces(state.visualSurfaces, state.nodesById);
+  if (eligibleCount === 0) return;
+
+  const header = "[visual surfaces]";
+  let headerEmitted = false;
+  let emitted = 0;
+  for (const surface of selected) {
+    const ref = `e${state.nextRef}`;
+    const line = visualSurfaceLine(surface, ref);
+    const nextTokens =
+      state.tokens + estimateTokens(line) + (headerEmitted ? 0 : estimateTokens(header));
+    if (nextTokens > state.maxTokens) break;
+    if (!headerEmitted) {
+      state.lines.push(header);
+      state.tokens += estimateTokens(header);
+      headerEmitted = true;
+    }
+    state.lines.push(line);
+    state.tokens += estimateTokens(line);
+    const label = cleaned(surface.label) ?? `${surface.renderingKind} visual surface`;
+    state.refs.push({
+      ref,
+      backendNodeId: surface.backendNodeId,
+      frameId: surface.frameId,
+      visibleRect: surface.visibleRect,
+      role: "surface",
+      name: label,
+      kind: "surface",
+      capabilities: ["screenshot"],
+      line: state.lines.length - 1,
+    });
+    state.nextRef += 1;
+    emitted += 1;
+  }
+
+  const omitted = eligibleCount - emitted;
+  if (omitted > 0) {
+    state.truncated = true;
+    if (headerEmitted) {
+      const summary = `  … ${omitted} additional visual surfaces omitted`;
+      const nextTokens = state.tokens + estimateTokens(summary);
+      if (nextTokens <= state.maxTokens) {
+        state.lines.push(summary);
+        state.tokens = nextTokens;
+      }
+    }
+  }
 }
 
 function emitActiveScopeBlock(scope: ActiveScopeBlock, depth: number, state: RenderState): void {
@@ -873,6 +977,8 @@ function renderTree(children: Map<number | null, VomNode[]>, state: RenderState)
         ...(node.role ? { role: node.role } : {}),
         ...(name ? { name } : {}),
         ...(context.length > 0 ? { ctx: context.join(" > ") } : {}),
+        kind: "dom",
+        capabilities: ["interact", "screenshot"],
         line: state.lines.length - 1,
       });
       state.nextRef += 1;
@@ -894,6 +1000,7 @@ function renderNodes(
   initialLines: string[],
   surfaces: CondSurface[] = [],
   activeScopeBlocks: ActiveScopeBlock[] = [],
+  visualSurfaces: VomVisualSurface[] = [],
 ): RenderState {
   const children = buildChildren(nodes);
   const state: RenderState = {
@@ -913,6 +1020,7 @@ function renderNodes(
     duplicateRefNames: duplicateReferenceNames(nodes),
     surfaceMap: new Map(surfaces.map((surface) => [surface.triggerId, surface])),
     scopeMap: new Map(activeScopeBlocks.map((scope) => [scope.triggerId, scope])),
+    visualSurfaces,
   };
 
   renderTree(children, state);
@@ -1124,8 +1232,20 @@ function renderDoubleLayer(scene: VomScene, layer: BlockingLayer, options: VomOp
     "@layers 2 focus=L1",
     `L1 ${layer.kind} cover=${Math.round(layer.coverage * 100)}%`,
   ];
-  const state = renderNodes(visibleNodes, options, header, scene.surfaces, scene.activeScopeBlocks);
-  state.lines.push(renderPageOcclusionLine(hiddenCount));
+  const visibleSurface = (parentId: number | null): boolean =>
+    parentId === null || included.has(parentId);
+  const state = renderNodes(
+    visibleNodes,
+    options,
+    header,
+    scene.surfaces,
+    scene.activeScopeBlocks,
+    scene.visualSurfaces?.filter((surface) => visibleSurface(surface.parentId)),
+  );
+  const occlusionLine = renderPageOcclusionLine(hiddenCount);
+  state.lines.push(occlusionLine);
+  state.tokens += estimateTokens(occlusionLine);
+  emitVisualAppendix(state);
 
   return {
     text: state.lines.join("\n"),
@@ -1152,7 +1272,9 @@ export function renderVom(scene: VomScene, options: VomOptions = {}): VomResult
     ],
     scene.surfaces,
     scene.activeScopeBlocks,
+    scene.visualSurfaces,
   );
+  emitVisualAppendix(state);
 
   return {
     text: state.lines.join("\n"),

packages/vom/src/types.ts

@@ -61,6 +61,8 @@ export interface VomScene {
   rootFrameId?: string;
   surfaces?: CondSurface[];
   activeScopeBlocks?: ActiveScopeBlock[];
+  /** Non-semantic rendered regions discovered alongside the AX/DOM tree. */
+  visualSurfaces?: VomVisualSurface[];
 }
 
 export interface VomOptions {
@@ -94,6 +96,22 @@ export interface VomRef {
   ref: string;
   backendNodeId: number;
   frameId?: string;
+  /** Observation-time top-viewport crop for screenshot-only surfaces. */
+  visibleRect?: Rect;
+  kind?: "dom" | "surface";
+  capabilities?: VomRefCapability[];
+}
+
+export type VomRefCapability = "interact" | "screenshot";
+
+export interface VomVisualSurface {
+  parentId: number | null;
+  backendNodeId: number;
+  frameId: string;
+  renderingKind: "canvas";
+  visibleRect: Rect;
+  label?: string;
+  memberCount: number;
 }
 
 export interface RenderedRef extends VomRef {

scripts/release.mjs

@@ -1,299 +0,0 @@
-#!/usr/bin/env node
-/**
- * Unified release script for BrowserSkill.
- *
- * Bumps CLI, Extension, and DSH Plugin to the same version, updates
- * CHANGELOG.md, commits, tags, and optionally pushes.
- *
- * Usage:
- *   node scripts/release.mjs <version>                  # bump + commit + tag
- *   node scripts/release.mjs <version> --dry-run        # preview changes only
- *   node scripts/release.mjs <version> --push           # also push to origin
- *   node scripts/release.mjs <version> --push --only ext          # push only ext tag
- *   node scripts/release.mjs <version> --push --only cli,dsh      # push only cli + dsh tags
- *   node scripts/release.mjs --push-tags --only ext     # push tag(s) for an already-committed release
- *
- * Options:
- *   --dry-run      Show what would change without writing anything
- *   --push         After commit+tag, push the branch and all release tags
- *   --push-tags    Push release tags only (skip version bump; useful for staged rollout)
- *   --only <list>  Comma-separated subset of: cli, ext, dsh  (default: all three)
- *   --no-commit    Write files but skip git commit and tagging
- */
-
-import { execSync } from "node:child_process";
-import { readFileSync, writeFileSync } from "node:fs";
-import { parseArgs } from "node:util";
-
-const TAG_PREFIX = { cli: "cli-v", ext: "ext-v", dsh: "dsh-plugin-v" };
-const ALL_COMPONENTS = ["cli", "ext", "dsh"];
-
-const VERSION_FILES = {
-  cargo: "Cargo.toml",
-  extension: "apps/extension/package.json",
-  dshPlugin: "packages/dsh-plugin-browserskill/package.json",
-};
-
-// ---------------------------------------------------------------------------
-// CLI argument parsing
-// ---------------------------------------------------------------------------
-
-const { values, positionals } = parseArgs({
-  allowPositionals: true,
-  options: {
-    "dry-run": { type: "boolean", default: false },
-    push: { type: "boolean", default: false },
-    "push-tags": { type: "boolean", default: false },
-    "no-commit": { type: "boolean", default: false },
-    only: { type: "string", default: "" },
-    help: { type: "boolean", short: "h", default: false },
-  },
-});
-
-if (values.help) {
-  console.log(
-    readFileSync(new URL(import.meta.url), "utf8").match(/\/\*\*([\s\S]*?)\*\//)?.[1] ?? "",
-  );
-  process.exit(0);
-}
-
-const dryRun = values["dry-run"];
-const pushAfter = values.push;
-const pushTagsOnly = values["push-tags"];
-const noCommit = values["no-commit"];
-const onlyRaw = values.only;
-const components = onlyRaw ? onlyRaw.split(",").map((s) => s.trim().toLowerCase()) : ALL_COMPONENTS;
-
-for (const c of components) {
-  if (!ALL_COMPONENTS.includes(c)) {
-    console.error(`Unknown component "${c}". Valid: ${ALL_COMPONENTS.join(", ")}`);
-    process.exit(1);
-  }
-}
-
-// ---------------------------------------------------------------------------
-// Push-tags-only mode (no version bump)
-// ---------------------------------------------------------------------------
-
-if (pushTagsOnly) {
-  const currentVersion = readCurrentVersions();
-  const ver = positionals[0] ?? currentVersion.cargo;
-  for (const c of components) {
-    const tag = `${TAG_PREFIX[c]}${ver}`;
-    console.log(`  pushing tag ${tag}`);
-    if (!dryRun) {
-      run(`git push origin ${tag}`);
-    }
-  }
-  console.log("\nDone.");
-  process.exit(0);
-}
-
-// ---------------------------------------------------------------------------
-// Normal mode — require version argument
-// ---------------------------------------------------------------------------
-
-const newVersion = positionals[0];
-if (!newVersion) {
-  console.error(
-    "Usage: node scripts/release.mjs <version> [--dry-run] [--push] [--only cli,ext,dsh]",
-  );
-  process.exit(1);
-}
-
-if (!/^\d+\.\d+\.\d+(-[\w.]+)?$/.test(newVersion)) {
-  console.error(`Invalid semver: "${newVersion}"`);
-  process.exit(1);
-}
-
-// ---------------------------------------------------------------------------
-// Read current versions
-// ---------------------------------------------------------------------------
-
-function readCurrentVersions() {
-  const cargo = readFileSync(VERSION_FILES.cargo, "utf8");
-  const cargoVer =
-    cargo.match(/^\[workspace\.package\]\s*\nversion\s*=\s*"([^"]+)"/m)?.[1] ??
-    cargo.match(/^version\s*=\s*"([^"]+)"/m)?.[1];
-
-  const extPkg = JSON.parse(readFileSync(VERSION_FILES.extension, "utf8"));
-  const dshPkg = JSON.parse(readFileSync(VERSION_FILES.dshPlugin, "utf8"));
-
-  return { cargo: cargoVer, extension: extPkg.version, dshPlugin: dshPkg.version };
-}
-
-const current = readCurrentVersions();
-
-console.log("Current versions:");
-console.log(`  CLI/Daemon (Cargo.toml):         ${current.cargo}`);
-console.log(`  Extension (package.json):         ${current.extension}`);
-console.log(`  DSH Plugin (package.json):        ${current.dshPlugin}`);
-console.log(`\nBumping all → ${newVersion}`);
-if (dryRun) console.log("  (dry-run mode — no files will be written)\n");
-
-// ---------------------------------------------------------------------------
-// Bump Cargo.toml
-// ---------------------------------------------------------------------------
-
-function bumpCargo(version) {
-  let content = readFileSync(VERSION_FILES.cargo, "utf8");
-
-  // workspace.package version
-  content = content.replace(
-    /^(\[workspace\.package\]\s*\nversion\s*=\s*")([^"]+)(")/m,
-    `$1${version}$3`,
-  );
-
-  // workspace.dependencies.bsk-protocol version (for cargo publish)
-  content = content.replace(
-    /(bsk-protocol\s*=\s*\{\s*version\s*=\s*")([^"]+)(")/,
-    `$1${version}$3`,
-  );
-
-  return content;
-}
-
-function bumpPackageJson(filePath, version) {
-  const pkg = JSON.parse(readFileSync(filePath, "utf8"));
-  pkg.version = version;
-  return JSON.stringify(pkg, null, 2) + "\n";
-}
-
-// ---------------------------------------------------------------------------
-// Update CHANGELOG.md
-// ---------------------------------------------------------------------------
-
-function updateChangelog(version) {
-  const changelogPath = "CHANGELOG.md";
-  let content;
-  try {
-    content = readFileSync(changelogPath, "utf8");
-  } catch {
-    return null;
-  }
-
-  const today = new Date().toISOString().slice(0, 10);
-  const escapedVer = version.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
-
-  // First: fill in any date-placeholder line for this version (e.g. "## [0.2.0] - 2026-09-XX")
-  const versionPlaceholder = new RegExp(`^## \\[${escapedVer}\\]\\s*-\\s*\\d{4}-\\d{2}-XX`, "m");
-  if (versionPlaceholder.test(content)) {
-    content = content.replace(versionPlaceholder, `## [${version}] - ${today}`);
-  } else {
-    // Otherwise: insert a new section after [Unreleased]
-    const unreleased = /^## \[Unreleased\]/m;
-    if (unreleased.test(content)) {
-      content = content.replace(unreleased, `## [Unreleased]\n\n## [${version}] - ${today}`);
-    }
-  }
-
-  return { path: changelogPath, content };
-}
-
-// ---------------------------------------------------------------------------
-// Write files
-// ---------------------------------------------------------------------------
-
-const writes = [
-  { path: VERSION_FILES.cargo, content: bumpCargo(newVersion), label: "Cargo.toml" },
-  {
-    path: VERSION_FILES.extension,
-    content: bumpPackageJson(VERSION_FILES.extension, newVersion),
-    label: "Extension package.json",
-  },
-  {
-    path: VERSION_FILES.dshPlugin,
-    content: bumpPackageJson(VERSION_FILES.dshPlugin, newVersion),
-    label: "DSH Plugin package.json",
-  },
-];
-
-const changelogUpdate = updateChangelog(newVersion);
-if (changelogUpdate) {
-  writes.push({
-    path: changelogUpdate.path,
-    content: changelogUpdate.content,
-    label: "CHANGELOG.md",
-  });
-}
-
-for (const w of writes) {
-  console.log(`  ✓ ${w.label}`);
-  if (!dryRun) {
-    writeFileSync(w.path, w.content);
-  }
-}
-
-// ---------------------------------------------------------------------------
-// Update Cargo.lock (cargo check triggers it)
-// ---------------------------------------------------------------------------
-
-if (!dryRun) {
-  console.log("\n  Updating Cargo.lock...");
-  run("cargo check --workspace 2>&1 || true");
-}
-
-// ---------------------------------------------------------------------------
-// Git commit + tag
-// ---------------------------------------------------------------------------
-
-if (noCommit || dryRun) {
-  if (dryRun) {
-    console.log(`\nWould commit: "chore(release): ${newVersion}"`);
-    console.log("Would create tags:");
-    for (const c of components) {
-      console.log(`  ${TAG_PREFIX[c]}${newVersion}`);
-    }
-  }
-  console.log("\nDone.");
-  process.exit(0);
-}
-
-console.log("\nCommitting...");
-run(
-  "git add Cargo.toml Cargo.lock apps/extension/package.json packages/dsh-plugin-browserskill/package.json CHANGELOG.md 2>/dev/null || true",
-);
-run(`git commit -m "chore(release): ${newVersion}"`);
-
-console.log("Tagging...");
-for (const c of components) {
-  const tag = `${TAG_PREFIX[c]}${newVersion}`;
-  run(`git tag ${tag}`);
-  console.log(`  ✓ ${tag}`);
-}
-
-// ---------------------------------------------------------------------------
-// Push (optional)
-// ---------------------------------------------------------------------------
-
-if (pushAfter) {
-  console.log("\nPushing...");
-  run("git push origin HEAD");
-  for (const c of components) {
-    const tag = `${TAG_PREFIX[c]}${newVersion}`;
-    console.log(`  pushing tag ${tag}`);
-    run(`git push origin ${tag}`);
-  }
-}
-
-console.log(`\nRelease ${newVersion} prepared successfully.`);
-if (!pushAfter) {
-  console.log("\nNext steps:");
-  console.log("  # Push branch");
-  console.log("  git push origin main");
-  console.log("");
-  console.log("  # Push extension tag first (store review takes time)");
-  console.log(`  git push origin ext-v${newVersion}`);
-  console.log("");
-  console.log("  # Then push CLI and DSH plugin tags");
-  console.log(`  git push origin cli-v${newVersion}`);
-  console.log(`  git push origin dsh-plugin-v${newVersion}`);
-}
-
-// ---------------------------------------------------------------------------
-// Helpers
-// ---------------------------------------------------------------------------
-
-function run(cmd) {
-  execSync(cmd, { stdio: "inherit" });
-}

skill/SKILL.md

@@ -73,6 +73,16 @@ CSS-only hover menu looks like from here. It hovers a bounded set of likely trig
 few seconds and touches the live page; once you know which element hides the menu, `bsk hover <ref>`
 is cheaper and more precise.
 
+When `bsk observe` renders
+`@eN surface ... [visual-only; requires=image-understanding; ...]`, the ref identifies rendered
+canvas content that is not represented by the text observation, not an interactable DOM control.
+Keep using any reliable DOM/AX text and controls that appear alongside it. If you can actually
+inspect image output, use `bsk screenshot --ref @eN --session <id>` to obtain the visible crop. If
+you lack multimodal or image-reading capability, do not take a screenshot and pretend to know its
+contents, and never guess coordinates; tell the user that they need to switch to a model with
+image-understanding capability. Do not pass a visual surface ref to click, fill, hover, or select;
+those commands intentionally reject screenshot-only refs.
+
 Escalate page reading only as needed:
 
 1. `bsk observe` for normal semantic understanding, text, controls, and refs.
@@ -116,15 +126,156 @@ navigate   navigate-back   navigate-forward   reload   wait-for-navigation   wai
 observe   snapshot   get-html   screenshot   console   network
 click   hover   fill   select   press   evaluate
 tab list|create|close|select|borrow|return   window resize   emulate
-upload   download   request-help   record start|stop
+request-help   record start|stop
 ```
 
 Required flags that are easy to get wrong:
 
-```text
-bsk fill <ref> --value <text>      bsk select <ref> --value <option-value>
-bsk screenshot --out <path>        bsk emulate --device <preset-id>
-bsk upload <ref> --file <path>     bsk download <ref> --out <path>
+### Tabs (require `--session <id>`)
+
+| Command | Summary |
+|---------|---------|
+| `bsk tab list` | List tabs (`--scope user\|agent\|all`, default `all`) |
+| `bsk tab create` | New tab in Agent Window (`--url`, `--no-active`, `--index`) |
+| `bsk tab close <tab-id>` | Close an agent tab |
+| `bsk tab select <tab-id>` | Focus an agent tab |
+| `bsk tab borrow <tab-id>` | Move a user tab into the Agent Window |
+| `bsk tab return <tab-id>` | Return a borrowed tab to its original window |
+
+### Observation (require `--session` unless noted)
+
+| Command | Summary |
+|---------|---------|
+| `bsk snapshot` | First-choice static page understanding: accessibility tree with `@eN` element refs |
+| `bsk observe` | Semantic VOM observation with bounded perception probes for conditional surfaces |
+| `bsk get-html` | Raw HTML dump after snapshot is insufficient (high token cost) |
+| `bsk screenshot` | PNG capture after snapshot is insufficient: full visible tab, or `--ref @eN` to crop to one element (`--out` path optional) |
+
+### Console & network debugging (read-only; require `--session`)
+
+| Command | Summary |
+|---------|---------|
+| `bsk console` | Buffered page console messages, JS exceptions, and browser log entries (`--include-stack` for stack traces) |
+| `bsk network` | Buffered network responses (status, method, URL, MIME/resource type) and failures (`net::ERR_*` reason) |
+
+Both capture from the moment the tab is attached and read a bounded per-tab buffer: `--since <seq>` pages from a cursor (`next_since` in the result), `--limit` (default 50, max 200), `--max-text-chars` (default 1000, max 4096), `--tab-id` to target a non-active tab. Both are strictly read-only — they never intercept or modify traffic, and request/response headers, bodies, and timings are not captured.
+
+### Navigation
+
+| Command | Summary |
+|---------|---------|
+| `bsk navigate <url>` | Go to URL in agent tab (`--wait-until`, `--timeout`) |
+| `bsk navigate-back` | History back one step |
+| `bsk navigate-forward` | History forward one step |
+| `bsk reload` | Reload current tab (`--hard` bypass cache) |
+
+(`bsk navigate back` / `bsk navigate forward` are equivalent subcommands.)
+
+### Interaction
+
+| Command | Summary |
+|---------|---------|
+| `bsk click <ref-or-selector>` | Click element (`--button`, `--click-count`, `--modifiers`) |
+| `bsk hover <ref-or-selector>` | Move the mouse to an element and wait for hover UI to settle (`--settle`, `--modifiers`) |
+| `bsk fill <ref-or-selector> --value <text>` | Clear and type into input |
+| `bsk select <ref-or-selector> --value <v>` | Set `<select>` option(s) by `value` (repeat `--value` for multi-select) |
+| `bsk press <key>` | Key/combo (`Enter`, `Ctrl+A`, …; optional `--ref` to focus first) |
+
+### File transfer (require `--session`)
+
+| Command | Summary |
+|---------|---------|
+| `bsk upload <ref-or-selector> --file <path>` | Click one upload trigger and attach an agent-readable local file (`--file` is repeatable) |
+| `bsk download <ref-or-selector> --out <path>` | Click one download trigger and copy the single completed file to an exact local path (`--overwrite` is opt-in) |
+
+The agent/harness decides whether a file transfer is appropriate and which local path belongs to the task. Treat upload as disclosure of that file to the current website, and download as accepting website-controlled bytes onto the local filesystem. Use only paths that are necessary for the user's bounded goal.
+
+BrowserSkill enforces the mechanical boundary: files are staged under a session-scoped opaque transfer, only daemon-minted capabilities reach the extension, upload/download still obey Agent Window tab checks, and transfers are chunk/size bounded. Upload intercepts the native chooser for one transaction, locates the input activated in the resolved target's document, and assigns only the staged file paths. Download uniquely correlates one exact-target browser intent with one Chrome download in either event order, routes it through a daemon-minted relative directory, and lets the daemon validate and import it. Upload staging remains available for a later form submission and is removed when the session ends. Downloads cannot overwrite an existing destination unless `--overwrite` is explicit. BrowserSkill does not inspect file content or decide whether its meaning is sensitive.
+
+Do not use `request-help` merely because a native file chooser or browser download is involved; try these commands first. For transfer failures, use the structured error instead of retrying blindly:
+
+- `reason=file_input_not_activated` means the requested click did not activate exactly one `<input type="file">`; the page may use a non-input picker such as `window.showOpenFilePicker()`. Do not retry blindly. If human help is available, call `request-help` and tell the user the exact original local path to choose. The staged daemon path is internal and must not be shown to the user.
+- `reason=file_input_probe_failed` means BrowserSkill could not safely establish the browser-side upload transaction. Do not repeat the same action; use `request-help` when available.
+- `reason=set_file_input_failed` means BrowserSkill found the activated file input but Chrome rejected the staged path or assignment. Check the extension's file-URL access permission; otherwise use `request-help`.
+- `reason=download_capture_failed` means BrowserSkill could not attribute exactly one completed download to the requested target. Do not retry blindly or accept an unrelated browser download; use `request-help` when available.
+- `effect_state=none` means BrowserSkill confirmed that no file-transfer effect was committed. Follow the accompanying reason; a corrected target or explicit human fallback may be attempted.
+- `effect_state=unknown` means the browser may already have attached or created the file. Do not repeat the transfer. Observe the page if that can establish the result; otherwise stop and report the uncertainty.
+- `effect_state=committed` means the browser-side effect occurred even if later completion or cleanup failed. Do not repeat it; continue only after verifying the resulting page/download state.
+
+If `request-help` returns `outcome="disabled"`, do not retry it. Stop gracefully and report the transfer mechanism that requires human intervention.
+
+### Scripting & timing
+
+| Command | Summary |
+|---------|---------|
+| `bsk evaluate <expression>` | Run JS in agent tab (see red lines). JS throw → stderr, **exit 0** (RPC success); use `--json` and check `.ok` to detect JS errors |
+| `bsk wait-for-navigation` | Block until load/DOM idle/etc. (`--wait-until`, `--timeout`) |
+| `bsk wait-ms <duration>` | Sleep (`500ms`, `2s`, `1m`; **no** `--session`) |
+
+### Ask the human for help — `bsk request-help`
+
+When a step needs a human (captcha, login, OTP) or you want the user to
+confirm an important action, pause and ask:
+
+    bsk request-help --session <id> --prompt "Solve the captcha, then click Done only after the site accepts it" \
+      --title "Captcha required" --target @e7 --target "#submit" --timeout 5m
+
+- `--prompt` (required): what the user should do.
+- `--title` (optional): custom title for the overlay panel. When omitted,
+  the extension shows its default localized title.
+- `--target` (repeatable): a snapshot ref (`@e7`) or CSS selector
+  (`#submit`) to scroll to and flash-highlight. **Strongly recommended** —
+  whenever the prompt refers to a concrete element (a button to click, a
+  field to fill, a checkbox to toggle), pass its `@eN` ref / selector so the
+  user is guided straight to the right spot instead of hunting for it. For
+  interaction scenarios, always include the relevant target(s); reserve a
+  prompt with no `--target` for cases where there is genuinely no specific
+  element to point at (e.g. "wait for the page to finish loading").
+- `--timeout` (default `5m`): how long to wait.
+- `--completion-criteria` (optional): JSON success detector. Use it only
+  when there is a concrete post-help success signal, e.g.
+  `{"any":[{"url_contains":"/dashboard"},{"selector_exists":"[data-testid='account-menu']"}],"stable_for_ms":1000}`.
+
+The target tab is brought to the foreground; the page stays interactive
+while the agent control mask is hidden. The call blocks until the user
+explicitly acts, the timeout expires, cancellation arrives, or explicit
+completion criteria match. Page reloads, SPA route changes, and captcha
+refreshes do not return control by themselves. The result `outcome` is one of:
+
+- `continued` — the user finished and clicked Done / return control (treat as confirm).
+- `cancelled` — the user clicked Cancel (treat as reject/abort).
+- `timed_out` — nobody acted within the timeout.
+- `completed` — the explicit `--completion-criteria` matched while the user had control.
+- `navigated` — deprecated legacy outcome. Do not rely on navigation as a completion signal.
+
+`note` carries any text the user typed back. `resolved_targets` reports
+which refs/selectors matched a live element.
+
+`request-help` does not refresh the page model after the user returns
+control. After a `continued` or `completed` result, issue a separate
+observation tool call (usually `bsk snapshot --session <id>`) before using
+new refs or reasoning about the post-help page state.
+#### Disabling request-help (unattended mode)
+Set `BSK_REQUEST_HELP=off` on unattended servers: `bsk request-help` then
+returns immediately with `outcome="disabled"` (no overlay, no waiting,
+exit 0). Any other value keeps it enabled. If you get `disabled`, do not
+retry — complete the task autonomously or stop gracefully.
+
+### Recording — `bsk record`
+
+Capture the user's own actions in the Agent Window to a **trace bundle**, for later LLM-driven automation. New CLI builds request **trace v3** (page observations + action chain); older extensions may still return **trace v2** (actions only), which the CLI exports as a single `trace.json` without `states/`.
+
+```bash
+bsk record start --browser <instance-id-or-label> \
+  [--url https://…] [--purpose "publish a wiki doc"] \
+  [--max-page-tokens 3000] [--redact-values] \
+  [--output trace]
+# `--url` is optional; default https://example.com/ when omitted (must be http(s)).
+# Blocks until the user clicks Finish in the recording panel, then writes:
+#   trace/trace.json    — action chain (+ state index when v3)
+#   trace/states/       — v3 only: one `sN.txt` observe snapshot per settled page state
+
+bsk record stop [--output trace]   # terminal fallback if the browser panel is unavailable
 ```
 
 `select` matches an option's `value` attribute, not its visible label. Device preset ids are
@@ -139,34 +290,6 @@ lowercase and hyphenated, such as `iphone-14`.
 - `record` captures a user's actions for later replay. Read `bsk record start --help` before use,
   and never record banking, SSO, password-manager, or other sensitive pages.
 
-## File transfer
-
-`upload` and `download` stage files through the daemon; the agent never touches browser-internal
-paths. Treat upload as disclosure to the website, download as accepting website-controlled bytes.
-
-Upload has two independent mechanisms — choose explicitly, never rely on automatic fallback:
-
-- **Default (input mode):** for upload buttons, file-input labels, or "upload from computer"
-  actions. The command clicks the target and intercepts the native file chooser.
-- **`--mode drop`:** for reliably identified attachment-receiving areas — an explicit drop zone,
-  chat composer, email editor, or form attachment area. Do not target page whitespace, generic
-  containers, or areas whose attachment ownership is ambiguous.
-
-Decision sequence when uploading:
-
-1. Try input mode (the default).
-2. If it returns `reason=file_input_not_activated` with `effect_state=none`, re-observe. When a
-   reliable attachment target exists, try `--mode drop` once against that target.
-3. Otherwise fall back to `request-help`.
-4. **Never** switch mechanisms or repeat when `effect_state` is `unknown` or `committed` — the
-   browser may already have applied the file.
-
-A successful drop means Chrome dispatched the native file-drop event; it does not prove the site
-accepted the attachment. Observe the page once after the command.
-
-Download default-refuses to overwrite; pass `--overwrite` when replacing an existing file is
-intended. Read `bsk upload --help` and `bsk download --help` for all flags and error details.
-
 ## Recover without wandering
 
 - Stale ref: observe again and retry the intended action once.