Compare commits

..
Author SHA1 Message Date
drakezhang a62a28d8cd fix(geometry): preserve OOPIF scale with occupied scrollbars 2026-09-09 16:58:25 +08:00
Hannah Liu 896c7944d8 Merge pull request #203 from Tencent/fix/capture-document-consistency
fix(observation): validate DOM identity across capture phases | 校验跨采集阶段的 DOM 身份一致性
2026-09-09 16:26:24 +08:00
Hannah Liu 7e5f3bae26 Merge pull request #200 from Tencent/fix/observation-document-consistency
refactor(observation):  unify cross-frame capture coordination and data ownership | 统一跨 frame 采集协调与数据归属
2026-09-09 16:25:52 +08:00
Hannah Liu af4c2fa229 Merge pull request #198 from Tencent/refactor/document-capture-facts
refactor(observation): decouple DOM processing from capture orchestration | 解耦 DOM 处理与采集流程
2026-09-09 16:24:42 +08:00
Hannah Liu ee10da6fe9 Merge pull request #195 from Tencent/fix/frame-geometry-contracts
fix(geometry): unify cross-frame coordinates and screenshot conversion | 统一跨 frame 坐标与截图转换
2026-09-09 16:24:06 +08:00
drakezhang 4834d8b5fa test(geometry): identify the regression browser attachment 2026-09-09 16:11:39 +08:00
Ljy-0827 b4006d5b85 perf(observation): validate document identity against snapshot roots 2026-09-09 16:09:09 +08:00
Ljy-0827 2d1e5c6d9d fix(observation): validate document consistency with bounded identity checks 2026-09-09 16:09:09 +08:00
drakezhang 282e6f5623 fix(observation): carry snapshot units through capture coordination 2026-09-09 16:05:42 +08:00
Ljy-0827 bd68c293b6 perf(observation): collect frame AX trees with bounded concurrency 2026-09-09 15:58:56 +08:00
Ljy-0827 35a20582ad refactor(observation): separate capture coordination from document consistency 2026-09-09 15:58:56 +08:00
Ljy-0827 4e3365450a refactor(observation): integrate capture orchestration and document consistency 2026-09-09 15:58:56 +08:00
Ljy-0827 df5ed1a495 fix(observation): validate document consistency during capture 2026-09-09 15:58:40 +08:00
drakezhang eefbe03258 fix(observation): preserve snapshot unit normalization after document extraction 2026-09-09 15:58:28 +08:00
Ljy-0827 1d275f047c perf(observation): reduce document processing scheduling overhead 2026-09-09 15:54:41 +08:00
Ljy-0827 388b3ae2d5 refactor(observation): isolate document processing from capture orchestration 2026-09-09 15:54:41 +08:00
Ljy-0827 36d0d91419 fix(observation): preserve snapshot document provenance and URL metadata 2026-09-09 15:54:41 +08:00
Ljy-0827 bcb9beb4f8 refactor(observation): defer visual facts until visual discovery 2026-09-09 15:54:41 +08:00
Ljy-0827 34ccb96fd6 fix(observation): await capture task cleanup before propagating cancellation 2026-09-09 15:54:41 +08:00
Ljy-0827 8dfa097e55 refactor(observation): organize capture around document facts 2026-09-09 15:54:41 +08:00
drakezhang 383e3128eb fix(geometry): normalize snapshot layout bounds and scroll together 2026-09-09 15:53:35 +08:00
Ljy-0827 2d2636030d perf(geometry): batch iframe owner size measurements 2026-09-09 13:20:21 +08:00
Ljy-0827 a28b9d55e7 fix(vom): surface incomplete iframe geometry in observations 2026-09-09 13:20:21 +08:00
Ljy-0827 c39874591f perf(observation): measure sibling iframe geometry with bounded concurrency 2026-09-09 13:20:21 +08:00
Ljy-0827 1485addfd5 fix(geometry): unify frame projection and screenshot coordinates 2026-09-09 13:20:21 +08:00
Zhang GH 90b0ff301b Merge pull request #204 from Tencent/release/0.2.1
Release/0.2.1
2026-09-09 11:35:05 +08:00
drakezhang e74f354e90 fix(release): support changelogs without unreleased section 2026-09-08 20:38:17 +08:00
drakezhang 05c036518a chore(release): 0.2.1 2026-09-08 20:35:14 +08:00
Zhang GH 020fb3212e Merge pull request #201 from Tencent/fix/windows-install-and-cancellation
fix(windows): harden installation and parent cancellation
2026-09-08 20:15:12 +08:00
Zhang GH 0822c53c65 fix(windows): preserve transfer cleanup after cancellation 2026-09-08 20:04:24 +08:00
Zhang GH 4450fbe28c Merge pull request #196 from Tencent/codex/dsh-plugin-docs
docs(dsh-plugin): clarify installation, configuration, and releases
2026-09-08 19:43:24 +08:00
drakezhang a3bb81bfab docs(dsh-plugin): clarify patch editing and reload behavior 2026-09-08 19:34:25 +08:00
Zhang GH caea98cd52 fix(windows): harden installation and parent cancellation 2026-09-08 19:21:54 +08:00
drakezhang 187b088013 docs(dsh-plugin): restore configuration guidance and simplify releases 2026-09-08 19:03:40 +08:00
drakezhang 5046b5ab44 docs(dsh-plugin): clarify setup and verify npm publishing 2026-09-08 18:44:20 +08:00
56 changed files with 4620 additions and 1090 deletions

No files matched your search

+11 -3
View File
@@ -38,7 +38,7 @@ jobs:
run: cargo test --workspace --locked run: cargo test --workspace --locked
windows-daemon: windows-daemon:
name: Windows daemon and update regression tests name: Windows daemon, installer and cancellation regression tests
runs-on: windows-latest runs-on: windows-latest
steps: steps:
@@ -55,8 +55,16 @@ jobs:
- name: Run Windows update tests - name: Run Windows update tests
run: cargo test -p bsk --lib --locked cli::update::tests run: cargo test -p bsk --lib --locked cli::update::tests
- name: Run Windows update process tests - name: Run Windows update and cancellation process tests
run: cargo test -p bsk --test windows_update --locked run: cargo test -p bsk --test windows_update --test windows_parent_cancel --locked
- name: Run installer tests (Windows PowerShell 5.1)
shell: powershell
run: ./scripts/install-windows.test.ps1
- name: Run installer tests (PowerShell 7)
shell: pwsh
run: ./scripts/install-windows.test.ps1
frontend: frontend:
name: Frontend lint, typecheck, tests, build name: Frontend lint, typecheck, tests, build
+18 -1
View File
@@ -5,7 +5,24 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/).
Starting from 0.2.0, CLI / Extension / DSH Plugin share the same version number. Starting from 0.2.0, CLI / Extension / DSH Plugin share the same version number.
## [Unreleased] ## [0.2.1] - 2026-09-09
### Changed
- DSH Plugin: clarified installation, configuration, profile restart, and upgrade instructions
### Fixed
- CLI: reliable Windows self-updates, including executable replacement and daemon restart
- CLI: Windows named-pipe connection continuity and stale daemon process detection
- Windows installer: PowerShell 5.1 compatibility, path quoting, and staged executable replacement
- CLI / DSH Plugin: cancellation propagation and upload/download cleanup when the parent process exits
- Extension: tab ownership and session cleanup when agent tabs fail to open or borrowed tabs are returned or closed
- Extension: navigation recovery when other extensions restrict Chrome DevTools Protocol access
- Extension: background input filling, automatic append behavior, and verification of field values before reporting success
- Extension: match observed form state to the correct element after page changes
- Extension: UUID generation fallback for recording and overlays on insecure origins
- Browser locale matching for regional language variants
- DSH Plugin: observation state synchronization, thumbnail capture lifecycle, and stale preview cleanup
- DSH Plugin: removed the obsolete client runtime host dependency
## [0.2.0] - 2026-09-02 ## [0.2.0] - 2026-09-02
Generated
+2 -2
View File
@@ -134,7 +134,7 @@ dependencies = [
[[package]] [[package]]
name = "bsk" name = "bsk"
version = "0.2.0" version = "0.2.1"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"base64", "base64",
@@ -169,7 +169,7 @@ dependencies = [
[[package]] [[package]]
name = "bsk-protocol" name = "bsk-protocol"
version = "0.2.0" version = "0.2.1"
dependencies = [ dependencies = [
"schemars", "schemars",
"semver", "semver",
+2 -2
View File
@@ -7,7 +7,7 @@ members = [
default-members = ["crates/bsk-cli"] default-members = ["crates/bsk-cli"]
[workspace.package] [workspace.package]
version = "0.2.0" version = "0.2.1"
edition = "2024" edition = "2024"
rust-version = "1.85" rust-version = "1.85"
license = "MIT" license = "MIT"
@@ -43,7 +43,7 @@ rand = "0.8"
uuid = { version = "1", features = ["v4", "serde"] } uuid = { version = "1", features = ["v4", "serde"] }
# Workspace crates (version required for `cargo publish` of dependents) # Workspace crates (version required for `cargo publish` of dependents)
bsk-protocol = { version = "0.2.0", path = "crates/bsk-protocol" } bsk-protocol = { version = "0.2.1", path = "crates/bsk-protocol" }
[profile.release] [profile.release]
strip = "symbols" strip = "symbols"
+13 -7
View File
@@ -147,20 +147,26 @@ Start a new Agent session and write a prompt that needs the browser, for example
Using [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (`dsh`)? Using [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (`dsh`)?
BrowserSkill ships a first-class dsh plugin on npm as BrowserSkill ships a first-class dsh plugin on npm as
[`@wxg-prc-cpg/browser-skill-dsh-plugin`](https://www.npmjs.com/package/@wxg-prc-cpg/browser-skill-dsh-plugin). [`@wxg-prc-cpg/browser-skill-dsh-plugin`](https://www.npmjs.com/package/@wxg-prc-cpg/browser-skill-dsh-plugin).
It injects native `browser_*` tools (no shelling out to `bsk`) and a live Web UI It gives the agent native `browser_*` tools and a live view of its browser sessions
overlay of each Agent Window. in the Web UI. The plugin runs `bsk` on the agent's behalf.
Add it to a dsh profile, then start that profile: Install the `bsk` CLI and connect the browser extension first. Then add the plugin
to a dsh profile and start it (replace `web` with your profile name):
```sh ```sh
dsh plugin --profile web add @wxg-prc-cpg/browser-skill-dsh-plugin dsh plugin --profile web add @wxg-prc-cpg/browser-skill-dsh-plugin
dsh --profile web dsh --profile web
``` ```
The plugin carries its own copy of the skill, so `bsk install-skill` is not needed The plugin includes the `browser-skill` skill, so `bsk install-skill` is not needed
for dsh — but the `bsk` CLI and the browser extension are still prerequisites. See for dsh. Installed plugins do not update automatically. To upgrade this plugin:
the [plugin README](packages/dsh-plugin-browserskill/README.md) for the tool list,
configuration, and the observation overlay. ```sh
dsh plugin --profile web update @wxg-prc-cpg/browser-skill-dsh-plugin --latest
```
Restart the profile after upgrading. See the
[plugin README](packages/dsh-plugin-browserskill/README.md) for usage and configuration.
## How It Works ## How It Works
+9 -3
View File
@@ -125,16 +125,22 @@ bsk install-skill
## DeepSeek Harness 插件 ## DeepSeek Harness 插件
在用 [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness)(`dsh`)?BrowserSkill 提供了官方 dsh 插件,已发布到 npm:[`@wxg-prc-cpg/browser-skill-dsh-plugin`](https://www.npmjs.com/package/@wxg-prc-cpg/browser-skill-dsh-plugin)。它会注入原生 `browser_*` 工具(无需再通过 Shell 调用 `bsk`),并在 Web UI 中实时观察每个 Agent Window。 在用 [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness)(`dsh`)?BrowserSkill 提供了官方 dsh 插件,已发布到 npm:[`@wxg-prc-cpg/browser-skill-dsh-plugin`](https://www.npmjs.com/package/@wxg-prc-cpg/browser-skill-dsh-plugin)。它为 Agent 提供原生 `browser_*` 工具,由插件代为调用 `bsk`,并在 Web UI 中实时展示浏览器会话。
把它装进某个 dsh profile,然后启动该 profile: 先安装 `bsk` CLI 并连接浏览器扩展,再将插件装进 dsh profile 并启动(将 `web` 替换为你的 profile 名称):
```sh ```sh
dsh plugin --profile web add @wxg-prc-cpg/browser-skill-dsh-plugin dsh plugin --profile web add @wxg-prc-cpg/browser-skill-dsh-plugin
dsh --profile web dsh --profile web
``` ```
插件自带 skill,所以在 dsh 下无需执行 `bsk install-skill`;但 `bsk` CLI 和浏览器扩展仍是前置条件。工具清单、配置项与观察浮层见[插件 README](packages/dsh-plugin-browserskill/README.md)。 插件自带 `browser-skill` skill,所以在 dsh 下无需执行 `bsk install-skill`。已安装的插件不会自动更新;升级此插件请运行:
```sh
dsh plugin --profile web update @wxg-prc-cpg/browser-skill-dsh-plugin --latest
```
升级后重启该 profile。用法与配置见[插件 README](packages/dsh-plugin-browserskill/README.md)。
## 工作原理 ## 工作原理
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@browser-skill/extension", "name": "@browser-skill/extension",
"version": "0.2.0", "version": "0.2.1",
"private": true, "private": true,
"type": "module", "type": "module",
"scripts": { "scripts": {
@@ -2,24 +2,6 @@ import { describe, expect, it } from "vitest";
import { buildFrameGraph, type CdpFrameTreeNode, type CdpFrameTreeSource } from "../frame-graph"; import { buildFrameGraph, type CdpFrameTreeNode, type CdpFrameTreeSource } from "../frame-graph";
describe("buildFrameGraph", () => { describe("buildFrameGraph", () => {
it("preserves the owning target's loader identity", () => {
const graph = buildFrameGraph([
{
target: { tabId: 4 },
tree: {
frame: { id: "root", loaderId: "root-loader" },
childFrames: [{ frame: { id: "child", loaderId: "stale-placeholder" } }],
},
},
{
target: { tabId: 4, sessionId: "child-session" },
tree: { frame: { id: "child", loaderId: "child-loader" } },
},
]);
expect(graph?.frames.find((frame) => frame.frameId === "root")?.loaderId).toBe("root-loader");
expect(graph?.frames.find((frame) => frame.frameId === "child")?.loaderId).toBe("child-loader");
});
it("keeps sibling frames distinct and routes nested OOPIFs to their child sessions", () => { it("keeps sibling frames distinct and routes nested OOPIFs to their child sessions", () => {
const graph = buildFrameGraph([ const graph = buildFrameGraph([
{ {
@@ -8,7 +8,6 @@ export interface CdpFrame {
parentFrameId?: string; parentFrameId?: string;
ownerBackendNodeId?: number; ownerBackendNodeId?: number;
url?: string; url?: string;
loaderId?: string;
target: CdpTarget; target: CdpTarget;
} }
@@ -22,7 +21,6 @@ export interface CdpFrameTreeNode {
id: string; id: string;
parentId?: string; parentId?: string;
url?: string; url?: string;
loaderId?: string;
}; };
childFrames?: CdpFrameTreeNode[]; childFrames?: CdpFrameTreeNode[];
} }
@@ -50,7 +48,6 @@ function mergeFrame(
frameId: node.frame.id, frameId: node.frame.id,
...(parentFrameId ? { parentFrameId } : {}), ...(parentFrameId ? { parentFrameId } : {}),
...(url ? { url } : {}), ...(url ? { url } : {}),
...(node.frame.loaderId ? { loaderId: node.frame.loaderId } : {}),
target, target,
}); });
} }
@@ -436,6 +436,8 @@ describe("file transfer tools", () => {
if (method === "DOM.getContentQuads") { if (method === "DOM.getContentQuads") {
return { quads: [[10, 20, 210, 20, 210, 120, 10, 120]] }; return { quads: [[10, 20, 210, 20, 210, 120, 10, 120]] };
} }
if (method === "Runtime.evaluate")
return { result: { value: { width: 400, height: 300 } } };
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 400, clientHeight: 300 } }; return { cssLayoutViewport: { clientWidth: 400, clientHeight: 300 } };
} }
@@ -13,7 +13,138 @@ import {
import { GeometryContext } from "../geometry/frame-context"; import { GeometryContext } from "../geometry/frame-context";
import type { CdpRunner } from "../shared"; import type { CdpRunner } from "../shared";
function scrollbarDriver(
visible = { width: 185, height: 89 },
frameSize: unknown = { width: 200, height: 100 },
) {
const send = vi.fn(async (target: { sessionId?: string }, method: string) => {
if (method === "Runtime.evaluate") return { result: { value: frameSize } };
if (method === "Page.getLayoutMetrics")
return {
cssLayoutViewport: {
clientWidth: target.sessionId ? visible.width : 800,
clientHeight: target.sessionId ? visible.height : 600,
},
};
if (method === "DOM.getBoxModel")
return {
model: {
content: target.sessionId
? [0, 0, 200, 0, 200, 100, 0, 100]
: [100, 100, 500, 100, 500, 300, 100, 300],
},
};
if (method === "DOM.getContentQuads")
return { quads: [[180, 80, 230, 80, 230, 120, 180, 120]] };
throw new Error(`unexpected ${method}`);
});
const cdp: CdpRunner = {
send: (tabId, method) => send({ tabId } as never, method) as never,
sendToTarget: send as CdpRunner["sendToTarget"],
getFrameGraph: async () => ({
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "child",
parentFrameId: "main",
ownerBackendNodeId: 10,
target: { tabId: 4, sessionId: "child" },
},
{
frameId: "same-child",
parentFrameId: "child",
ownerBackendNodeId: 20,
target: { tabId: 4, sessionId: "child" },
},
],
}),
};
return { cdp, send };
}
describe("frame geometry projection", () => { describe("frame geometry projection", () => {
it.each([
{ width: 200, height: 100 },
{ width: 185, height: 100 },
{ width: 200, height: 89 },
{ width: 185, height: 89 },
])("preserves scale and clips scrollbar strips for visible viewport $width × $height", async (visible) => {
const { cdp, send } = scrollbarDriver(visible);
const geometry = await resolveNodeGeometry(cdp, 4, {
target: { tabId: 4, sessionId: "child" },
frameId: "child",
backendNodeId: 101,
});
expect(geometry).toMatchObject({
topBounds: {
x: 460,
y: 260,
width: (visible.width - 180) * 2,
height: (visible.height - 80) * 2,
},
actionPoint: { x: 460 + visible.width - 180, y: 260 + visible.height - 80 },
targetActionPoint: { x: (180 + visible.width) / 2, y: (80 + visible.height) / 2 },
});
expect(send.mock.calls.filter(([, method]) => method === "Runtime.evaluate")).toHaveLength(1);
expect(send.mock.calls.filter(([, method]) => method === "Page.getLayoutMetrics")).toHaveLength(
2,
);
});
it("shares the full viewport read across same-target frames, and refreshes it next operation", async () => {
const { cdp, send } = scrollbarDriver();
const context = new GeometryContext(cdp, 4);
await Promise.all([context.targetProjection("child"), context.targetProjection("same-child")]);
expect(send.mock.calls.filter(([, method]) => method === "Runtime.evaluate")).toHaveLength(1);
await new GeometryContext(cdp, 4).targetProjection("child");
expect(send.mock.calls.filter(([, method]) => method === "Runtime.evaluate")).toHaveLength(2);
});
it.each([
null,
{},
{ width: 0, height: 100 },
{ width: 200, height: NaN },
])("rejects unavailable full viewport dimensions: %j", async (size) => {
const { cdp } = scrollbarDriver(undefined, size);
expect(await new GeometryContext(cdp, 4).targetProjection("child")).toBeNull();
});
it("clips an inner OOPIF against an intermediate OOPIF's occupied scrollbar strips", async () => {
const { cdp } = scrollbarDriver();
const outer = cdp.sendToTarget!;
cdp.sendToTarget = async (target, method, params) => {
if (method === "DOM.getBoxModel")
return { model: { content: [150, 60, 250, 60, 250, 120, 150, 120] } } as never;
if (target.sessionId !== "inner") return outer(target, method, params);
if (method === "Runtime.evaluate")
return { result: { value: { width: 100, height: 60 } } } as never;
if (method === "Page.getLayoutMetrics")
return { cssLayoutViewport: { clientWidth: 90, clientHeight: 50 } } as never;
if (method === "DOM.getContentQuads")
return { quads: [[0, 0, 100, 0, 100, 60, 0, 60]] } as never;
throw new Error(`unexpected ${method}`);
};
const graph = await cdp.getFrameGraph!(4);
graph.frames[2] = {
frameId: "inner",
parentFrameId: "child",
ownerBackendNodeId: 20,
target: { tabId: 4, sessionId: "inner" },
};
cdp.getFrameGraph = async () => graph;
expect(
await resolveNodeGeometry(cdp, 4, {
target: { tabId: 4, sessionId: "inner" },
frameId: "inner",
backendNodeId: 101,
}),
).toMatchObject({
topBounds: { x: 400, y: 220, width: 70, height: 58 },
});
});
it("keeps region bounds separate from polygon area", () => { it("keeps region bounds separate from polygon area", () => {
const region = [ const region = [
rectPolygon({ x: 0, y: 0, w: 10, h: 10 }), rectPolygon({ x: 0, y: 0, w: 10, h: 10 }),
@@ -77,6 +208,8 @@ describe("frame geometry projection", () => {
const cdp: CdpRunner = { const cdp: CdpRunner = {
send: send as CdpRunner["send"], send: send as CdpRunner["send"],
sendToTarget: vi.fn(async (_target, method) => { sendToTarget: vi.fn(async (_target, method) => {
if (method === "Runtime.evaluate")
return { result: { value: { width: 200, height: 100 } } };
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } }; return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } };
} }
@@ -135,6 +268,8 @@ describe("frame geometry projection", () => {
throw new Error(`unexpected root command ${method}`); throw new Error(`unexpected root command ${method}`);
}) as CdpRunner["send"], }) as CdpRunner["send"],
sendToTarget: vi.fn(async (_target, method) => { sendToTarget: vi.fn(async (_target, method) => {
if (method === "Runtime.evaluate")
return { result: { value: { width: 200, height: 100 } } };
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } }; return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } };
} }
@@ -196,6 +331,8 @@ describe("frame geometry projection", () => {
}) as CdpRunner["send"], }) as CdpRunner["send"],
sendToTarget: vi.fn(async (_target, method) => { sendToTarget: vi.fn(async (_target, method) => {
if (method === "DOM.scrollIntoViewIfNeeded") return {}; if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "Runtime.evaluate")
return { result: { value: { width: 200, height: 100 } } };
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } }; return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } };
} }
@@ -5,9 +5,10 @@ import { rectPolygon } from "../geometry";
import { import {
projectSnapshotRect, projectSnapshotRect,
screenshotPageRect, screenshotPageRect,
snapshotCoordinates,
snapshotViewportRect, snapshotViewportRect,
} from "../geometry/coordinate-types"; } from "../geometry/coordinate-types";
import { GeometryContext } from "../geometry/frame-context"; import { GeometryContext, snapshotLayoutScale } from "../geometry/frame-context";
import type { CdpRunner } from "../shared"; import type { CdpRunner } from "../shared";
const target = { tabId: 4 }; const target = { tabId: 4 };
@@ -22,7 +23,11 @@ const graph: CdpFrameGraph = {
function driver() { function driver() {
const send = vi.fn(async (_tab: number, method: string): Promise<object> => { const send = vi.fn(async (_tab: number, method: string): Promise<object> => {
if (method === "Page.getLayoutMetrics") if (method === "Page.getLayoutMetrics")
return { cssLayoutViewport: { clientWidth: 1000, clientHeight: 1000 } }; return {
visualViewport: { clientWidth: 1000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 1000, clientHeight: 1000 },
};
if (method === "DOM.getBoxModel") if (method === "DOM.getBoxModel")
return { model: { content: [52.5, 612.5, 427.5, 612.5, 427.5, 862.5, 52.5, 862.5] } }; return { model: { content: [52.5, 612.5, 427.5, 612.5, 427.5, 862.5, 52.5, 862.5] } };
if (method === "DOM.resolveNode") return { object: { objectId: "owner" } }; if (method === "DOM.resolveNode") return { object: { objectId: "owner" } };
@@ -35,6 +40,63 @@ function driver() {
} }
describe("measurement geometry context", () => { describe("measurement geometry context", () => {
it.each([0.8, 1, 1.25, 2])("normalizes snapshot layout units at scale %s", (scale) => {
const measured = snapshotLayoutScale({
visualViewport: { clientWidth: 997.25, zoom: 1.1 },
cssVisualViewport: { clientWidth: 997.25 / scale, zoom: 1.1 },
// Rounded layout dimensions must not supply the scale.
layoutViewport: { clientWidth: 997 },
cssLayoutViewport: { clientWidth: Math.floor(997.25 / scale) },
});
expect(measured).toBeCloseTo(scale, 12);
const coordinates = snapshotCoordinates(
{ scrollOffsetX: 80 * scale, scrollOffsetY: 200 * scale },
measured,
{ x: 999, y: 999 },
);
expect(
snapshotViewportRect(
[160 * scale, 320 * scale, 120 * scale, 40 * scale],
{ target },
coordinates,
)?.rect,
).toEqual({ x: 80, y: 120, width: 120, height: 40 });
});
it("uses CSS root scroll only for missing snapshot offsets, without scaling it twice", () => {
const coordinates = snapshotCoordinates({ scrollOffsetX: 80 }, 2, { x: 999, y: 100 });
expect(snapshotViewportRect([200, 800, 240, 80], { target }, coordinates)?.rect).toEqual({
x: 60,
y: 300,
width: 120,
height: 40,
});
expect(snapshotCoordinates({ scrollOffsetX: 0 }, 2)).toBeNull();
expect(
snapshotCoordinates({ scrollOffsetX: NaN, scrollOffsetY: 0 }, 2, { x: 0, y: 0 }),
).toBeNull();
});
it("does not infer snapshot units from missing, invalid, or CSS-only metrics", () => {
expect(snapshotLayoutScale({ cssLayoutViewport: { clientWidth: 800 } })).toBeNull();
for (const raw of [0, -1, NaN, Infinity]) {
expect(
snapshotLayoutScale({
visualViewport: { clientWidth: raw },
cssVisualViewport: { clientWidth: 800 },
}),
).toBeNull();
}
expect(
snapshotLayoutScale({
visualViewport: { clientWidth: 0, clientHeight: 600 },
cssVisualViewport: { clientWidth: 0, clientHeight: 300 },
}),
).toBe(2);
expect(snapshotCoordinates({ scrollOffsetX: 0, scrollOffsetY: 0 }, null)).toBeNull();
expect(snapshotViewportRect([10, 20, 120, 40], { target }, null)).toBeNull();
});
it("shares in-flight frame measurements, but never shares them with the next context", async () => { it("shares in-flight frame measurements, but never shares them with the next context", async () => {
const cdp = driver(); const cdp = driver();
const context = new GeometryContext(cdp, 4); const context = new GeometryContext(cdp, 4);
@@ -60,15 +122,22 @@ describe("measurement geometry context", () => {
width: 1000, width: 1000,
height: 1000, height: 1000,
}); });
const input = snapshotViewportRect([17, 23, 120, 40], source, { x: 0, y: 0 }); if (projection.status !== "available") throw new Error("expected available projection");
expect(projectSnapshotRect(input!, projection!)).toEqual({ const input = snapshotViewportRect([17, 23, 120, 40], source, {
layoutUnitsPerCssPixel: 1,
scrollCss: { x: 0, y: 0 },
});
expect(projectSnapshotRect(input!, projection.projection)).toEqual({
x: 73.75, x: 73.75,
y: 641.25, y: 641.25,
width: 150, width: 150,
height: 50, height: 50,
}); });
const scrolled = snapshotViewportRect([17, 23, 120, 40], source, { x: 0, y: 10 }); const scrolled = snapshotViewportRect([17, 23, 120, 40], source, {
expect(projectSnapshotRect(scrolled!, projection!)?.y).toBe(628.75); layoutUnitsPerCssPixel: 1,
scrollCss: { x: 0, y: 10 },
});
expect(projectSnapshotRect(scrolled!, projection.projection)?.y).toBe(628.75);
await context.snapshotProjection(source, 10, [], { width: 1000, height: 1000 }); await context.snapshotProjection(source, 10, [], { width: 1000, height: 1000 });
expect(cdp.calls.mock.calls.filter(([, method]) => method === "DOM.resolveNode")).toHaveLength( expect(cdp.calls.mock.calls.filter(([, method]) => method === "DOM.resolveNode")).toHaveLength(
1, 1,
@@ -78,11 +147,101 @@ describe("measurement geometry context", () => {
).toHaveLength(1); ).toHaveLength(1);
}); });
it("falls back only for missing or invalid batch entries and does not reuse sizes across contexts", async () => {
const cdp = driver();
const original = cdp.send;
const calls: string[] = [];
cdp.send = async (tabId, method, params) => {
calls.push(method);
if (method === "Runtime.evaluate")
return {
result: {
deepSerializedValue: {
type: "array",
value: [
{
type: "array",
value: [
{ type: "node", value: { backendNodeId: 10 } },
{ type: "string", value: '{"width":300,"height":200}' },
],
},
{
type: "array",
value: [
{ type: "node", value: { backendNodeId: 11 } },
{ type: "string", value: '{"width":0,"height":200}' },
],
},
],
},
},
} as never;
if (method === "Runtime.releaseObjectGroup") return {} as never;
return original(tabId, method, params);
};
const context = new GeometryContext(cdp, 4);
context.registerSnapshotOwners(target, new Set([10, 11]));
const values = await Promise.all(
[10, 11].map((id) =>
context.snapshotProjection({ target, frameId: String(id) }, id, [], {
width: 1000,
height: 1000,
}),
),
);
expect(values.every((value) => value.status === "available")).toBe(true);
expect(calls.filter((method) => method === "DOM.resolveNode")).toHaveLength(1);
expect(calls.filter((method) => method === "Runtime.evaluate")).toHaveLength(1);
await new GeometryContext(cdp, 4).snapshotProjection({ target, frameId: "child" }, 10, [], {
width: 1000,
height: 1000,
});
expect(calls.filter((method) => method === "DOM.resolveNode")).toHaveLength(2);
});
it("waits for batch object cleanup and propagates cancellation", async () => {
const cdp = driver();
const original = cdp.send;
const controller = new AbortController();
let release!: () => void;
let cleanup!: () => void;
const started = new Promise<void>((resolve) => {
cleanup = resolve;
});
cdp.send = async (tabId, method, params) => {
if (method === "Runtime.evaluate") return {} as never;
if (method === "Runtime.releaseObjectGroup") {
cleanup();
await new Promise<void>((resolve) => {
release = resolve;
});
return {} as never;
}
return original(tabId, method, params);
};
const context = new GeometryContext(cdp, 4, undefined, controller.signal);
context.registerSnapshotOwners(target, new Set([10, 11]));
let settled = false;
const operation = context
.snapshotProjection({ target, frameId: "child" }, 10, [], { width: 1000, height: 1000 })
.finally(() => {
settled = true;
});
const rejected = expect(operation).rejects.toMatchObject({ name: "AbortError" });
await started;
controller.abort();
expect(settled).toBe(false);
release();
await rejected;
expect(cdp.calls.mock.calls.some(([, method]) => method === "DOM.resolveNode")).toBe(false);
});
it("rejects owner mismatches and invalid numeric bounds", () => { it("rejects owner mismatches and invalid numeric bounds", () => {
const input = snapshotViewportRect( const input = snapshotViewportRect(
[30, 50, 120, 40], [30, 50, 120, 40],
{ target, frameId: "main" }, { target, frameId: "main" },
{ x: 10, y: 20 }, { layoutUnitsPerCssPixel: 1, scrollCss: { x: 10, y: 20 } },
)!; )!;
const projection = { const projection = {
source: { target, frameId: "main" }, source: { target, frameId: "main" },
@@ -103,8 +262,20 @@ describe("measurement geometry context", () => {
source: { target: { tabId: 5 }, frameId: "main" }, source: { target: { tabId: 5 }, frameId: "main" },
}), }),
).toBeNull(); ).toBeNull();
expect(snapshotViewportRect([0, 0, NaN, 40], { target }, { x: 0, y: 0 })).toBeNull(); expect(
expect(snapshotViewportRect([0, 0, 10, 40], { target }, { x: Infinity, y: 0 })).toBeNull(); snapshotViewportRect(
[0, 0, NaN, 40],
{ target },
{ layoutUnitsPerCssPixel: 1, scrollCss: { x: 0, y: 0 } },
),
).toBeNull();
expect(
snapshotViewportRect(
[0, 0, 10, 40],
{ target },
{ layoutUnitsPerCssPixel: 1, scrollCss: { x: Infinity, y: 0 } },
),
).toBeNull();
}); });
it("does not turn a border quad or a failed owner read into a content projection", async () => { it("does not turn a border quad or a failed owner read into a content projection", async () => {
@@ -118,7 +289,11 @@ describe("measurement geometry context", () => {
width: 100, width: 100,
height: 100, height: 100,
}), }),
).toBeNull(); ).toEqual({
status: "unavailable",
source: { target, frameId: "child" },
ownerBackendNodeId: 10,
});
}); });
it("keeps ancestor clips in target coordinates instead of applying nested offsets twice", async () => { it("keeps ancestor clips in target coordinates instead of applying nested offsets twice", async () => {
@@ -129,12 +304,13 @@ describe("measurement geometry context", () => {
[rectPolygon({ x: 60, y: 620, w: 100, h: 100 })], [rectPolygon({ x: 60, y: 620, w: 100, h: 100 })],
{ width: 1000, height: 1000 }, { width: 1000, height: 1000 },
); );
if (projection.status !== "available") throw new Error("expected available projection");
const input = snapshotViewportRect( const input = snapshotViewportRect(
[0, 0, 300, 200], [0, 0, 300, 200],
{ target, frameId: "nested" }, { target, frameId: "nested" },
{ x: 0, y: 0 }, { layoutUnitsPerCssPixel: 1, scrollCss: { x: 0, y: 0 } },
); );
expect(projectSnapshotRect(input!, projection!)).toEqual({ expect(projectSnapshotRect(input!, projection.projection)).toEqual({
x: 60, x: 60,
y: 620, y: 620,
width: 100, width: 100,
@@ -182,7 +358,11 @@ describe("measurement geometry context", () => {
peak = Math.max(peak, active); peak = Math.max(peak, active);
await new Promise<void>((resolve) => release.push(resolve)); await new Promise<void>((resolve) => release.push(resolve));
active--; active--;
return { cssLayoutViewport: { clientWidth: 100, clientHeight: 100 } }; return {
visualViewport: { clientWidth: 1000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 100, clientHeight: 100 },
};
}); });
const controller = new AbortController(); const controller = new AbortController();
const context = new GeometryContext( const context = new GeometryContext(
@@ -433,6 +433,7 @@ describe("handleRequestHelp", () => {
if (method === "DOM.getContentQuads") { if (method === "DOM.getContentQuads") {
return { quads: [[10, 20, 110, 20, 110, 60, 10, 60]] }; return { quads: [[10, 20, 110, 20, 110, 60, 10, 60]] };
} }
if (method === "Runtime.evaluate") return { result: { value: { width: 200, height: 100 } } };
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } }; return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } };
} }
@@ -201,6 +201,7 @@ describe("handleClick", () => {
if (method === "DOM.getContentQuads") { if (method === "DOM.getContentQuads") {
return { quads: [[10, 20, 110, 20, 110, 60, 10, 60]] }; return { quads: [[10, 20, 110, 20, 110, 60, 10, 60]] };
} }
if (method === "Runtime.evaluate") return { result: { value: { width: 200, height: 100 } } };
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } }; return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } };
} }
@@ -220,6 +221,7 @@ describe("handleClick", () => {
{ sessionId: "child-session", method: "DOM.scrollIntoViewIfNeeded" }, { sessionId: "child-session", method: "DOM.scrollIntoViewIfNeeded" },
{ sessionId: "child-session", method: "DOM.getContentQuads" }, { sessionId: "child-session", method: "DOM.getContentQuads" },
{ sessionId: "child-session", method: "Page.getLayoutMetrics" }, { sessionId: "child-session", method: "Page.getLayoutMetrics" },
{ sessionId: "child-session", method: "Runtime.evaluate" },
]); ]);
expect(fake.sent.filter((call) => call.method === "Input.dispatchMouseEvent")).toHaveLength(3); expect(fake.sent.filter((call) => call.method === "Input.dispatchMouseEvent")).toHaveLength(3);
}); });
@@ -93,7 +93,11 @@ function makeFakeCdp(handlers: Record<string, (params?: object) => unknown>) {
sent.push({ method, params }); sent.push({ method, params });
const handler = handlers[method]; const handler = handlers[method];
if (!handler && method === "Page.getLayoutMetrics") { if (!handler && method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } }; return {
visualViewport: { clientWidth: 1000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 },
};
} }
if (!handler) throw new Error(`unexpected CDP call ${method}`); if (!handler) throw new Error(`unexpected CDP call ${method}`);
return handler(params); return handler(params);
@@ -681,6 +685,59 @@ describe("buildFrameVomScene", () => {
expect(nestedInput?.parentId).toBe(result.nodes.find((item) => item.backendNodeId === 210)?.id); expect(nestedInput?.parentId).toBe(result.nodes.find((item) => item.backendNodeId === 210)?.id);
}); });
it("excludes an overlay only in its owning document when target backend ids collide", () => {
const owner = (id: number): CapturedNode => ({
backendNodeId: id,
parentBackendNodeId: null,
frameId: "main",
tag: "iframe",
attrs: {},
rect: { x: id * 10, y: 0, w: 200, h: 100 },
paintOrder: 0,
position: "static",
pointerEvents: "auto",
});
const owners = [owner(10), owner(20)];
const result = buildFrameVomScene(
[
{
frameId: "main",
contextScopeId: "main",
target: { tabId: 7 },
domNodes: owners,
axNodes: [],
},
...["left", "right"].map((frameId, i) => ({
frameId,
contextScopeId: frameId,
parentFrameId: "main",
ownerBackendNodeId: owners[i].backendNodeId,
target: { tabId: 7, sessionId: frameId },
excludedBackendNodeIds: new Set(i === 0 ? [7] : []),
domNodes: [],
axNodes: [
{
nodeId: "button",
frameId,
backendDOMNodeId: 7,
role: { type: "role", value: "button" },
name: { type: "computedString", value: frameId + " action" },
},
],
})),
],
{
nodes: owners,
viewport: { width: 800, height: 600 },
excludedBackendNodeIds: new Set(),
},
);
const rendered = renderVom(result);
expect(rendered.text).not.toContain("left action");
expect(rendered.text).toContain("right action");
expect(rendered.refs.find((ref) => ref.backendNodeId === 7)?.frameId).toBe("right");
});
it("does not place a child document at the page root when its frame boundary is unresolved", () => { it("does not place a child document at the page root when its frame boundary is unresolved", () => {
const childNode: CapturedNode = { const childNode: CapturedNode = {
backendNodeId: 101, backendNodeId: 101,
@@ -2530,7 +2587,11 @@ describe("handleSnapshot", () => {
if (method === "Accessibility.enable") return {}; if (method === "Accessibility.enable") return {};
if (method === "Accessibility.getFullAXTree") return { nodes }; if (method === "Accessibility.getFullAXTree") return { nodes };
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 } }; return {
visualViewport: { clientWidth: 1000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
};
} }
if (method === "DOMSnapshot.enable") return {}; if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") throw new Error("snapshot unsupported"); if (method === "DOMSnapshot.captureSnapshot") throw new Error("snapshot unsupported");
@@ -2617,7 +2678,11 @@ describe("handleSnapshot", () => {
if (method === "Accessibility.enable") return {}; if (method === "Accessibility.enable") return {};
if (method === "Accessibility.getFullAXTree") return { nodes: [root, button] }; if (method === "Accessibility.getFullAXTree") return { nodes: [root, button] };
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 } }; return {
visualViewport: { clientWidth: 1000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
};
} }
if (method === "DOMSnapshot.enable") return {}; if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") { if (method === "DOMSnapshot.captureSnapshot") {
@@ -2625,6 +2690,8 @@ describe("handleSnapshot", () => {
strings, strings,
documents: [ documents: [
{ {
scrollOffsetX: 0,
scrollOffsetY: 0,
frameId: "root", frameId: "root",
nodes: { nodes: {
parentIndex: [-1, 0], parentIndex: [-1, 0],
@@ -2709,7 +2776,11 @@ describe("handleSnapshot", () => {
if (method === "Accessibility.enable") return {}; if (method === "Accessibility.enable") return {};
if (method === "Accessibility.getFullAXTree") return { nodes: [root, button] }; if (method === "Accessibility.getFullAXTree") return { nodes: [root, button] };
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 } }; return {
visualViewport: { clientWidth: 1000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
};
} }
if (method === "DOMSnapshot.enable") return {}; if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") { if (method === "DOMSnapshot.captureSnapshot") {
@@ -2717,6 +2788,8 @@ describe("handleSnapshot", () => {
strings, strings,
documents: [ documents: [
{ {
scrollOffsetX: 0,
scrollOffsetY: 0,
frameId: "root", frameId: "root",
nodes: { nodes: {
parentIndex: [-1, 0], parentIndex: [-1, 0],
@@ -2804,7 +2877,11 @@ describe("handleSnapshot", () => {
if (method === "Accessibility.enable") return {}; if (method === "Accessibility.enable") return {};
if (method === "Accessibility.getFullAXTree") return { nodes: [root, button] }; if (method === "Accessibility.getFullAXTree") return { nodes: [root, button] };
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 } }; return {
visualViewport: { clientWidth: 1000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
};
} }
if (method === "DOMSnapshot.enable") return {}; if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") { if (method === "DOMSnapshot.captureSnapshot") {
@@ -2812,6 +2889,8 @@ describe("handleSnapshot", () => {
strings, strings,
documents: [ documents: [
{ {
scrollOffsetX: 0,
scrollOffsetY: 0,
frameId: "root", frameId: "root",
nodes: { nodes: {
parentIndex: [-1, 0], parentIndex: [-1, 0],
@@ -2970,7 +3049,11 @@ describe("handleSnapshot", () => {
}; };
} }
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1000, clientHeight: 800 } }; return {
visualViewport: { clientWidth: 1000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800 },
};
} }
if (method === "DOMSnapshot.enable") return {}; if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") { if (method === "DOMSnapshot.captureSnapshot") {
@@ -3077,6 +3160,8 @@ describe("handleSnapshot", () => {
strings: S, strings: S,
documents: [ documents: [
{ {
scrollOffsetX: 0,
scrollOffsetY: 0,
frameId: "root", frameId: "root",
nodes: { nodes: {
parentIndex: [-1, 0, 1, 2], parentIndex: [-1, 0, 1, 2],
@@ -3135,6 +3220,8 @@ describe("handleSnapshot", () => {
strings: S, strings: S,
documents: [ documents: [
{ {
scrollOffsetX: 0,
scrollOffsetY: 0,
frameId: "root", frameId: "root",
nodes: { nodes: {
parentIndex: [-1, 0, 1, 1, 1, 1], parentIndex: [-1, 0, 1, 1, 1, 1],
@@ -3185,10 +3272,14 @@ describe("handleSnapshot", () => {
} }
const VP_METRICS = { const VP_METRICS = {
visualViewport: { clientWidth: 1000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 }, cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
}; };
function makeFrameAwareDeps() { function makeFrameAwareDeps(
ownerGeometry: "unavailable" | "available" | "clipped" = "unavailable",
) {
const strings = ["html", "body", "iframe", "button", "title", "Remote", "static", "auto"]; const strings = ["html", "body", "iframe", "button", "title", "Remote", "static", "auto"];
const i = (value: string) => strings.indexOf(value); const i = (value: string) => strings.indexOf(value);
const styles = [i("static"), i("auto"), i("auto")]; const styles = [i("static"), i("auto"), i("auto")];
@@ -3196,6 +3287,8 @@ describe("handleSnapshot", () => {
strings, strings,
documents: [ documents: [
{ {
scrollOffsetX: 0,
scrollOffsetY: 0,
frameId: "main", frameId: "main",
nodes: { nodes: {
parentIndex: [-1, 0, 1], parentIndex: [-1, 0, 1],
@@ -3216,6 +3309,8 @@ describe("handleSnapshot", () => {
}, },
}, },
{ {
scrollOffsetX: 0,
scrollOffsetY: 0,
frameId: "child", frameId: "child",
nodes: { nodes: {
parentIndex: [-1, 0, 1], parentIndex: [-1, 0, 1],
@@ -3268,6 +3363,16 @@ describe("handleSnapshot", () => {
]; ];
const send = vi.fn(async (_tabId: number, method: string) => { const send = vi.fn(async (_tabId: number, method: string) => {
if (method === "Page.getLayoutMetrics") return VP_METRICS; if (method === "Page.getLayoutMetrics") return VP_METRICS;
if (ownerGeometry !== "unavailable") {
if (method === "DOM.getBoxModel") {
const x = ownerGeometry === "clipped" ? 10000 : 100;
return { model: { content: [x, 100, x + 400, 100, x + 400, 400, x, 400] } };
}
if (method === "DOM.resolveNode") return { object: { objectId: "owner" } };
if (method === "Runtime.callFunctionOn")
return { result: { value: { width: 400, height: 300 } } };
if (method === "Runtime.releaseObject") return {};
}
if (method === "DOMSnapshot.enable" || method === "Accessibility.enable") return {}; if (method === "DOMSnapshot.enable" || method === "Accessibility.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") if (method === "DOMSnapshot.captureSnapshot")
return { ...snapshot, documents: [snapshot.documents[0]] }; return { ...snapshot, documents: [snapshot.documents[0]] };
@@ -3275,6 +3380,13 @@ describe("handleSnapshot", () => {
throw new Error(`unexpected root CDP method ${method}`); throw new Error(`unexpected root CDP method ${method}`);
}); });
const sendToTarget = vi.fn(async (_target, method: string) => { const sendToTarget = vi.fn(async (_target, method: string) => {
if (method === "Runtime.evaluate") return { result: { value: { width: 400, height: 300 } } };
if (method === "Page.getLayoutMetrics" && ownerGeometry !== "unavailable")
return {
visualViewport: { clientWidth: 1000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 400, clientHeight: 300 },
};
if (method === "Accessibility.enable" || method === "DOMSnapshot.enable") return {}; if (method === "Accessibility.enable" || method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") if (method === "DOMSnapshot.captureSnapshot")
return { ...snapshot, documents: [snapshot.documents[1]] }; return { ...snapshot, documents: [snapshot.documents[1]] };
@@ -3309,6 +3421,68 @@ describe("handleSnapshot", () => {
}; };
} }
it.each([
false,
true,
])("preserves external-link hints without graph metadata (snapshot URL: %s)", async (hasSnapshotUrl) => {
const snapshot = {
strings: ["body", "a", "href", "//outside.test/docs", "https://page.test/start"],
documents: [
{
scrollOffsetX: 0,
scrollOffsetY: 0,
frameId: "main",
...(hasSnapshotUrl ? { documentURL: 4 } : {}),
nodes: {
backendNodeId: [1, 2],
parentIndex: [-1, 0],
nodeName: [0, 1],
attributes: [[], [2, 3]],
},
layout: {
nodeIndex: [0, 1],
bounds: [
[0, 0, 1000, 800],
[10, 20, 100, 40],
],
},
},
],
};
const ax: CdpAxNode[] = [
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["link"],
},
{
nodeId: "link",
parentId: "root",
backendDOMNodeId: 2,
role: { type: "role", value: "link" },
name: { type: "computedString", value: "Docs" },
},
];
const deps = makeOverlayDeps(ax, snapshot, VP_METRICS);
const cdp: CdpRunner = {
...deps.cdp,
getFrameGraph: async () => {
throw new Error("graph unavailable");
},
};
const result = await captureVomObservation(
cdp,
4,
hasSnapshotUrl ? undefined : "https://page.test/start",
);
expect(result.text).toContain("outside.test");
expect(result.text).toContain("Docs");
expect(
deps.send.mock.calls.filter(([, method]) => method === "DOMSnapshot.captureSnapshot"),
).toHaveLength(1);
});
it("exposes the frame-aware observation result for recording", async () => { it("exposes the frame-aware observation result for recording", async () => {
const ax: CdpAxNode[] = [ const ax: CdpAxNode[] = [
{ {
@@ -3378,6 +3552,8 @@ describe("handleSnapshot", () => {
strings, strings,
documents: [ documents: [
{ {
scrollOffsetX: 0,
scrollOffsetY: 0,
frameId: "root", frameId: "root",
nodes: { nodes: {
parentIndex: [-1, 0, 1, 1, 3], parentIndex: [-1, 0, 1, 1, 3],
@@ -3512,6 +3688,98 @@ describe("handleSnapshot", () => {
expect(result.text).toContain("•••"); expect(result.text).toContain("•••");
}); });
it.each([
undefined,
100,
0,
])("reports unavailable iframe geometry within the observation budget %s", async (maxTokens) => {
const result = await captureVomObservation(makeFrameAwareDeps().cdp, 4, "https://example.com", {
maxTokens,
});
expect(result.text.match(/@warning/g)).toHaveLength(2);
expect(result.text).toContain("document identity unverified");
expect(result.text).toContain("geometry incomplete");
expect(result.text.startsWith("@vom 1\n")).toBe(true);
if (maxTokens === undefined) {
expect(result.truncated).toBe(false);
expect(result.text).toContain("Frame action");
expect(result.matchNodes.find((node) => node.backendNodeId === 22)).toMatchObject({
rect: null,
localRect: null,
});
}
if (maxTokens === 100) expect(Math.ceil(result.text.length / 4)).toBeLessThanOrEqual(maxTokens);
});
it.each([
["DOMSnapshot.captureSnapshot", "dom"],
["Accessibility.getFullAXTree", "ax"],
])("reports partial capture when child %s fails", async (failedMethod, stage) => {
const { cdp } = makeFrameAwareDeps("available");
const send = cdp.sendToTarget;
cdp.sendToTarget = async (target, method, params) => {
if (method === failedMethod) throw new Error("unavailable child data");
return send(target, method, params);
};
for (const maxTokens of [undefined, 100, 0]) {
const result = await captureVomObservation(cdp, 4, "https://example.com", { maxTokens });
expect(result.text).toContain(`@warning observation incomplete: some ${stage} data`);
expect(result.text.match(/@warning/g)).toHaveLength(2);
expect(result.text).toContain("document identity unverified");
expect(result.text.startsWith("@vom 1\n")).toBe(true);
if (maxTokens === undefined) {
expect(result.frames.map((frame) => frame.frameId)).toEqual(["main", "child"]);
expect(result.truncated).toBe(false);
}
if (maxTokens === 100)
expect(Math.ceil(result.text.length / 4)).toBeLessThanOrEqual(maxTokens);
}
});
it("reports omitted child ownership without exposing capture internals", async () => {
const { cdp } = makeFrameAwareDeps("available");
const send = cdp.send;
cdp.send = async (tabId, method, params) => {
const result = await send(tabId, method, params);
if (method === "DOMSnapshot.captureSnapshot") {
const child = await cdp.sendToTarget({ tabId, sessionId: "child-session" }, method, params);
(result as { documents: unknown[] }).documents.push(
...(child as { documents: unknown[] }).documents,
);
}
return result as never;
};
const result = await captureVomObservation(cdp, 4, "https://example.com");
expect(result.frames.map((frame) => frame.frameId)).toEqual(["main"]);
expect(result.text).toContain("observation incomplete: some ownership data");
expect(result.text).not.toContain("Frame action");
expect(result).not.toHaveProperty("issues");
});
it("reports unavailable form state while retaining the observed document", async () => {
const { cdp } = makeOverlayDeps([], loginSnapshotReply(), VP_METRICS);
const result = await captureVomObservation(cdp, 4, "https://example.com");
expect(result.text).toContain("observation incomplete: some forms data");
expect(result.frames).toHaveLength(1);
expect(result.text).not.toContain("unexpected CDP method");
});
it.each([
"available",
"clipped",
] as const)("does not report successful %s geometry as an observation failure", async (ownerGeometry) => {
const result = await captureVomObservation(
makeFrameAwareDeps(ownerGeometry).cdp,
4,
"https://example.com",
);
expect(result.text).not.toContain("iframe geometry incomplete");
expect(result.text).toContain("document identity unverified");
const node = result.matchNodes.find((item) => item.backendNodeId === 22);
if (ownerGeometry === "clipped") expect(node?.rect).toBeNull();
else expect(node?.rect).toEqual({ x: 120, y: 130, w: 120, h: 40 });
});
it("keeps frames and rendered refs on the same frame identity", async () => { it("keeps frames and rendered refs on the same frame identity", async () => {
const deps = makeFrameAwareDeps(); const deps = makeFrameAwareDeps();
@@ -3528,7 +3796,7 @@ describe("handleSnapshot", () => {
expect(result.matchNodes.find((node) => node.backendNodeId === 22)).toMatchObject({ expect(result.matchNodes.find((node) => node.backendNodeId === 22)).toMatchObject({
frameId: "child", frameId: "child",
tag: "button", tag: "button",
localRect: { x: 20, y: 30, w: 120, h: 40 }, localRect: null,
}); });
expect(result.refs.find((ref) => ref.backendNodeId === 22)).toMatchObject({ expect(result.refs.find((ref) => ref.backendNodeId === 22)).toMatchObject({
frameId: "child", frameId: "child",
@@ -3872,6 +4140,8 @@ describe("handleSnapshot", () => {
strings: S, strings: S,
documents: [ documents: [
{ {
scrollOffsetX: 0,
scrollOffsetY: 0,
frameId: "root", frameId: "root",
nodes: { nodes: {
parentIndex: [-1, 0, 1, 0, 3, 4, 5], parentIndex: [-1, 0, 1, 0, 3, 4, 5],
@@ -0,0 +1,355 @@
// @vitest-environment node
import { describe, expect, it } from "vitest";
import type { CdpFrame, CdpFrameGraph, CdpTarget } from "@/browser-driver/frame-graph";
import { resolveNodeGeometry } from "../frame-geometry";
import type { CdpRunner } from "../shared";
import { captureObservationFacts } from "../vom/capture-coordinator";
type Send = <T = Record<string, unknown>>(
method: string,
params?: object,
sessionId?: string,
) => Promise<T>;
type Tree = { frame: { id: string; parentId?: string; name?: string }; childFrames?: Tree[] };
type Rect = { x: number; y: number; w: number; h: number };
type Oracle = {
probes: Record<string, Rect>;
viewport: { width: number; height: number };
owners: Record<string, { x: number; y: number; scale: number }>;
};
function clip(rect: Rect, viewport: Oracle["viewport"]): Rect | null {
const x = Math.max(0, rect.x),
y = Math.max(0, rect.y);
const right = Math.min(viewport.width, rect.x + rect.w);
const bottom = Math.min(viewport.height, rect.y + rect.h);
return right > x && bottom > y ? { x, y, w: right - x, h: bottom - y } : null;
}
// The independent oracle uses DOM border boxes and this fixture's axis-aligned
// iframe transforms. No production snapshot conversion/projection builds expectations.
const oracleExpression = `(() => {
const probes = {};
for (const node of document.querySelectorAll('[data-geometry-probe]')) {
const box = node.getBoundingClientRect();
probes[node.id] = { x: box.x, y: box.y, w: box.width, h: box.height };
}
const owners = {};
for (const frame of document.querySelectorAll('iframe')) {
const rect = frame.getBoundingClientRect();
const style = getComputedStyle(frame);
const scale = rect.width / frame.offsetWidth;
owners[frame.name] = {
x: rect.x + (frame.clientLeft + parseFloat(style.paddingLeft)) * scale,
y: rect.y + (frame.clientTop + parseFloat(style.paddingTop)) * scale,
scale,
};
}
return { probes, viewport: { width: document.documentElement.clientWidth, height: document.documentElement.clientHeight }, owners };
})()`;
describe.skipIf(!process.env.BSK_GEOMETRY_CHROME)("real DOMSnapshot coordinate contract", () => {
it.each(
[
{ deviceScale: 1, zoom: 1 },
{ deviceScale: 0.8, zoom: 1 },
{ deviceScale: 1, zoom: 1.25 },
{ deviceScale: 2, zoom: 1 },
{ deviceScale: 2, zoom: 0.8 },
]
.flatMap((configuration) => [
{ ...configuration, fixture: "snapshot-coordinates", scrollbars: "none" },
{ ...configuration, fixture: "oopif-scrollbars", scrollbars: "both" },
])
.concat(
["vertical", "horizontal", "none"].map((scrollbars) => ({
deviceScale: 1,
zoom: 1,
fixture: "oopif-scrollbars",
scrollbars,
})),
),
)("$fixture: device scale $deviceScale, zoom $zoom, scrollbars $scrollbars", async (configuration) => {
const evalRoot = new URL("../../../../../evals/browser/", import.meta.url);
const { createEvalServer } = await import(new URL("lib/server.mjs", evalRoot).href);
const { withChrome } = await import(
new URL("cases/regression/snapshot-coordinates/chrome.mjs", evalRoot).href
);
const server = createEvalServer();
const { baseUrl } = await server.start();
try {
await withChrome(
{ executable: process.env.BSK_GEOMETRY_CHROME, ...configuration },
async (send: Send) => {
const { targetId } = await send<{ targetId: string }>("Target.createTarget", {
url: "about:blank",
});
const { sessionId: rootSession } = await send<{ sessionId: string }>(
"Target.attachToTarget",
{ targetId, flatten: true },
);
await send(
"Page.navigate",
{
url: `${baseUrl}/${configuration.fixture}?run=coordinates&scrollbars=${configuration.scrollbars}`,
},
rootSession,
);
await expect
.poll(
() =>
server
.snapshot("coordinates")
.events.some(
(event: { type: string; data: { root?: boolean } }) =>
event.type === "geometry.ready" && event.data.root,
),
{ timeout: 10_000 },
)
.toBe(true);
const targets = await send<{ targetInfos: { targetId: string; type: string }[] }>(
"Target.getTargets",
);
const sessions = [rootSession];
for (const target of targets.targetInfos.filter((target) => target.type === "iframe")) {
const { sessionId } = await send<{ sessionId: string }>("Target.attachToTarget", {
targetId: target.targetId,
flatten: true,
});
sessions.push(sessionId);
}
expect(sessions.length).toBe(configuration.fixture === "oopif-scrollbars" ? 3 : 2);
const frames: CdpFrame[] = [];
const names = new Map<string, string>();
const sessionFor = (target: CdpTarget) => target.sessionId ?? rootSession;
for (const sessionId of sessions) {
const { frameTree } = await send<{ frameTree: Tree }>(
"Page.getFrameTree",
{},
sessionId,
);
const target: CdpTarget = {
tabId: 1,
...(sessionId === rootSession ? {} : { sessionId }),
};
const visit = (tree: Tree, parentFrameId = tree.frame.parentId) => {
frames.push({ frameId: tree.frame.id, parentFrameId, target });
names.set(tree.frame.id, tree.frame.name ?? "");
for (const child of tree.childFrames ?? []) visit(child, tree.frame.id);
};
visit(frameTree);
}
expect(frames).toHaveLength(configuration.fixture === "oopif-scrollbars" ? 3 : 5);
for (const frame of frames) {
if (!frame.parentFrameId) continue;
const parent = frames.find((item) => item.frameId === frame.parentFrameId)!;
const owner = await send<{ backendNodeId: number }>(
"DOM.getFrameOwner",
{ frameId: frame.frameId },
sessionFor(parent.target),
);
frame.ownerBackendNodeId = owner.backendNodeId;
}
const graph: CdpFrameGraph = { rootFrameId: frames[0].frameId, frames };
const calls: string[] = [];
const cdp: CdpRunner = {
send: (tabId, method, params) => {
calls.push(`${tabId}:${method}`);
return send(method, params, rootSession);
},
sendToTarget: (target, method, params) => {
calls.push(`${target.sessionId ?? target.tabId}:${method}`);
if (
method === "Runtime.evaluate" &&
(params as { expression?: string })?.expression?.includes("window.innerWidth")
)
calls.push(`${target.sessionId ?? target.tabId}:viewport-size`);
return send(method, params, sessionFor(target));
},
getFrameGraph: async () => graph,
getAttachmentId: () => rootSession,
};
const oracles = new Map<string, Oracle>();
for (const frame of frames) {
const session = sessionFor(frame.target);
const { executionContextId } = await send<{ executionContextId: number }>(
"Page.createIsolatedWorld",
{ frameId: frame.frameId, worldName: "coordinate-oracle" },
session,
);
await expect
.poll(
async () => {
const ready = await send<{ result: { value: boolean } }>(
"Runtime.evaluate",
{
expression: 'document.documentElement.dataset.geometryReady === "true"',
contextId: executionContextId,
returnByValue: true,
},
session,
);
return ready.result.value;
},
{ timeout: 5_000 },
)
.toBe(true);
// Establish scroll after cross-process target creation/layout settles.
const scroll = frame.parentFrameId
? names.get(frame.frameId) === "nested"
? [10, 20]
: [40, 100]
: [80, 240];
await send(
"Runtime.evaluate",
{
expression: `(async () => { scrollTo(${scroll.join(",")}); await new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve))); })()`,
contextId: executionContextId,
awaitPromise: true,
},
session,
);
const reply = await send<{ result: { value: Oracle } }>(
"Runtime.evaluate",
{ expression: oracleExpression, contextId: executionContextId, returnByValue: true },
session,
);
oracles.set(frame.frameId, reply.result.value);
}
const childViewport = await send<{
result: {
value: { width: number; height: number; clientWidth: number; clientHeight: number };
};
}>(
"Runtime.evaluate",
{
expression:
"({ width: innerWidth, height: innerHeight, clientWidth: document.documentElement.clientWidth, clientHeight: document.documentElement.clientHeight })",
returnByValue: true,
},
sessions[1],
);
for (const dimension of ["Width", "Height"] as const) {
const occupied =
childViewport.result.value[dimension === "Width" ? "width" : "height"] -
childViewport.result.value[`client${dimension}`];
if (
configuration.scrollbars === "both" ||
configuration.scrollbars === (dimension === "Width" ? "vertical" : "horizontal")
)
expect(occupied).toBeGreaterThan(0);
else expect(occupied).toBe(0);
}
const facts = await captureObservationFacts(cdp, 1);
expect(facts.issues).toEqual([]);
expect(calls.filter((call) => call.endsWith(":Page.getLayoutMetrics"))).toHaveLength(
sessions.length,
);
// Identity verification also uses Runtime.evaluate; count the full viewport reads separately.
const viewportReadsBeforeLive = calls.filter((call) => call.endsWith(":viewport-size"));
expect(new Set(viewportReadsBeforeLive).size).toBe(viewportReadsBeforeLive.length);
expect(viewportReadsBeforeLive.length).toBeLessThanOrEqual(sessions.length - 1);
for (const frame of frames) {
for (const [id, local] of Object.entries(oracles.get(frame.frameId)!.probes)) {
const node = facts.documents
.find((doc) => doc.frame.frameId === frame.frameId)
?.domNodes.find((node) => node.attrs.id === id);
expect(node, `missing ${id} in ${names.get(frame.frameId) || "root"}`).toBeDefined();
let top = clip(local, oracles.get(frame.frameId)!.viewport);
let current = frame;
while (top && current.parentFrameId) {
const parentOracle = oracles.get(current.parentFrameId)!;
const owner = parentOracle.owners[names.get(current.frameId)!];
top = clip(
{
x: owner.x + top.x * owner.scale,
y: owner.y + top.y * owner.scale,
w: top.w * owner.scale,
h: top.h * owner.scale,
},
parentOracle.viewport,
);
current = frames.find((item) => item.frameId === current.parentFrameId)!;
}
for (const key of ["x", "y", "w", "h"] as const)
expect(
Math.abs(node!.localRect![key] - local[key]),
`${names.get(frame.frameId)} ${id} local ${key}`,
).toBeLessThan(2);
if (!top) {
expect(node!.rect, `${id} must be clipped out`).toBeNull();
if (configuration.fixture === "oopif-scrollbars")
expect(
await resolveNodeGeometry(cdp, 1, {
target: frame.target,
frameId: frame.frameId,
backendNodeId: node!.backendNodeId,
}),
).toMatchObject({ code: "permission_denied" });
continue;
}
expect(node!.rect).not.toBeNull();
for (const key of ["x", "y", "w", "h"] as const)
expect(
Math.abs(node!.rect![key] - top[key]),
`${names.get(frame.frameId)} ${id} top ${key}`,
).toBeLessThan(2);
if (configuration.fixture !== "oopif-scrollbars") continue;
const live = await resolveNodeGeometry(cdp, 1, {
target: frame.target,
frameId: frame.frameId,
backendNodeId: node!.backendNodeId,
});
if ("code" in live) throw new Error(live.message);
for (const [key, liveKey] of [
["x", "x"],
["y", "y"],
["w", "width"],
["h", "height"],
] as const)
expect(
Math.abs(live.topBounds[liveKey] - top[key]),
`${id} live ${key}`,
).toBeLessThan(2);
const before = server.snapshot("coordinates").events.length;
for (const type of ["mousePressed", "mouseReleased"])
await send(
"Input.dispatchMouseEvent",
{ type, ...live.actionPoint, button: "left", clickCount: 1 },
rootSession,
);
await expect
.poll(
() =>
server
.snapshot("coordinates")
.events.slice(before)
.some(
(event: { type: string; path: string; data: { probe?: string } }) =>
event.type === "geometry.clicked" &&
event.data.probe === id &&
event.path ===
(frame.parentFrameId
? names.get(frame.frameId) === "nested"
? "/oopif-scrollbars/nested"
: "/oopif-scrollbars/frame"
: "/oopif-scrollbars"),
),
{ timeout: 3000 },
)
.toBe(true);
}
}
const metrics = await send<{
cssVisualViewport: { zoom: number };
visualViewport: { clientWidth: number };
}>("Page.getLayoutMetrics", {}, rootSession);
expect(metrics.cssVisualViewport.zoom).toBeCloseTo(configuration.zoom, 4);
},
);
} finally {
await server.stop();
}
}, 30_000);
});
+2 -1
View File
@@ -161,7 +161,8 @@ export async function resolveNodeGeometry(
return geometryError(`could not resolve frame geometry for ${address.frameId}`); return geometryError(`could not resolve frame geometry for ${address.frameId}`);
topVisibleRegions = projectRegionToViewport(localRegion, projection); topVisibleRegions = projectRegionToViewport(localRegion, projection);
if (address.target.sessionId) { if (address.target.sessionId) {
const localViewport = projection.edges[0]?.sourceViewport ?? projection.topViewport; const localViewport = await context.viewport(address.target);
if (!localViewport) return geometryError("could not resolve target viewport geometry");
const localVisibleRegions = localRegion const localVisibleRegions = localRegion
.map((polygon) => .map((polygon) =>
clipPolygon( clipPolygon(
@@ -18,21 +18,94 @@ export interface SnapshotProjection {
geometry: GeometryProjection; geometry: GeometryProjection;
} }
/** DOMSnapshot bounds are document-relative CSS pixels, independent of raster DPR. */ /** A failed owner read is different from a successfully projected, clipped-out rect. */
export interface UnavailableFrameProjection {
status: "unavailable";
source: CoordinateOwner;
ownerBackendNodeId: number;
}
export interface UnavailableSnapshotCoordinates {
status: "unavailable";
source: CoordinateOwner;
reason: "snapshot-coordinates-unavailable";
}
export type SnapshotProjectionResult =
| { status: "available"; projection: SnapshotProjection }
| UnavailableFrameProjection;
/** Descendants inherit the failed boundary without attempting another owner read. */
export type FrameProjectionIssue =
| UnavailableFrameProjection
| UnavailableSnapshotCoordinates
| {
status: "blocked";
source: CoordinateOwner;
cause: UnavailableFrameProjection | UnavailableSnapshotCoordinates;
};
export type FrameProjectionState = SnapshotProjectionResult | FrameProjectionIssue;
export interface SnapshotCoordinates {
layoutUnitsPerCssPixel: number;
scrollCss: { x: number; y: number };
}
/** Snapshot scroll offsets share the raw layout units of its bounds. Only a
* target's root document may fall back to that target's CSS layout viewport. */
export function snapshotCoordinates(
document: { scrollOffsetX?: number; scrollOffsetY?: number },
layoutUnitsPerCssPixel: number | null,
rootScrollCss?: { x?: number; y?: number },
): SnapshotCoordinates | null {
if (
layoutUnitsPerCssPixel === null ||
!Number.isFinite(layoutUnitsPerCssPixel) ||
layoutUnitsPerCssPixel <= 0
)
return null;
const x =
document.scrollOffsetX === undefined
? rootScrollCss?.x
: document.scrollOffsetX / layoutUnitsPerCssPixel;
const y =
document.scrollOffsetY === undefined
? rootScrollCss?.y
: document.scrollOffsetY / layoutUnitsPerCssPixel;
if (x === undefined || y === undefined || !Number.isFinite(x) || !Number.isFinite(y)) return null;
return { layoutUnitsPerCssPixel, scrollCss: { x, y } };
}
/** Convert raw document-relative Blink layout units exactly once, before any
* CSS viewport clipping or frame projection. This scale is not screenshot zoom. */
export function snapshotViewportRect( export function snapshotViewportRect(
bounds: number[], bounds: number[],
owner: CoordinateOwner, owner: CoordinateOwner,
scroll: { x: number; y: number }, coordinates: SnapshotCoordinates | null,
): FrameViewportRect | null { ): FrameViewportRect | null {
if (bounds.length < 4 || !bounds.slice(0, 4).every(Number.isFinite)) return null; if (!coordinates || bounds.length < 4 || !bounds.slice(0, 4).every(Number.isFinite)) return null;
const { layoutUnitsPerCssPixel: scale, scrollCss: scroll } = coordinates;
const [x, y, width, height] = bounds; const [x, y, width, height] = bounds;
if (width <= 0 || height <= 0 || !Number.isFinite(scroll.x) || !Number.isFinite(scroll.y)) { if (
!Number.isFinite(scale) ||
scale <= 0 ||
width <= 0 ||
height <= 0 ||
!Number.isFinite(scroll.x) ||
!Number.isFinite(scroll.y)
) {
return null; return null;
} }
return { return {
space: "frame-viewport-css", space: "frame-viewport-css",
owner, owner,
rect: { x: x - scroll.x, y: y - scroll.y, width, height }, rect: {
x: x / scale - scroll.x,
y: y / scale - scroll.y,
width: width / scale,
height: height / scale,
},
}; };
} }
@@ -9,15 +9,19 @@ import {
type Polygon, type Polygon,
type ProjectiveEdge, type ProjectiveEdge,
parseCdpQuad, parseCdpQuad,
projectPolygon,
type Quad, type Quad,
type Size, type Size,
viewportPolygon,
} from "../geometry"; } from "../geometry";
import { type CdpRunner, sendToCdpTarget } from "../shared"; import { type CdpRunner, sendToCdpTarget } from "../shared";
import type { CoordinateOwner, CssViewport, SnapshotProjection } from "./coordinate-types"; import type { CoordinateOwner, CssViewport, SnapshotProjectionResult } from "./coordinate-types";
import { readSnapshotOwnerSizes } from "./snapshot-owner-sizes";
export interface LayoutMetrics { export interface LayoutMetrics {
cssVisualViewport?: { zoom?: number }; cssVisualViewport?: { zoom?: number; clientWidth?: number; clientHeight?: number };
visualViewport?: { zoom?: number }; visualViewport?: { zoom?: number; clientWidth?: number; clientHeight?: number };
cssLayoutViewport?: { cssLayoutViewport?: {
clientWidth?: number; clientWidth?: number;
clientHeight?: number; clientHeight?: number;
@@ -27,6 +31,29 @@ export interface LayoutMetrics {
layoutViewport?: { clientWidth?: number; clientHeight?: number; pageX?: number; pageY?: number }; layoutViewport?: { clientWidth?: number; clientHeight?: number; pageX?: number; pageY?: number };
} }
/** Chromium emits these paired floating-point dimensions before/after dividing
* by LayoutZoomFactor. Integer layout viewport dimensions lose precision at
* fractional zoom. Neither visualViewport.scale nor .zoom is this conversion.
* See InspectorPageAgent::getLayoutMetrics in Chromium. */
export function snapshotLayoutScale(metrics: LayoutMetrics): number | null {
for (const dimension of ["clientWidth", "clientHeight"] as const) {
const raw = metrics.visualViewport?.[dimension];
const css = metrics.cssVisualViewport?.[dimension];
if (
raw !== undefined &&
css !== undefined &&
Number.isFinite(raw) &&
Number.isFinite(css) &&
raw > 0 &&
css > 0
) {
const scale = raw / css;
if (Number.isFinite(scale) && scale > 0) return scale;
}
}
return null;
}
/** The legacy field is a compatibility fallback, never a source of raster DPR. */ /** The legacy field is a compatibility fallback, never a source of raster DPR. */
export function cssViewport(metrics: LayoutMetrics): CssViewport { export function cssViewport(metrics: LayoutMetrics): CssViewport {
const value = metrics.cssLayoutViewport ?? metrics.layoutViewport; const value = metrics.cssLayoutViewport ?? metrics.layoutViewport;
@@ -42,9 +69,11 @@ export function cssViewport(metrics: LayoutMetrics): CssViewport {
/** One read-only measurement phase. Discard after scrolling or any later operation. */ /** One read-only measurement phase. Discard after scrolling or any later operation. */
export class GeometryContext { export class GeometryContext {
private readonly metrics = new Map<string, Promise<LayoutMetrics>>(); private readonly metrics = new Map<string, Promise<LayoutMetrics>>();
private readonly frameViewports = new Map<string, Promise<Size | null>>();
private readonly owners = new Map<string, Promise<Quad | null>>(); private readonly owners = new Map<string, Promise<Quad | null>>();
private readonly projections = new Map<string, Promise<GeometryProjection | null>>(); private readonly projections = new Map<string, Promise<GeometryProjection | null>>();
private readonly snapshotOwners = new Map<string, Promise<{ quad: Quad; size: Size } | null>>(); private readonly snapshotOwners = new Map<string, Promise<{ quad: Quad; size: Size } | null>>();
private readonly sizeBatches = new Map<string, () => Promise<Map<number, Size>>>();
private active = 0; private active = 0;
private readonly waiting: Array<() => void> = []; private readonly waiting: Array<() => void> = [];
private graphPromise?: Promise<CdpFrameGraph | null>; private graphPromise?: Promise<CdpFrameGraph | null>;
@@ -131,6 +160,35 @@ export class GeometryContext {
: null; : null;
} }
/** The full OOPIF viewport maps to its owner's content quad. Layout metrics
* exclude occupied scrollbars and remain the clipping boundary, not the scale. */
private frameViewport(target: CdpTarget): Promise<Size | null> {
const key = cdpTargetKey(target);
let promise = this.frameViewports.get(key);
if (!promise) {
promise = this.request<{ result?: { value?: Size }; exceptionDetails?: unknown }>(
target,
"Runtime.evaluate",
{
expression: "({ width: window.innerWidth, height: window.innerHeight })",
returnByValue: true,
},
).then(({ result, exceptionDetails }) => {
const size = result?.value;
return !exceptionDetails &&
size &&
Number.isFinite(size.width) &&
Number.isFinite(size.height) &&
size.width > 0 &&
size.height > 0
? size
: null;
});
this.frameViewports.set(key, promise);
}
return promise;
}
ownerContent(target: CdpTarget, backendNodeId: number): Promise<Quad | null> { ownerContent(target: CdpTarget, backendNodeId: number): Promise<Quad | null> {
const key = `${cdpTargetKey(target)}:${backendNodeId}`; const key = `${cdpTargetKey(target)}:${backendNodeId}`;
let promise = this.owners.get(key); let promise = this.owners.get(key);
@@ -149,30 +207,60 @@ export class GeometryContext {
ownerBackendNodeId: number, ownerBackendNodeId: number,
ancestorClips: Polygon[], ancestorClips: Polygon[],
viewport: Size, viewport: Size,
): Promise<SnapshotProjection | null> { ): Promise<SnapshotProjectionResult> {
const key = `${cdpTargetKey(source.target)}:${ownerBackendNodeId}`; const key = `${cdpTargetKey(source.target)}:${ownerBackendNodeId}`;
let promise = this.snapshotOwners.get(key); let promise = this.snapshotOwners.get(key);
if (!promise) { if (!promise) {
promise = this.snapshotOwner(source.target, ownerBackendNodeId); promise = this.snapshotOwner(source.target, ownerBackendNodeId);
this.snapshotOwners.set(key, promise); this.snapshotOwners.set(key, promise);
} }
const owner = await promise; let owner: { quad: Quad; size: Size } | null;
try {
owner = await promise;
} catch (error) {
if (error && typeof error === "object" && "name" in error && error.name === "AbortError")
throw error;
console.debug("[bsk geometry] frame owner unavailable", error);
owner = null;
}
return owner return owner
? { ? {
source, status: "available",
geometry: { projection: {
sourceClips: [], source,
edges: [ geometry: {
{ sourceClips: [],
sourceViewport: owner.size, edges: [
destinationQuad: owner.quad, {
destinationClips: ancestorClips, sourceViewport: owner.size,
}, destinationQuad: owner.quad,
], destinationClips: ancestorClips,
topViewport: viewport, },
],
topViewport: viewport,
},
}, },
} }
: null; : { status: "unavailable", source, ownerBackendNodeId };
}
/** Register one target's snapshot owners; the batch starts only when a
* projection actually needs a size. Single owners keep the cheaper old path. */
registerSnapshotOwners(target: CdpTarget, owners: ReadonlySet<number>): void {
const key = cdpTargetKey(target);
if (owners.size < 2 || this.sizeBatches.has(key)) return;
let pending: Promise<Map<number, Size>> | undefined;
this.sizeBatches.set(key, () => {
pending ??= readSnapshotOwnerSizes(
(method, params, cleanup) => this.request(target, method, params, cleanup),
owners,
).catch((error) => {
if (error && typeof error === "object" && "name" in error && error.name === "AbortError")
throw error;
return new Map<number, Size>();
});
return pending;
});
} }
private async snapshotOwner( private async snapshotOwner(
@@ -181,6 +269,9 @@ export class GeometryContext {
): Promise<{ quad: Quad; size: Size } | null> { ): Promise<{ quad: Quad; size: Size } | null> {
const quad = await this.ownerContent(target, backendNodeId); const quad = await this.ownerContent(target, backendNodeId);
if (!quad) return null; if (!quad) return null;
const batchSize = (await this.sizeBatches.get(cdpTargetKey(target))?.())?.get(backendNodeId);
if (this.signal?.aborted) throw new DOMException("geometry aborted", "AbortError");
if (batchSize) return { quad, size: batchSize };
const resolved = await this.request<{ object?: { objectId?: string } }>( const resolved = await this.request<{ object?: { objectId?: string } }>(
target, target,
"DOM.resolveNode", "DOM.resolveNode",
@@ -283,12 +374,19 @@ export class GeometryContext {
const parent = frames.get(root.parentFrameId); const parent = frames.get(root.parentFrameId);
if (!parent || root.ownerBackendNodeId === undefined) return null; if (!parent || root.ownerBackendNodeId === undefined) return null;
const destinationQuad = await this.ownerContent(parent.target, root.ownerBackendNodeId); const destinationQuad = await this.ownerContent(parent.target, root.ownerBackendNodeId);
const source = await this.viewport(root.target); if (!destinationQuad) return null;
const source = await this.frameViewport(root.target);
const visible = await this.viewport(root.target);
const parentRoot = this.targetRoot(frames, parent); const parentRoot = this.targetRoot(frames, parent);
if (!destinationQuad || !source || !parentRoot) return null; if (!source || !visible || !parentRoot) return null;
const destinationClips = await this.clips(frames, parent, parentRoot); const destinationClips = await this.clips(frames, parent, parentRoot);
if (!destinationClips) return null; if (!destinationClips) return null;
edges.push({ sourceViewport: source, destinationQuad, destinationClips }); const edge = { sourceViewport: source, destinationQuad, destinationClips };
// Keep scrollbar strips clipped at every target boundary, including
// intermediate OOPIFs. Project the clip with the same full-viewport scale.
if (source.width !== visible.width || source.height !== visible.height)
destinationClips.push(projectPolygon(viewportPolygon(visible), edge));
edges.push(edge);
root = parentRoot; root = parentRoot;
} }
const topViewport = await this.viewport(root.target); const topViewport = await this.viewport(root.target);
@@ -0,0 +1,74 @@
import type { Size } from "../geometry";
interface SerializedValue {
type: string;
value?: unknown;
}
type Send = <T>(method: string, params: object, cleanup?: boolean) => Promise<T>;
/** Batch only accessible frame owners. Missing/shadow/cross-origin owners keep
* the existing node-scoped read; no page-wide style columns are requested. */
export async function readSnapshotOwnerSizes(
send: Send,
owners: ReadonlySet<number>,
): Promise<Map<number, Size>> {
const sizes = new Map<number, Size>();
const objectGroup = `bsk-frame-sizes-${crypto.randomUUID()}`;
try {
const reply = await send<{ result?: { deepSerializedValue?: SerializedValue } }>(
"Runtime.evaluate",
{
expression: `(() => {
const result = [];
const documents = [document];
for (let i = 0; i < documents.length && result.length < ${owners.size}; i++) {
for (const owner of documents[i].querySelectorAll('iframe,frame')) {
if (result.length >= ${owners.size}) break;
if (!owner.isConnected) continue;
const style = getComputedStyle(owner);
result.push([owner, JSON.stringify({
width: owner.clientWidth - parseFloat(style.paddingLeft) - parseFloat(style.paddingRight),
height: owner.clientHeight - parseFloat(style.paddingTop) - parseFloat(style.paddingBottom)
})]);
try { if (owner.contentDocument) documents.push(owner.contentDocument); } catch {}
}
}
return result;
})()`,
objectGroup,
serializationOptions: {
serialization: "deep",
maxDepth: 3,
additionalParameters: { maxNodeDepth: 0, includeShadowTree: "none" },
},
},
);
const result = reply.result?.deepSerializedValue;
if (result?.type !== "array" || !Array.isArray(result.value)) return sizes;
for (const entry of result.value as SerializedValue[]) {
if (entry.type !== "array" || !Array.isArray(entry.value)) continue;
const [node, data] = entry.value as SerializedValue[];
if (node?.type !== "node" || data?.type !== "string" || typeof data.value !== "string")
continue;
const id = (node.value as { backendNodeId?: number } | undefined)?.backendNodeId;
if (id === undefined || !owners.has(id)) continue;
try {
const size = JSON.parse(data.value) as Size;
if (
size &&
Number.isFinite(size.width) &&
Number.isFinite(size.height) &&
size.width > 0 &&
size.height > 0
)
sizes.set(id, size);
} catch {
/* An invalid entry uses the existing individual read. */
}
}
} finally {
await send("Runtime.releaseObjectGroup", { objectGroup }, true).catch(() => {});
}
return sizes;
}
+33 -2
View File
@@ -894,7 +894,7 @@ export async function captureVomObservation(
throwIfAborted(options.signal, "observation"); throwIfAborted(options.signal, "observation");
await cdp.ensureAttachedToUrl?.(tabId, url); await cdp.ensureAttachedToUrl?.(tabId, url);
throwIfAborted(options.signal, "observation"); throwIfAborted(options.signal, "observation");
const facts = await captureObservationFacts<CdpAxNode>(cdp, tabId, options.signal); const facts = await captureObservationFacts<CdpAxNode>(cdp, tabId, options.signal, url);
const { captured, documents: normalizedDocuments } = semanticCapture(facts); const { captured, documents: normalizedDocuments } = semanticCapture(facts);
throwIfAborted(options.signal, "observation"); throwIfAborted(options.signal, "observation");
const semanticGraph = buildSemanticGraph({ const semanticGraph = buildSemanticGraph({
@@ -924,12 +924,43 @@ export async function captureVomObservation(
captured, captured,
hoverProbes.surfaceProbes, hoverProbes.surfaceProbes,
); );
// Reserve space using the renderer's character-based token estimate. Like VOM
// headers, this integrity notice remains visible even under a tiny token budget.
const notices: string[] = [];
if (facts.issues.some((issue) => issue.stage === "geometry"))
notices.push(
"@warning geometry incomplete: some page or frame content has no top-level coordinates.",
);
const incompleteStages = ["dom", "ax", "forms", "ownership"].filter((stage) =>
facts.issues.some((issue) => issue.stage === stage),
);
if (incompleteStages.length)
notices.push(
`@warning observation incomplete: some ${incompleteStages.join(", ")} data is unavailable or omitted.`,
);
if (
facts.issues.some(
(issue) => issue.stage === "identity" && issue.reason !== "identity-unverified",
)
)
notices.push(
"@warning observation incomplete: some documents were omitted because their identity changed or could not be revalidated.",
);
if (facts.issues.some((issue) => issue.reason === "identity-unverified"))
notices.push(
"@warning document identity unverified: some retained documents could not be checked for changes during capture.",
);
const captureNotice = notices.join("\n");
const rendered = renderVom(decoratedScene, { const rendered = renderVom(decoratedScene, {
maxDepth: options.maxDepth, maxDepth: options.maxDepth,
maxTokens: options.maxTokens, maxTokens:
!captureNotice || options.maxTokens === undefined
? options.maxTokens
: Math.max(0, options.maxTokens - Math.ceil((captureNotice.length + 1) / 4)),
redactValues: options.redactValues, redactValues: options.redactValues,
activeRegionPolicy: options.activeRegionPolicy, activeRegionPolicy: options.activeRegionPolicy,
}); });
if (captureNotice) rendered.text += `\n${captureNotice}`;
throwIfAborted(options.signal, "observation"); throwIfAborted(options.signal, "observation");
return projectRecordSafeObservation({ return projectRecordSafeObservation({
rootFrameId: captured.rootFrameId ?? normalizedDocuments[0]?.frameId ?? "root", rootFrameId: captured.rootFrameId ?? normalizedDocuments[0]?.frameId ?? "root",
File diff suppressed because it is too large. Load diff
@@ -215,7 +215,11 @@ function makeCdp(snapshot: unknown) {
if (method === "DOMSnapshot.enable") return {}; if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") return snapshot; if (method === "DOMSnapshot.captureSnapshot") return snapshot;
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 } }; return {
visualViewport: { clientWidth: 1000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
};
} }
if (method === "Runtime.evaluate") { if (method === "Runtime.evaluate") {
return { return {
@@ -277,6 +281,8 @@ describe("snapshot capture and hover", () => {
if (method === "DOMSnapshot.captureSnapshot") return hoverTriggerSnapshotReply(); if (method === "DOMSnapshot.captureSnapshot") return hoverTriggerSnapshotReply();
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { return {
visualViewport: { clientWidth: 1000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 }, cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
}; };
} }
@@ -340,6 +346,8 @@ describe("snapshot capture and hover", () => {
if (method === "DOMSnapshot.captureSnapshot") return twoHoverTriggerSnapshotReply(); if (method === "DOMSnapshot.captureSnapshot") return twoHoverTriggerSnapshotReply();
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { return {
visualViewport: { clientWidth: 1000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 }, cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
}; };
} }
@@ -388,6 +396,8 @@ describe("snapshot capture and hover", () => {
if (method === "DOMSnapshot.captureSnapshot") return nestedHoverTriggerSnapshotReply(); if (method === "DOMSnapshot.captureSnapshot") return nestedHoverTriggerSnapshotReply();
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { return {
visualViewport: { clientWidth: 1000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 }, cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
}; };
} }
@@ -733,6 +743,8 @@ describe("snapshot capture and hover", () => {
if (method === "DOMSnapshot.captureSnapshot") return fakeSnapshotReply(); if (method === "DOMSnapshot.captureSnapshot") return fakeSnapshotReply();
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { return {
visualViewport: { clientWidth: 1000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 200 }, cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 200 },
}; };
} }
@@ -745,7 +757,7 @@ describe("snapshot capture and hover", () => {
expect(div?.rect).toMatchObject({ y: 0, h: 600 }); expect(div?.rect).toMatchObject({ y: 0, h: 600 });
}); });
it("keeps CSS snapshot bounds independent of legacy metrics ratio", async () => { it("normalizes raw snapshot bounds before viewport clipping", async () => {
const S = ["html", "body", "div", "position", "fixed", "static", "pointer-events", "auto"]; const S = ["html", "body", "div", "position", "fixed", "static", "pointer-events", "auto"];
const i = (s: string) => S.indexOf(s); const i = (s: string) => S.indexOf(s);
const snapshot = { const snapshot = {
@@ -780,6 +792,8 @@ describe("snapshot capture and hover", () => {
if (method === "DOMSnapshot.captureSnapshot") return snapshot; if (method === "DOMSnapshot.captureSnapshot") return snapshot;
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { return {
visualViewport: { clientWidth: 2000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 }, cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
layoutViewport: { clientWidth: 2000, clientHeight: 1600 }, layoutViewport: { clientWidth: 2000, clientHeight: 1600 },
}; };
@@ -797,8 +811,8 @@ describe("snapshot capture and hover", () => {
expect(nodes.find((n) => n.backendNodeId === 12)?.localRect).toEqual({ expect(nodes.find((n) => n.backendNodeId === 12)?.localRect).toEqual({
x: 0, x: 0,
y: 0, y: 0,
w: 2000, w: 1000,
h: 1600, h: 800,
}); });
}); });
@@ -856,6 +870,8 @@ describe("snapshot capture and hover", () => {
if (method === "DOMSnapshot.captureSnapshot") return snapshot; if (method === "DOMSnapshot.captureSnapshot") return snapshot;
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { return {
visualViewport: { clientWidth: 1000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 }, cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
}; };
} }
@@ -946,6 +962,8 @@ describe("snapshot capture and hover", () => {
if (method === "DOMSnapshot.captureSnapshot") return snapshot; if (method === "DOMSnapshot.captureSnapshot") return snapshot;
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { return {
visualViewport: { clientWidth: 1000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 }, cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
}; };
} }
@@ -1035,6 +1053,8 @@ describe("snapshot capture and hover", () => {
if (method === "DOMSnapshot.captureSnapshot") return snapshot; if (method === "DOMSnapshot.captureSnapshot") return snapshot;
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { return {
visualViewport: { clientWidth: 1000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 }, cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
}; };
} }
@@ -1097,6 +1117,8 @@ describe("snapshot capture and hover", () => {
}, },
}, },
{ {
scrollOffsetX: 0,
scrollOffsetY: 0,
frameId: "child", frameId: "child",
nodes: { nodes: {
parentIndex: [-1, 0, 1], parentIndex: [-1, 0, 1],
@@ -1119,6 +1141,8 @@ describe("snapshot capture and hover", () => {
}, },
}, },
{ {
scrollOffsetX: 0,
scrollOffsetY: 0,
frameId: "nested", frameId: "nested",
nodes: { nodes: {
parentIndex: [-1, 0], parentIndex: [-1, 0],
@@ -1146,6 +1170,8 @@ describe("snapshot capture and hover", () => {
if (method === "DOMSnapshot.captureSnapshot") return snapshot; if (method === "DOMSnapshot.captureSnapshot") return snapshot;
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { return {
visualViewport: { clientWidth: 1000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 }, cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
}; };
} }
@@ -1225,6 +1251,8 @@ describe("snapshot capture and hover", () => {
if (method === "DOMSnapshot.captureSnapshot") return snapshot; if (method === "DOMSnapshot.captureSnapshot") return snapshot;
if (method === "Page.getLayoutMetrics") { if (method === "Page.getLayoutMetrics") {
return { return {
visualViewport: { clientWidth: 2000 },
cssVisualViewport: { clientWidth: 1000 },
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 100 }, cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 100 },
layoutViewport: { clientWidth: 2000, clientHeight: 1600 }, layoutViewport: { clientWidth: 2000, clientHeight: 1600 },
}; };
@@ -1233,7 +1261,7 @@ describe("snapshot capture and hover", () => {
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>, }) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
}; };
const { nodes } = semanticCapture(await captureObservationFacts(cdp, 4)).captured; const { nodes } = semanticCapture(await captureObservationFacts(cdp, 4)).captured;
expect(nodes.find((n) => n.backendNodeId === 12)?.rect?.y).toBe(400 - 100); expect(nodes.find((n) => n.backendNodeId === 12)?.rect?.y).toBe(400 / 2 - 100);
}); });
it("collectOverlayExcludedBackendIds walks the pierced overlay host subtree", async () => { it("collectOverlayExcludedBackendIds walks the pierced overlay host subtree", async () => {
@@ -1,5 +1,6 @@
import { describe, expect, it } from "vitest"; import { describe, expect, it, vi } from "vitest";
import { OVERLAY_HOST_MARKER_ATTR } from "@/lib/overlay-bridge"; import { OVERLAY_HOST_MARKER_ATTR } from "@/lib/overlay-bridge";
import { createCaptureCheckpoint } from "../capture-abort";
import { buildDocumentIndex, type DecodedNode } from "../facts"; import { buildDocumentIndex, type DecodedNode } from "../facts";
import { decodeDocument, REQUESTED_STYLES } from "../snapshot"; import { decodeDocument, REQUESTED_STYLES } from "../snapshot";
@@ -23,6 +24,7 @@ describe("document facts", () => {
let reads = 0; let reads = 0;
const input = Array.from({ length: count }, (_, i) => { const input = Array.from({ length: count }, (_, i) => {
const item = node(i, i ? (wide ? 0 : i - 1) : null); const item = node(i, i ? (wide ? 0 : i - 1) : null);
if (i === 0) item.attrs[OVERLAY_HOST_MARKER_ATTR] = "";
Object.defineProperty(item, "parentBackendNodeId", { Object.defineProperty(item, "parentBackendNodeId", {
get: () => { get: () => {
reads++; reads++;
@@ -33,14 +35,13 @@ describe("document facts", () => {
}).reverse(); // parent need not precede child }).reverse(); // parent need not precede child
const index = await buildDocumentIndex(input); const index = await buildDocumentIndex(input);
expect(index.nodes.size).toBe(count); expect(index.nodes.size).toBe(count);
expect(index.ancestry.get(count - 1)).toEqual({ complete: true, overlay: false }); expect(index.excludedBackendNodeIds.size).toBe(count);
expect(index.children.get(0)?.length).toBe(wide ? count - 1 : 1);
expect(reads).toBeLessThanOrEqual(count * 8); expect(reads).toBeLessThanOrEqual(count * 8);
expect(index.nodes.get(count - 1)).toBe(input[0]); expect(index.nodes.get(count - 1)).toBe(input[0]);
} }
}); });
it("marks cycles and orphans incomplete and propagates overlay through shadow roots", async () => { it("handles cycles and orphans and propagates overlay through shadow roots", async () => {
const host = { ...node(1, null), attrs: { [OVERLAY_HOST_MARKER_ATTR]: "" } }; const host = { ...node(1, null), attrs: { [OVERLAY_HOST_MARKER_ATTR]: "" } };
const input = [ const input = [
node(3, 2), node(3, 2),
@@ -53,7 +54,7 @@ describe("document facts", () => {
]; ];
const index = await buildDocumentIndex(input); const index = await buildDocumentIndex(input);
expect([...index.excludedBackendNodeIds].sort()).toEqual([1, 2, 3]); expect([...index.excludedBackendNodeIds].sort()).toEqual([1, 2, 3]);
for (const id of [4, 5, 6, 7]) expect(index.ancestry.get(id)?.complete).toBe(false); expect(index.nodes.size).toBe(input.length);
}); });
it("does not propagate a descendant overlay backwards into a cycle", async () => { it("does not propagate a descendant overlay backwards into a cycle", async () => {
@@ -62,81 +63,51 @@ describe("document facts", () => {
expect([...index.excludedBackendNodeIds]).toEqual([3]); expect([...index.excludedBackendNodeIds]).toEqual([3]);
}); });
it("does not treat a missing snapshot parent as a complete root", async () => { it("decodes current observation bounds, styles and attributes without projecting coordinates", async () => {
const { nodes } = await decodeDocument( const strings = ["div", "aria-hidden", "true", "inert", "", "visible", "0", "static", "auto"];
{ nodes: { backendNodeId: [1], nodeName: [0], parentIndex: [99] } },
["div"],
);
const index = await buildDocumentIndex(nodes);
expect(index.ancestry.get(1)?.complete).toBe(false);
});
it("retains raw box units and independent style/interaction evidence", async () => {
const strings = [
"html",
"canvas",
"aria-hidden",
"true",
"inert",
"",
"hidden",
"visible",
"0",
"auto",
"scroll",
"matrix(2,0,0,2,0,0)",
];
const styles = REQUESTED_STYLES.map((style) => const styles = REQUESTED_STYLES.map((style) =>
style === "visibility" style === "visibility" ? 5 : style === "opacity" ? 6 : style === "position" ? 7 : 8,
? 7
: style === "opacity"
? 8
: style === "overflow-x"
? 10
: style === "transform"
? 11
: 9,
); );
const { nodes } = await decodeDocument( const { nodes } = await decodeDocument(
{ {
nodes: { nodes: { backendNodeId: [1], parentIndex: [-1], nodeName: [0], attributes: [[1, 2, 3, 4]] },
backendNodeId: [1, 2], layout: { nodeIndex: [0], bounds: [[10, 20, 120, 40]], styles: [styles] },
parentIndex: [-1, 0],
nodeName: [0, 1],
attributes: [[2, 3, 4, 5], []],
},
layout: {
nodeIndex: [0, 1],
bounds: [
[0, 0, 200, 100],
[10, 20, 120, 40],
],
clientRects: [
[5, 5, 90, 40],
[0, 0, 120, 40],
],
styles: [styles, styles],
},
}, },
strings, strings,
); );
expect(nodes[0].attrs).toEqual({ "aria-hidden": "true", inert: "" }); expect(nodes[0].attrs).toEqual({ "aria-hidden": "true", inert: "" });
expect(nodes[0].layout).toMatchObject({ expect(nodes[0].layout).toEqual({
boundsSpace: "snapshot-document-css", boundsSpace: "snapshot-document-layout",
clientSpace: "unscaled-client-offset-css", bounds: [10, 20, 120, 40],
clientRect: [5, 5, 90, 40],
styles: { styles: {
position: "static",
"pointer-events": "auto",
cursor: "auto",
visibility: "visible", visibility: "visible",
opacity: "0", opacity: "0",
"overflow-x": "scroll",
transform: "matrix(2,0,0,2,0,0)",
}, },
}); });
expect(nodes[0]).not.toHaveProperty("rect"); expect(nodes[0]).not.toHaveProperty("rect");
expect(nodes[0]).not.toHaveProperty("excluded"); expect(nodes[0]).not.toHaveProperty("excluded");
}); });
it("yields during large inputs so cancellation interrupts indexing", async () => { it("keeps short blocks synchronous and checks cancellation after yielding", async () => {
const clock = vi.spyOn(performance, "now").mockReturnValue(0);
try {
const controller = new AbortController();
const checkpoint = createCaptureCheckpoint(controller.signal);
for (let i = 0; i < 20; i++) expect(checkpoint()).toBeUndefined();
clock.mockReturnValue(9);
const pending = checkpoint();
expect(pending).toBeInstanceOf(Promise);
controller.abort();
await expect(pending).rejects.toMatchObject({ name: "AbortError" });
} finally {
clock.mockRestore();
}
});
it("checks cancellation at bounded indexing blocks", async () => {
const controller = new AbortController(); const controller = new AbortController();
const input = Array.from({ length: 100_000 }, (_, i) => node(i, i ? i - 1 : null)); const input = Array.from({ length: 100_000 }, (_, i) => node(i, i ? i - 1 : null));
let reads = 0; let reads = 0;
@@ -1,6 +1,7 @@
import { describe, expect, it, vi } from "vitest"; import { describe, expect, it, vi } from "vitest";
import { type CdpRunner, cdpRunnerForTarget } from "../../shared"; import { type CdpRunner, cdpRunnerForTarget } from "../../shared";
import { captureObservationFacts } from "../capture-coordinator"; import { enrichFormControlStates } from "../form-capture";
import { decodeDocument } from "../snapshot";
interface Control { interface Control {
id: number; id: number;
@@ -77,21 +78,32 @@ function reply(entries: Array<[number, unknown]>) {
}; };
} }
function fakeCdp(groups: Control[][], evaluate: (params: Record<string, unknown>) => unknown) { async function fakeCdp(
groups: Control[][],
evaluate: (params: Record<string, unknown>) => unknown,
) {
const snap = snapshot(groups); const snap = snapshot(groups);
const send = vi.fn(async (_tabId: number, method: string, params?: object) => { const send = vi.fn(async (_tabId: number, method: string, params?: object) => {
if (method === "DOMSnapshot.captureSnapshot") return snap;
if (method === "Runtime.evaluate") return evaluate(params as Record<string, unknown>); if (method === "Runtime.evaluate") return evaluate(params as Record<string, unknown>);
return {}; return {};
}); });
return { send, cdp: { send: send as CdpRunner["send"] } }; const nodes = await decodeControls(snap);
return { send, cdp: { send: send as CdpRunner["send"] }, nodes };
}
async function decodeControls(snap: ReturnType<typeof snapshot>) {
return Promise.all(
snap.documents.map(async (doc) =>
(await decodeDocument(doc, snap.strings)).nodes.map((node) => ({ ...node, rect: null })),
),
);
} }
describe("form state identity", () => { describe("form state identity", () => {
it("matches reordered controls by identity and preserves controls missing from the batch", async () => { it("matches reordered controls by identity and preserves controls missing from the batch", async () => {
// The snapshot includes a shadow input. The runtime query skips it, // The snapshot includes a shadow input. The runtime query skips it,
// encounters a new input, and sees the remaining inputs in a new order. // encounters a new input, and sees the remaining inputs in a new order.
const { cdp } = fakeCdp( const { cdp, nodes } = await fakeCdp(
[ [
[ [
{ id: 10, value: "" }, { id: 10, value: "" },
@@ -106,23 +118,21 @@ describe("form state identity", () => {
[11, state("alice@example.com")], [11, state("alice@example.com")],
]), ]),
); );
const captured = await captureObservationFacts(cdp, 7); await enrichFormControlStates(cdp, 7, nodes);
expect(captured.documents[0].domNodes.find((n) => n.backendNodeId === 10)).toMatchObject({ expect(nodes[0].find((n) => n.backendNodeId === 10)).toMatchObject({
formValue: "", formValue: "",
formState: "empty", formState: "empty",
formPlaceholder: "snapshot placeholder", formPlaceholder: "snapshot placeholder",
}); });
expect(captured.documents[0].domNodes.find((n) => n.backendNodeId === 11)).toMatchObject({ expect(nodes[0].find((n) => n.backendNodeId === 11)).toMatchObject({
formValue: "alice@example.com", formValue: "alice@example.com",
}); });
expect(captured.documents[0].domNodes.find((n) => n.backendNodeId === 12)).toMatchObject({ expect(nodes[0].find((n) => n.backendNodeId === 12)).toMatchObject({ formValue: "上海" });
formValue: "上海", expect(nodes[0].some((n) => n.backendNodeId === 99)).toBe(false);
});
expect(captured.documents[0].domNodes.some((n) => n.backendNodeId === 99)).toBe(false);
}); });
it("does not shift state between frames when one frame is absent from the runtime batch", async () => { it("does not shift state between frames when one frame is absent from the runtime batch", async () => {
const { cdp } = fakeCdp( const { cdp, nodes } = await fakeCdp(
[ [
[], [],
[{ id: 11, value: "inaccessible frame" }], [{ id: 11, value: "inaccessible frame" }],
@@ -135,10 +145,8 @@ describe("form state identity", () => {
[21, state("second frame")], [21, state("second frame")],
]), ]),
); );
const captured = await captureObservationFacts(cdp, 7); await enrichFormControlStates(cdp, 7, nodes);
const controls = captured.documents const controls = nodes.slice(1).flat();
.filter((doc) => doc.frame.parentFrameId)
.flatMap((doc) => doc.domNodes);
expect(controls.find((n) => n.backendNodeId === 11)?.formValue).toBe("inaccessible frame"); expect(controls.find((n) => n.backendNodeId === 11)?.formValue).toBe("inaccessible frame");
expect(controls.find((n) => n.backendNodeId === 21)?.formValue).toBe("second frame"); expect(controls.find((n) => n.backendNodeId === 21)?.formValue).toBe("second frame");
expect(controls.find((n) => n.backendNodeId === 31)?.formValue).toBe("third frame"); expect(controls.find((n) => n.backendNodeId === 31)?.formValue).toBe("third frame");
@@ -146,7 +154,6 @@ describe("form state identity", () => {
it("keeps identical backend ids in different OOPIF targets separate", async () => { it("keeps identical backend ids in different OOPIF targets separate", async () => {
const sendToTarget = vi.fn(async (target, method) => { const sendToTarget = vi.fn(async (target, method) => {
if (method === "DOMSnapshot.captureSnapshot") return snapshot([[{ id: 11, value: "old" }]]);
if (method === "Runtime.evaluate") return reply([[11, state(target.sessionId)]]); if (method === "Runtime.evaluate") return reply([[11, state(target.sessionId)]]);
return {}; return {};
}); });
@@ -155,13 +162,9 @@ describe("form state identity", () => {
}); });
const cdp = { send, sendToTarget } as unknown as CdpRunner; const cdp = { send, sendToTarget } as unknown as CdpRunner;
for (const sessionId of ["left-frame", "right-frame"]) { for (const sessionId of ["left-frame", "right-frame"]) {
const captured = await captureObservationFacts( const nodes = await decodeControls(snapshot([[{ id: 11, value: "old" }]]));
cdpRunnerForTarget(cdp, { tabId: 7, sessionId }), await enrichFormControlStates(cdpRunnerForTarget(cdp, { tabId: 7, sessionId }), 7, nodes);
7, expect(nodes[0].find((n) => n.backendNodeId === 11)?.formValue).toBe(sessionId);
);
expect(captured.documents[0].domNodes.find((n) => n.backendNodeId === 11)?.formValue).toBe(
sessionId,
);
expect(sendToTarget).toHaveBeenCalledWith( expect(sendToTarget).toHaveBeenCalledWith(
{ tabId: 7, sessionId }, { tabId: 7, sessionId },
"Runtime.releaseObjectGroup", "Runtime.releaseObjectGroup",
@@ -177,7 +180,7 @@ describe("form state identity", () => {
"malformed", "malformed",
"invalid-state", "invalid-state",
])("preserves snapshot data when enrichment is %s", async (mode) => { ])("preserves snapshot data when enrichment is %s", async (mode) => {
const { cdp, send } = fakeCdp([[{ id: 11, value: "snapshot value" }]], () => { const { cdp, send, nodes } = await fakeCdp([[{ id: 11, value: "snapshot value" }]], () => {
if (mode === "unsupported") throw new Error("serialization unsupported"); if (mode === "unsupported") throw new Error("serialization unsupported");
if (mode === "legacy") return { result: { value: { controls: [state("wrong value")] } } }; if (mode === "legacy") return { result: { value: { controls: [state("wrong value")] } } };
const result = reply([ const result = reply([
@@ -187,8 +190,8 @@ describe("form state identity", () => {
result.result.deepSerializedValue.value[0].value[1].value = "not json"; result.result.deepSerializedValue.value[0].value[1].value = "not json";
return result; return result;
}); });
const captured = await captureObservationFacts(cdp, 7); expect(await enrichFormControlStates(cdp, 7, nodes)).toBe(mode !== "unsupported");
expect(captured.documents[0].domNodes.find((n) => n.backendNodeId === 11)).toMatchObject({ expect(nodes[0].find((n) => n.backendNodeId === 11)).toMatchObject({
formValue: "snapshot value", formValue: "snapshot value",
formPlaceholder: "snapshot placeholder", formPlaceholder: "snapshot placeholder",
}); });
@@ -197,18 +200,34 @@ describe("form state identity", () => {
it("releases the batch's remote objects when capture is cancelled", async () => { it("releases the batch's remote objects when capture is cancelled", async () => {
const controller = new AbortController(); const controller = new AbortController();
const { cdp, send } = fakeCdp([[{ id: 11, value: "old" }]], () => { const { cdp, send, nodes } = await fakeCdp([[{ id: 11, value: "old" }]], () => {
controller.abort(); controller.abort();
return reply([[11, state("new")]]); return reply([[11, state("new")]]);
}); });
await expect(captureObservationFacts(cdp, 7, controller.signal)).rejects.toMatchObject({ await expect(enrichFormControlStates(cdp, 7, nodes, controller.signal)).rejects.toMatchObject({
name: "AbortError", name: "AbortError",
}); });
expect(send).toHaveBeenCalledWith(7, "Runtime.releaseObjectGroup", expect.anything()); expect(send).toHaveBeenCalledWith(7, "Runtime.releaseObjectGroup", expect.anything());
}); });
it("does not report success when cancelled during object cleanup", async () => {
const controller = new AbortController();
const nodes = await decodeControls(snapshot([[{ id: 11, value: "old" }]]));
const cdp: CdpRunner = {
send: vi.fn(async (_tabId, method) => {
if (method === "Runtime.evaluate") return reply([[11, state("new")]]);
if (method === "Runtime.releaseObjectGroup") controller.abort();
return {};
}) as CdpRunner["send"],
};
await expect(enrichFormControlStates(cdp, 7, nodes, controller.signal)).rejects.toMatchObject({
name: "AbortError",
});
expect(cdp.send).toHaveBeenCalledWith(7, "Runtime.releaseObjectGroup", expect.anything());
});
it("skips entries without node identity while still applying valid entries", async () => { it("skips entries without node identity while still applying valid entries", async () => {
const { cdp } = fakeCdp( const { cdp, nodes } = await fakeCdp(
[ [
[ [
{ id: 11, value: "snapshot value" }, { id: 11, value: "snapshot value" },
@@ -225,24 +244,20 @@ describe("form state identity", () => {
return result; return result;
}, },
); );
const captured = await captureObservationFacts(cdp, 7); await enrichFormControlStates(cdp, 7, nodes);
expect(captured.documents[0].domNodes.find((n) => n.backendNodeId === 11)?.formValue).toBe( expect(nodes[0].find((n) => n.backendNodeId === 11)?.formValue).toBe("snapshot value");
"snapshot value", expect(nodes[0].find((n) => n.backendNodeId === 12)?.formValue).toBe("live value");
);
expect(captured.documents[0].domNodes.find((n) => n.backendNodeId === 12)?.formValue).toBe(
"live value",
);
}); });
it.each([ it.each([
"password", "password",
"text", "text",
])("redacts a sensitive %s control after identity matching", async (type) => { ])("redacts a sensitive %s control after identity matching", async (type) => {
const { cdp } = fakeCdp([[{ id: 11, value: "secret", type }]], () => const { cdp, nodes } = await fakeCdp([[{ id: 11, value: "secret", type }]], () =>
reply([[11, { ...state("secret"), sensitive: true }]]), reply([[11, { ...state("secret"), sensitive: true }]]),
); );
const captured = await captureObservationFacts(cdp, 7); await enrichFormControlStates(cdp, 7, nodes);
const control = captured.documents[0].domNodes.find((n) => n.backendNodeId === 11); const control = nodes[0].find((n) => n.backendNodeId === 11);
expect(control?.formState).toBe("filled"); expect(control?.formState).toBe("filled");
expect(control?.formValue).toBeUndefined(); expect(control?.formValue).toBeUndefined();
expect(control?.formDefaultValue).toBeUndefined(); expect(control?.formDefaultValue).toBeUndefined();
@@ -259,7 +274,7 @@ describe("form state identity", () => {
defaultValue: "", defaultValue: "",
placeholder: "", placeholder: "",
})); }));
const { cdp, send } = fakeCdp([controls], (params) => { const { cdp, send, nodes } = await fakeCdp([controls], (params) => {
const doc = { const doc = {
querySelectorAll: (selector: string) => querySelectorAll: (selector: string) =>
selector === "input,textarea,select" ? elements : [], selector === "input,textarea,select" ? elements : [],
@@ -270,13 +285,9 @@ describe("form state identity", () => {
expect(entries).toHaveLength(250); expect(entries).toHaveLength(250);
return reply(entries.map(([element, json]) => [element.backendNodeId, JSON.parse(json)])); return reply(entries.map(([element, json]) => [element.backendNodeId, JSON.parse(json)]));
}); });
const captured = await captureObservationFacts(cdp, 7); await enrichFormControlStates(cdp, 7, nodes);
expect(captured.documents[0].domNodes.find((n) => n.backendNodeId === 250)?.formValue).toBe( expect(nodes[0].find((n) => n.backendNodeId === 250)?.formValue).toBe("live 250");
"live 250", expect(nodes[0].find((n) => n.backendNodeId === 251)?.formValue).toBe("old");
);
expect(captured.documents[0].domNodes.find((n) => n.backendNodeId === 251)?.formValue).toBe(
"old",
);
expect(send.mock.calls.filter(([, method]) => method === "Runtime.evaluate")).toHaveLength(1); expect(send.mock.calls.filter(([, method]) => method === "Runtime.evaluate")).toHaveLength(1);
expect( expect(
send.mock.calls.filter(([, method]) => method === "Runtime.releaseObjectGroup"), send.mock.calls.filter(([, method]) => method === "Runtime.releaseObjectGroup"),
@@ -159,10 +159,12 @@ describe("buildFrameDocuments", () => {
["child", [domNode("child", 9)]], ["child", [domNode("child", 9)]],
]); ]);
const capture = captured(frameNodes); const capture = captured(frameNodes);
capture.frameOwnerBackendNodeIds = new Map([["child", 7]]);
capture.frameParentIds = new Map([["child", "left"]]);
const frames = ["main", "left", "right", "child"].map((frameId) => ({ const frames = ["main", "left", "right", "child"].map((frameId) => ({
frame: { frameId, target: { tabId: 4 } }, frame: {
frameId,
target: { tabId: 4 },
...(frameId === "child" ? { parentFrameId: "left", ownerBackendNodeId: 7 } : {}),
},
nodes: [], nodes: [],
})); }));
@@ -0,0 +1,40 @@
function captureAbortError(): Error {
const error = new Error("observation aborted");
error.name = "AbortError";
return error;
}
export function isAbortError(error: unknown): boolean {
return (
typeof error === "object" &&
error !== null &&
(error as { name?: string }).name === "AbortError"
);
}
export function throwIfAborted(signal: AbortSignal | undefined): void {
if (signal?.aborted) throw captureAbortError();
}
/** Check at bounded work blocks; yield only after a processing slice is spent.
* Each caller owns its deadline, so unrelated captures do not share state. */
export function createCaptureCheckpoint(signal?: AbortSignal): () => Promise<void> | undefined {
throwIfAborted(signal);
let deadline = performance.now() + 8;
return () => {
throwIfAborted(signal);
if (performance.now() < deadline) return;
return new Promise<void>((resolve) => {
const channel = new MessageChannel();
channel.port1.onmessage = () => {
channel.port1.close();
channel.port2.close();
resolve();
};
channel.port2.postMessage(null);
}).then(() => {
throwIfAborted(signal);
deadline = performance.now() + 8;
});
};
}
@@ -1,24 +1,19 @@
import { import {
buildFrameGraph,
type CdpFrame, type CdpFrame,
type CdpFrameGraph, type CdpFrameGraph,
type CdpFrameTreeNode,
type CdpTarget, type CdpTarget,
cdpTargetKey, cdpTargetKey,
} from "@/browser-driver/frame-graph"; } from "@/browser-driver/frame-graph";
import { cssViewport, GeometryContext } from "../geometry/frame-context"; import { cssViewport, GeometryContext } from "../geometry/frame-context";
import { type CdpRunner, cdpRunnerForTarget, sendToCdpTarget } from "../shared"; import { type CdpRunner, cdpRunnerForTarget } from "../shared";
import { collectOverlayExcludedBackendIds } from "./capture"; import { collectOverlayExcludedBackendIds } from "./capture";
import { readDocumentIdentity, sameDocument } from "./document-identity";
import type { CapturedSceneInput } from "./facts";
import { import {
buildDocumentIndex, isAbortError as isCaptureAbort,
type CaptureIssue, throwIfAborted as throwCaptureAborted,
type DocumentIdentity, } from "./capture-abort";
isCaptureAbort, import { verifyDocumentIdentity } from "./document-identity";
type ObservationFacts, import type { CapturedSceneInput, DocumentIdentity } from "./facts";
throwCaptureAborted, import { buildDocumentIndex, type CaptureIssue, type ObservationFacts } from "./facts";
} from "./facts";
import { enrichFormControlStates } from "./form-capture"; import { enrichFormControlStates } from "./form-capture";
import { import {
buildFrameDocuments, buildFrameDocuments,
@@ -26,67 +21,54 @@ import {
type FrameDocument, type FrameDocument,
type FrameOwnedAxNode, type FrameOwnedAxNode,
} from "./frame-document"; } from "./frame-document";
import { type NormalizedDocument, normalizeSnapshot } from "./normalize"; import { type NormalizedFrameDocument, normalizeSnapshot } from "./normalize";
import { REQUESTED_STYLES, type SnapshotReply, snapshotFrameId } from "./snapshot"; import { describeSnapshotFrames, REQUESTED_STYLES, type SnapshotReply } from "./snapshot";
interface TargetBatch<T extends FrameOwnedAxNode> { interface TargetBatch<T extends FrameOwnedAxNode> {
target: CdpTarget; target: CdpTarget;
frames: CdpFrame[]; frames: CdpFrame[];
before: Map<string, DocumentIdentity>; snapshotAttachmentId?: string;
documents: NormalizedDocument[]; documents: NormalizedFrameDocument[];
ax: FrameAxBatch<T>[]; ax: FrameAxBatch<T>[];
fallbackExcluded: Set<number>; fallbackExcluded: Set<number>;
snapshotFrameIds?: Set<string>;
rootFrameId?: string;
} }
/** A fixed worker pool scoped to one capture; each worker issues at most one /** A fixed worker pool scoped to one capture; each worker issues at most one
* collection request at a time. Geometry has its own measurement-phase bound. */ * collection request at a time. Geometry has its own measurement-phase bound. */
async function collectTargets<T>( async function collectTasks<T>(
items: readonly T[], items: readonly T[],
collect: (item: T) => Promise<void>, collect: (item: T) => Promise<void>,
signal?: AbortSignal, signal?: AbortSignal,
): Promise<void> { ): Promise<void> {
let cursor = 0; let cursor = 0;
let failed = false;
let failure: unknown;
await Promise.all( await Promise.all(
Array.from({ length: Math.min(4, items.length) }, async () => { Array.from({ length: Math.min(4, items.length) }, async () => {
while (cursor < items.length) { try {
throwCaptureAborted(signal); while (cursor < items.length && !failed && !signal?.aborted) {
const item = items[cursor++]; const item = items[cursor++];
await collect(item); await collect(item);
}
} catch (error) {
if (!failed) failure = error;
failed = true;
} }
}), }),
); );
} // A worker's rejection must not let the capture outlive its own task scope.
// In-flight reads and their finally blocks settle before cancellation/failure.
/** When topology discovery is unavailable, snapshot documents still carry throwCaptureAborted(signal);
* explicit target/frame provenance. Only explicit contentDocumentIndex edges if (failed) throw failure;
* establish ancestry; a foreign frame is never renamed to fit a request. */
function snapshotFrames(snapshot: SnapshotReply, target: CdpTarget): CdpFrame[] {
const raw = snapshot.documents ?? [];
const strings = snapshot.strings ?? [];
const frames = raw.map((doc) => {
const frameId = snapshotFrameId(doc, strings);
return frameId ? ({ frameId, target } as CdpFrame) : undefined;
});
for (let i = 0; i < raw.length; i++) {
const edges = raw[i].nodes?.contentDocumentIndex;
const parent = frames[i];
if (!parent || !edges) continue;
for (let e = 0; e < edges.index.length; e++) {
const child = frames[edges.value[e]];
const owner = raw[i].nodes?.backendNodeId?.[edges.index[e]];
if (child && owner !== undefined && child !== parent) {
child.parentFrameId = parent.frameId;
child.ownerBackendNodeId = owner;
}
}
}
return frames.filter((frame): frame is CdpFrame => !!frame);
} }
export async function captureObservationFacts<T extends FrameOwnedAxNode>( export async function captureObservationFacts<T extends FrameOwnedAxNode>(
cdp: CdpRunner, cdp: CdpRunner,
tabId: number, tabId: number,
signal?: AbortSignal, signal?: AbortSignal,
pageUrl?: string,
): Promise<ObservationFacts<T>> { ): Promise<ObservationFacts<T>> {
const startedAt = Date.now(); const startedAt = Date.now();
throwCaptureAborted(signal); throwCaptureAborted(signal);
@@ -96,8 +78,10 @@ export async function captureObservationFacts<T extends FrameOwnedAxNode>(
} catch (error) { } catch (error) {
if (isCaptureAbort(error)) throw error; if (isCaptureAbort(error)) throw error;
} }
throwCaptureAborted(signal);
const geometry = new GeometryContext(cdp, tabId, graph, signal); const geometry = new GeometryContext(cdp, tabId, graph, signal);
const issues: CaptureIssue[] = []; const issues: CaptureIssue[] = [];
const graphFrames = new Map(graph?.frames.map((frame) => [frame.frameId, frame]) ?? []);
const groups = new Map<string, TargetBatch<T>>(); const groups = new Map<string, TargetBatch<T>>();
for (const frame of graph?.frames ?? []) { for (const frame of graph?.frames ?? []) {
const key = cdpTargetKey(frame.target); const key = cdpTargetKey(frame.target);
@@ -106,20 +90,22 @@ export async function captureObservationFacts<T extends FrameOwnedAxNode>(
group = { group = {
target: frame.target, target: frame.target,
frames: [], frames: [],
before: new Map(),
documents: [], documents: [],
ax: [], ax: [],
fallbackExcluded: new Set(), fallbackExcluded: new Set(),
}; };
groups.set(key, group); groups.set(key, group);
} }
group.frames.push(frame); group.frames.push(
!frame.target.sessionId && frame.frameId === graph?.rootFrameId && !frame.url && pageUrl
? { ...frame, url: pageUrl }
: frame,
);
} }
if (!groups.size) if (!groups.has(cdpTargetKey({ tabId })))
groups.set(cdpTargetKey({ tabId }), { groups.set(cdpTargetKey({ tabId }), {
target: { tabId }, target: { tabId },
frames: [], frames: [],
before: new Map(),
documents: [], documents: [],
ax: [], ax: [],
fallbackExcluded: new Set(), fallbackExcluded: new Set(),
@@ -128,196 +114,242 @@ export async function captureObservationFacts<T extends FrameOwnedAxNode>(
const unavailable = (batch: TargetBatch<T>, stage: CaptureIssue["stage"], frameId?: string) => const unavailable = (batch: TargetBatch<T>, stage: CaptureIssue["stage"], frameId?: string) =>
issues.push({ target: batch.target, frameId, stage, reason: "capture-unavailable" }); issues.push({ target: batch.target, frameId, stage, reason: "capture-unavailable" });
let firstFailure: unknown; let firstFailure: unknown;
await collectTargets( const axReady = new Set<TargetBatch<T>>();
await collectTasks(
batches, batches,
async (batch) => { async (batch) => {
const scoped = cdpRunnerForTarget(cdp, batch.target); const scoped = cdpRunnerForTarget(cdp, batch.target);
for (const frame of batch.frames) {
const identity = await readDocumentIdentity(cdp, frame, signal);
if (identity) batch.before.set(frame.frameId, identity);
}
try { try {
throwCaptureAborted(signal); throwCaptureAborted(signal);
await scoped.send(tabId, "DOMSnapshot.enable", {}); await scoped.send(tabId, "DOMSnapshot.enable", {});
throwCaptureAborted(signal); throwCaptureAborted(signal);
batch.snapshotAttachmentId = cdp.getAttachmentId?.(tabId);
const snapshot = await scoped.send<SnapshotReply>(tabId, "DOMSnapshot.captureSnapshot", { const snapshot = await scoped.send<SnapshotReply>(tabId, "DOMSnapshot.captureSnapshot", {
computedStyles: REQUESTED_STYLES, computedStyles: REQUESTED_STYLES,
includePaintOrder: true, includePaintOrder: true,
includeDOMRects: true, includeDOMRects: true,
}); });
throwCaptureAborted(signal); throwCaptureAborted(signal);
if (!graph) batch.frames = snapshotFrames(snapshot, batch.target); const observed = describeSnapshotFrames(snapshot, batch.target, graphFrames, pageUrl);
batch.snapshotFrameIds = observed.ids;
batch.rootFrameId = observed.rootFrameId;
// Keep graph-only frames for AX fallback, never in preference to observed membership.
batch.frames = [
...observed.frames,
...batch.frames.filter((frame) => !observed.ids.has(frame.frameId)),
];
batch.documents = await normalizeSnapshot( batch.documents = await normalizeSnapshot(
snapshot, snapshot,
batch.target, batch.target,
batch.frames, batch.frames.filter((frame) => observed.ids.has(frame.frameId)),
geometry, geometry,
issues, issues,
signal, signal,
observed.rootFrameId,
); );
if ( const formsAvailable = await enrichFormControlStates(
!(await enrichFormControlStates( scoped,
scoped, tabId,
tabId, batch.documents.map((doc) => doc.nodes),
batch.documents.map((doc) => doc.domNodes), signal,
signal, );
)) throwCaptureAborted(signal);
) if (!formsAvailable) unavailable(batch, "forms");
unavailable(batch, "forms");
const present = new Set(batch.documents.map((doc) => doc.frame.frameId)); const present = new Set(batch.documents.map((doc) => doc.frame.frameId));
for (const frame of batch.frames) for (const frame of batch.frames)
if (!present.has(frame.frameId)) unavailable(batch, "dom", frame.frameId); if (!present.has(frame.frameId)) unavailable(batch, "dom", frame.frameId);
} catch (error) { } catch (error) {
throwCaptureAborted(signal);
if (isCaptureAbort(error)) throw error; if (isCaptureAbort(error)) throw error;
firstFailure ??= error; firstFailure ??= error;
unavailable(batch, "dom"); unavailable(batch, "dom");
batch.fallbackExcluded = await collectOverlayExcludedBackendIds(scoped, tabId, signal); batch.fallbackExcluded = await collectOverlayExcludedBackendIds(scoped, tabId, signal);
} }
if (!batch.frames.length && !graph) if (!batch.frames.length && !graph)
batch.frames = [{ frameId: "root", target: batch.target }]; batch.frames = [
{
frameId: "root",
target: batch.target,
...(!batch.target.sessionId && pageUrl ? { url: pageUrl } : {}),
},
];
try { try {
throwCaptureAborted(signal); throwCaptureAborted(signal);
await scoped.send(tabId, "Accessibility.enable", {}); await scoped.send(tabId, "Accessibility.enable", {});
} catch (error) { } catch (error) {
throwCaptureAborted(signal);
if (isCaptureAbort(error)) throw error; if (isCaptureAbort(error)) throw error;
firstFailure ??= error; firstFailure ??= error;
unavailable(batch, "ax"); unavailable(batch, "ax");
return; return;
} }
for (const frame of batch.frames) { axReady.add(batch);
try {
throwCaptureAborted(signal);
const result = await scoped.send<{ nodes?: T[] }>(
tabId,
"Accessibility.getFullAXTree",
frame.frameId === "root" ? {} : { frameId: frame.frameId },
);
batch.ax.push({ frame, nodes: result.nodes ?? [] });
} catch (error) {
if (isCaptureAbort(error)) throw error;
firstFailure ??= error;
unavailable(batch, "ax", frame.frameId);
}
}
}, },
signal, signal,
); );
// Validate after every target's static data and geometry have been collected. // Schedule AX by frame across all ready targets, rather than nesting pools.
const identities = new Map<string, DocumentIdentity>(); // Keep result slots in source order even when CDP replies arrive out of order.
const invalid = new Set<string>(); const axTasks: { batch: TargetBatch<T>; frame: CdpFrame; result?: FrameAxBatch<T> }[] =
await collectTargets( batches.flatMap((batch) =>
batches, axReady.has(batch) ? batch.frames.map((frame) => ({ batch, frame })) : [],
async (batch) => { );
let fresh: Map<string, CdpFrame> | undefined; await collectTasks(
if (batch.before.size) { axTasks,
try { async (task) => {
throwCaptureAborted(signal); const { batch, frame } = task;
const reply = await sendToCdpTarget<{ frameTree?: CdpFrameTreeNode }>( try {
cdp, throwCaptureAborted(signal);
batch.target, const result = await cdpRunnerForTarget(cdp, batch.target).send<{ nodes?: T[] }>(
"Page.getFrameTree", tabId,
{}, "Accessibility.getFullAXTree",
); frame.frameId === "root" ? {} : { frameId: frame.frameId },
const tree = reply.frameTree );
? buildFrameGraph([{ target: batch.target, tree: reply.frameTree }]) throwCaptureAborted(signal);
: null; task.result = { frame, nodes: result.nodes ?? [] };
fresh = new Map(tree?.frames.map((frame) => [frame.frameId, frame])); } catch (error) {
} catch (error) { throwCaptureAborted(signal);
if (isCaptureAbort(error)) throw error; if (isCaptureAbort(error)) throw error;
} firstFailure ??= error;
} unavailable(batch, "ax", frame.frameId);
const docs = new Map(batch.documents.map((doc) => [doc.frame.frameId, doc]));
for (const frame of batch.frames) {
const before = batch.before.get(frame.frameId);
const currentFrame = fresh?.get(frame.frameId);
const after = currentFrame
? await readDocumentIdentity(cdp, currentFrame, signal)
: undefined;
const doc = docs.get(frame.frameId);
const snapshotMismatch =
before &&
doc?.documentElementBackendNodeId !== undefined &&
before.documentElementBackendNodeId !== doc.documentElementBackendNodeId;
if (
before &&
(snapshotMismatch ||
(fresh && !currentFrame) ||
(after && !sameDocument(before, after)) ||
cdp.getAttachmentId?.(tabId) !== before.attachmentId)
) {
invalid.add(frame.frameId);
issues.push({
target: frame.target,
frameId: frame.frameId,
stage: "identity",
reason: "document-changed",
});
} else if (before && !after) {
invalid.add(frame.frameId);
issues.push({
target: frame.target,
frameId: frame.frameId,
stage: "identity",
reason: "identity-unverified",
});
} else if (
before &&
after &&
sameDocument(before, after) &&
(!doc || doc.documentElementBackendNodeId === before.documentElementBackendNodeId)
)
identities.set(frame.frameId, after);
else
issues.push({
target: frame.target,
frameId: frame.frameId,
stage: "identity",
reason: "identity-unverified",
});
} }
}, },
signal, signal,
); );
const allFrames = batches.flatMap((batch) => batch.frames); for (const task of axTasks) if (task.result) task.batch.ax.push(task.result);
const rootFrameId = graph?.rootFrameId ?? allFrames[0]?.frameId ?? "root";
if (invalid.has(rootFrameId)) let viewport = { width: 0, height: 0 };
throw new Error( try {
"observation document identity changed or could not be verified; observe again", const measured = cssViewport(await geometry.layoutMetrics({ tabId }));
); viewport = { width: measured.width, height: measured.height };
// A descendant cannot retain geometry measured through an invalid ancestor. } catch (error) {
const childFrames = new Map<string, string[]>(); if (isCaptureAbort(error)) throw error;
for (const frame of allFrames) }
if (frame.parentFrameId) {
const children = childFrames.get(frame.parentFrameId); // A snapshot claim outranks an old graph hint. Conflicting actual claims
if (children) children.push(frame.frameId); // cannot be resolved by completion order or by overwriting a frameId map.
else childFrames.set(frame.parentFrameId, [frame.frameId]); const claims = new Map<string, string>();
const invalid = new Set<string>();
for (const batch of batches) {
const targetKey = cdpTargetKey(batch.target);
for (const id of batch.snapshotFrameIds ?? []) {
const previous = claims.get(id);
if (previous !== undefined && previous !== targetKey) invalid.add(id);
claims.set(id, targetKey);
} }
const queue = [...invalid]; const accepted = new Set(batch.frames.map((frame) => frame.frameId));
for (let i = 0; i < queue.length; i++) for (const id of batch.snapshotFrameIds ?? []) if (!accepted.has(id)) invalid.add(id);
for (const child of childFrames.get(queue[i]) ?? []) { }
const currentFrames = batches.flatMap((batch) =>
batch.frames.filter(
(frame) =>
!claims.has(frame.frameId) || claims.get(frame.frameId) === cdpTargetKey(batch.target),
),
);
const rootBatch = batches.find((batch) => !batch.target.sessionId);
const rootFrameId =
rootBatch?.rootFrameId ?? graph?.rootFrameId ?? rootBatch?.frames[0]?.frameId ?? "root";
const ownershipInvalid = new Set(invalid);
const identities = new Map<string, DocumentIdentity>();
const selected = new Map(currentFrames.map((frame) => [frame.frameId, frame]));
const checks = batches.flatMap((batch) => {
const docs = new Map(batch.documents.map((doc) => [doc.frame.frameId, doc]));
return batch.frames
.filter(
(frame) =>
!invalid.has(frame.frameId) &&
cdpTargetKey(selected.get(frame.frameId)!.target) === cdpTargetKey(batch.target),
)
.map((frame) => ({ batch, frame, doc: docs.get(frame.frameId) }));
});
await collectTasks(
checks,
async ({ batch, frame, doc }) => {
throwCaptureAborted(signal);
if (!batch.snapshotAttachmentId || doc?.documentElementBackendNodeId === undefined) {
issues.push({
target: frame.target,
frameId: frame.frameId,
stage: "identity",
reason: "identity-unverified",
});
return;
}
const identity: DocumentIdentity = {
attachmentId: batch.snapshotAttachmentId,
target: frame.target,
frameId: frame.frameId,
documentElementBackendNodeId: doc.documentElementBackendNodeId,
};
const verified = await verifyDocumentIdentity(cdp, identity, signal);
const status = cdp.getAttachmentId?.(tabId) === identity.attachmentId ? verified : "changed";
if (status === "current") identities.set(frame.frameId, identity);
else {
invalid.add(frame.frameId);
issues.push({
target: frame.target,
frameId: frame.frameId,
stage: "identity",
reason: status === "changed" ? "document-changed" : "identity-unavailable",
});
}
},
signal,
);
const children = new Map<string, string[]>();
for (const frame of currentFrames) {
if (!frame.parentFrameId) continue;
const siblings = children.get(frame.parentFrameId);
if (siblings) siblings.push(frame.frameId);
else children.set(frame.parentFrameId, [frame.frameId]);
}
const blocked = [...invalid];
for (let i = 0; i < blocked.length; i++) {
for (const child of children.get(blocked[i]) ?? []) {
if (!invalid.has(child)) { if (!invalid.has(child)) {
invalid.add(child); invalid.add(child);
queue.push(child); blocked.push(child);
} }
} }
const frames = allFrames.filter((frame) => !invalid.has(frame.frameId)); }
for (const frame of currentFrames) {
if (ownershipInvalid.has(frame.frameId))
issues.push({
target: frame.target,
frameId: frame.frameId,
stage: "ownership",
reason: "frame-ownership-unresolved",
});
}
if (invalid.has(rootFrameId))
throw new Error(
ownershipInvalid.has(rootFrameId)
? "observation root document ownership is ambiguous"
: "observation document identity changed or could not be verified; observe again",
);
const frames = currentFrames.filter((frame) => !invalid.has(frame.frameId));
const frameById = new Map(frames.map((frame) => [frame.frameId, frame]));
const belongs = (frame: CdpFrame) => {
const current = frameById.get(frame.frameId);
return current && cdpTargetKey(current.target) === cdpTargetKey(frame.target);
};
const decoded = new Map( const decoded = new Map(
batches batches
.flatMap((batch) => batch.documents) .flatMap((batch) => batch.documents)
.filter((doc) => !invalid.has(doc.frame.frameId)) .filter((doc) => belongs(doc.frame))
.map((doc) => [doc.frame.frameId, doc]), .map((doc) => [doc.frame.frameId, doc]),
); );
const ax = batches const ax = batches.flatMap((batch) => batch.ax).filter((batch) => belongs(batch.frame));
.flatMap((batch) => batch.ax)
.filter((batch) => !invalid.has(batch.frame.frameId));
const documents = await buildFrameDocuments( const documents = await buildFrameDocuments(
{ rootFrameId, frames }, { rootFrameId, frames },
ax, ax,
{ {
nodes: [], nodes: [],
rootFrameId, rootFrameId,
frameNodes: new Map([...decoded].map(([id, doc]) => [id, doc.domNodes])), frameNodes: new Map([...decoded].map(([id, doc]) => [id, doc.nodes])),
frameExcludedBackendNodeIds: new Map(
[...decoded].map(([id, doc]) => [id, doc.index.excludedBackendNodeIds]),
),
}, },
signal, signal,
(frame) => (frame) =>
@@ -328,19 +360,18 @@ export async function captureObservationFacts<T extends FrameOwnedAxNode>(
reason: "frame-ownership-unresolved", reason: "frame-ownership-unresolved",
}), }),
); );
let viewport = { width: 0, height: 0 };
try {
const measured = cssViewport(await geometry.layoutMetrics({ tabId }));
viewport = { width: measured.width, height: measured.height };
} catch (error) {
if (isCaptureAbort(error)) throw error;
}
const facts: ObservationFacts<T>["documents"][number][] = []; const facts: ObservationFacts<T>["documents"][number][] = [];
for (const document of documents) { for (const document of documents) {
const doc = decoded.get(document.frameId); const doc = decoded.get(document.frameId);
const index = doc?.index ?? (await buildDocumentIndex([], signal)); const index = doc?.index ?? (await buildDocumentIndex([], signal));
const fallback = groups.get(cdpTargetKey(document.target))?.fallbackExcluded; const fallback = groups.get(cdpTargetKey(document.target))?.fallbackExcluded;
const { domNodes, axNodes, contextScopeId: _scope, ...frame } = document; const {
domNodes,
axNodes,
contextScopeId: _scope,
excludedBackendNodeIds: _excluded,
...frame
} = document;
facts.push({ facts.push({
frame, frame,
identity: identities.get(document.frameId), identity: identities.get(document.frameId),
@@ -355,8 +386,8 @@ export async function captureObservationFacts<T extends FrameOwnedAxNode>(
return { rootFrameId, viewport, documents: facts, issues, startedAt, finishedAt: Date.now() }; return { rootFrameId, viewport, documents: facts, issues, startedAt, finishedAt: Date.now() };
} }
/** Existing semantic consumers get a narrow view; no raw snapshot or identity /** Existing semantic consumers get a narrow view; no raw snapshot
* metadata can leak through recording's allowlisted output DTO. */ * data can leak through recording's allowlisted output DTO. */
export function semanticCapture<T extends FrameOwnedAxNode>( export function semanticCapture<T extends FrameOwnedAxNode>(
facts: ObservationFacts<T>, facts: ObservationFacts<T>,
): { captured: CapturedSceneInput; documents: FrameDocument<T>[] } { ): { captured: CapturedSceneInput; documents: FrameDocument<T>[] } {
@@ -0,0 +1,37 @@
import type { Rect } from "@browser-skill/vom";
export interface CapturedNode {
backendNodeId: number;
parentBackendNodeId: number | null;
frameId?: string;
/** Owning iframe backend node id; `null` for the top-level document. */
ownerFrameBackendNodeId?: number | null;
tag: string;
attrs: Record<string, string>;
/** Top-level viewport-relative CSS px, clipped to the owning frame viewport. */
rect: Rect | null;
/** Frame-local viewport-relative CSS px before top-level projection. */
localRect?: Rect | null;
paintOrder: number;
position: string;
pointerEvents: string;
/**
* computed `cursor`. `cursor: pointer` is the strongest CDP-free signal
* that a non-semantic element (a `<div>`/`<span>` with a click handler)
* is actually an interactive control — used by the adapter to surface
* custom buttons/checkboxes the AX tree drops as `generic`. Optional like
* `textContent`: the live parser always sets it, hand-built fixtures may not.
*/
cursor?: string;
/**
* Whether the live DOM snapshot provides a painted, non-hidden box for this
* node. Semantic resolution uses this only for DOM fallback nodes; AX-backed
* nodes remain authoritative even when they are outside the viewport.
*/
rendered?: boolean;
textContent?: string;
formState?: "empty" | "filled" | "default";
formValue?: string;
formDefaultValue?: string;
formPlaceholder?: string;
}
+3 -2
View File
@@ -3,14 +3,15 @@
import { evaluateHoverTrigger } from "@/lib/hover-trigger-policy"; import { evaluateHoverTrigger } from "@/lib/hover-trigger-policy";
import { OVERLAY_HOST_SELECTOR } from "../../lib/overlay-bridge"; import { OVERLAY_HOST_SELECTOR } from "../../lib/overlay-bridge";
import type { CdpRunner } from "../shared"; import type { CdpRunner } from "../shared";
import type { CapturedNode, CapturedSurfaceProbe } from "./facts"; import type { CapturedNode } from "./capture-types";
import type { CapturedSurfaceProbe } from "./facts";
export type { export type {
CapturedNode, CapturedNode,
CapturedSurfaceProbe, CapturedSurfaceProbe,
} from "./facts"; } from "./facts";
import { isCaptureAbort as isAbortError, throwCaptureAborted as throwIfAborted } from "./facts"; import { isAbortError, throwIfAborted } from "./capture-abort";
import { clearHover, ProbeBudget, waitForHover } from "./hover-perception"; import { clearHover, ProbeBudget, waitForHover } from "./hover-perception";
interface RuntimeEvaluateReply { interface RuntimeEvaluateReply {
@@ -1,65 +1,53 @@
import type { CdpFrame } from "@/browser-driver/frame-graph";
import type { CdpRunner } from "../shared"; import type { CdpRunner } from "../shared";
import { sendToCdpTarget } from "../shared"; import { sendToCdpTarget } from "../shared";
import { type DocumentIdentity, isCaptureAbort, throwCaptureAborted } from "./facts"; import { isAbortError, throwIfAborted } from "./capture-abort";
import type { DocumentIdentity } from "./facts";
/** Fresh, frame-scoped identity read. An isolated world supplies the owning /** Compare the snapshot root with the current root in that exact frame.
* document without querying each node or installing a page observer. */ * Deep serialization supplies the backend ID without a separate describeNode. */
export async function readDocumentIdentity( export async function verifyDocumentIdentity(
cdp: CdpRunner, cdp: CdpRunner,
frame: CdpFrame, identity: DocumentIdentity,
signal?: AbortSignal, signal?: AbortSignal,
): Promise<DocumentIdentity | undefined> { ): Promise<"current" | "changed" | "unavailable"> {
const attachmentId = cdp.getAttachmentId?.(frame.target.tabId); const attached = () => cdp.getAttachmentId?.(identity.target.tabId) === identity.attachmentId;
if (!attachmentId || !frame.loaderId) return undefined; throwIfAborted(signal);
let objectId: string | undefined; if (!attached()) return "changed";
const objectGroup = `bsk-document-identity-${crypto.randomUUID()}`;
const send = <T>(method: string, params: object) => { const send = <T>(method: string, params: object) => {
throwCaptureAborted(signal); throwIfAborted(signal);
return sendToCdpTarget<T>(cdp, frame.target, method, params); return sendToCdpTarget<T>(cdp, identity.target, method, params);
}; };
try { try {
const world = await send<{ executionContextId: number }>("Page.createIsolatedWorld", { const world = await send<{ executionContextId: number }>("Page.createIsolatedWorld", {
frameId: frame.frameId, frameId: identity.frameId,
worldName: "bsk-document-identity", worldName: "bsk-document-identity",
}); });
const element = await send<{ result?: { objectId?: string } }>("Runtime.evaluate", { const reply = await send<{
result?: { deepSerializedValue?: { type: string; value?: { backendNodeId?: number } } };
}>("Runtime.evaluate", {
expression: "document.documentElement", expression: "document.documentElement",
contextId: world.executionContextId, contextId: world.executionContextId,
objectGroup,
serializationOptions: {
serialization: "deep",
additionalParameters: { maxNodeDepth: 0, includeShadowTree: "none" },
},
}); });
objectId = element.result?.objectId; if (!attached()) return "changed";
if (!objectId) return undefined; const root = reply.result?.deepSerializedValue;
const reply = await send<{ node?: { backendNodeId?: number } }>("DOM.describeNode", { if (root?.type === "null") return "changed";
objectId, const id = root?.type === "node" ? root.value?.backendNodeId : undefined;
depth: 0, if (id === undefined) return "unavailable";
}); return id === identity.documentElementBackendNodeId ? "current" : "changed";
const backendNodeId = reply.node?.backendNodeId;
if (!backendNodeId || cdp.getAttachmentId?.(frame.target.tabId) !== attachmentId)
return undefined;
return {
attachmentId,
target: frame.target,
frameId: frame.frameId,
loaderId: frame.loaderId,
documentElementBackendNodeId: backendNodeId,
};
} catch (error) { } catch (error) {
if (isCaptureAbort(error)) throw error; throwIfAborted(signal);
return undefined; if (isAbortError(error)) throw error;
return attached() ? "unavailable" : "changed";
} finally { } finally {
if (objectId) await sendToCdpTarget(cdp, identity.target, "Runtime.releaseObjectGroup", {
await sendToCdpTarget(cdp, frame.target, "Runtime.releaseObject", { objectId }).catch( objectGroup,
() => {}, }).catch(() => {});
); throwIfAborted(signal);
} }
} }
export function sameDocument(a: DocumentIdentity, b: DocumentIdentity): boolean {
return (
a.attachmentId === b.attachmentId &&
a.target.tabId === b.target.tabId &&
a.target.sessionId === b.target.sessionId &&
a.frameId === b.frameId &&
a.loaderId === b.loaderId &&
a.documentElementBackendNodeId === b.documentElementBackendNodeId
);
}
+109 -175
View File
@@ -1,50 +1,107 @@
import type { Rect, Viewport } from "@browser-skill/vom"; import type { Viewport } from "@browser-skill/vom";
import type { CdpFrame, CdpTarget } from "@/browser-driver/frame-graph"; import type { CdpFrame, CdpTarget } from "@/browser-driver/frame-graph";
import { isOverlayHostNode } from "@/lib/overlay-bridge"; import { isOverlayHostNode } from "@/lib/overlay-bridge";
import type { FrameProjectionIssue } from "../geometry/coordinate-types";
import { createCaptureCheckpoint } from "./capture-abort";
import type { CapturedNode } from "./capture-types";
import type { FrameOwnedAxNode } from "./frame-document"; import type { FrameOwnedAxNode } from "./frame-document";
export interface CapturedNode { /** Bounds retain their raw snapshot document layout units until normalization. */
backendNodeId: number; export interface SnapshotLayout {
parentBackendNodeId: number | null; readonly boundsSpace: "snapshot-document-layout";
frameId?: string; bounds?: number[];
/** Owning iframe backend node id; `null` for the top-level document. */ styles: Readonly<Record<string, string>>;
ownerFrameBackendNodeId?: number | null;
tag: string;
attrs: Record<string, string>;
/** Top-level viewport-relative CSS px, clipped to the owning frame viewport. */
rect: Rect | null;
/** Frame-local viewport-relative CSS px before top-level projection. */
localRect?: Rect | null;
paintOrder: number;
position: string;
pointerEvents: string;
/**
* computed `cursor`. `cursor: pointer` is the strongest CDP-free signal
* that a non-semantic element (a `<div>`/`<span>` with a click handler)
* is actually an interactive control — used by the adapter to surface
* custom buttons/checkboxes the AX tree drops as `generic`. Optional like
* `textContent`: the live parser always sets it, hand-built fixtures may not.
*/
cursor?: string;
/**
* Whether the live DOM snapshot provides a painted, non-hidden box for this
* node. Semantic resolution uses this only for DOM fallback nodes; AX-backed
* nodes remain authoritative even when they are outside the viewport.
*/
rendered?: boolean;
textContent?: string;
formState?: "empty" | "filled" | "default";
formValue?: string;
formDefaultValue?: string;
formPlaceholder?: string;
} }
export interface CapturedSurfaceProbe { export interface DecodedNode extends Omit<CapturedNode, "rect" | "localRect" | "rendered"> {
triggerBackendNodeId: number; nodeType?: number;
triggerPoint?: { x: number; y: number }; layout?: SnapshotLayout;
triggerAction: "hover" | "focus" | string; }
subItems: string[];
confidence?: "high" | "medium" | "low"; export interface NodeFacts extends CapturedNode {
nodeType?: number;
layout?: SnapshotLayout;
}
export interface DocumentIndex<T extends DecodedNode = NodeFacts> {
readonly nodes: ReadonlyMap<number, T>;
readonly excludedBackendNodeIds: ReadonlySet<number>;
}
export interface DecodedDocument {
nodes: DecodedNode[];
}
/** All snapshot nodes participate, including document and shadow roots. This
* preserves overlay propagation when the semantic adapter omits those nodes. */
export async function buildDocumentIndex<T extends DecodedNode>(
input: readonly T[],
signal?: AbortSignal,
): Promise<DocumentIndex<T>> {
const checkpoint = createCaptureCheckpoint(signal);
const nodes = new Map<number, T>();
const overlayByNode = new Map<number, boolean>();
const excludedBackendNodeIds = new Set<number>();
for (let i = 0; i < input.length; i++) {
if (i % 256 === 0) {
const pending = checkpoint();
if (pending) await pending;
}
const node = input[i];
nodes.set(node.backendNodeId, node);
}
let work = 0;
for (const node of input) {
if (work++ % 256 === 0) {
const pending = checkpoint();
if (pending) await pending;
}
if (overlayByNode.has(node.backendNodeId)) continue;
const path: DecodedNode[] = [];
const visiting = new Set<number>();
let current: DecodedNode | undefined = node;
let overlay = false;
while (current && !overlayByNode.has(current.backendNodeId)) {
if (work++ % 256 === 0) {
const pending = checkpoint();
if (pending) await pending;
}
if (visiting.has(current.backendNodeId)) {
// Only an overlay inside the cycle may mark the cycle and its descendants.
overlay = path
.slice(path.findIndex((item) => item.backendNodeId === current!.backendNodeId))
.some((item) => isOverlayHostNode(item.tag, Object.keys(item.attrs)));
break;
}
visiting.add(current.backendNodeId);
path.push(current);
if (current.parentBackendNodeId === null) {
current = undefined;
break;
}
current = nodes.get(current.parentBackendNodeId);
}
if (current && overlayByNode.has(current.backendNodeId))
overlay = overlayByNode.get(current.backendNodeId)!;
for (let i = path.length - 1; i >= 0; i--) {
if (work++ % 256 === 0) {
const pending = checkpoint();
if (pending) await pending;
}
const item = path[i];
overlay = overlay || isOverlayHostNode(item.tag, Object.keys(item.attrs));
overlayByNode.set(item.backendNodeId, overlay);
if (overlay) excludedBackendNodeIds.add(item.backendNodeId);
}
}
return { nodes, excludedBackendNodeIds };
}
export interface DocumentIdentity {
attachmentId: string;
target: CdpTarget;
frameId: string;
documentElementBackendNodeId: number;
} }
/** Narrow input shared by the existing semantic scene/hover consumers. */ /** Narrow input shared by the existing semantic scene/hover consumers. */
@@ -55,61 +112,17 @@ export interface CapturedSceneInput {
excludedBackendNodeIds: ReadonlySet<number>; excludedBackendNodeIds: ReadonlySet<number>;
} }
export interface DocumentIdentity {
attachmentId: string;
target: CdpTarget;
frameId: string;
loaderId: string;
documentElementBackendNodeId: number;
}
/** Snapshot client/offset rectangles retain their protocol units. In particular,
* transformed bounds cannot be combined with unscaled client offsets. */
export interface SnapshotLayout {
readonly boundsSpace: "snapshot-document-css";
readonly clientSpace: "unscaled-client-offset-css";
bounds?: number[];
clientRect?: number[];
offsetRect?: number[];
scrollRect?: number[];
styles: Readonly<Record<string, string>>;
}
export interface DecodedNode extends Omit<CapturedNode, "rect" | "localRect" | "rendered"> {
nodeType?: number;
parentMissing?: boolean;
layout?: SnapshotLayout;
}
export interface AncestorState {
complete: boolean;
overlay: boolean;
}
export interface NodeFacts extends CapturedNode {
nodeType?: number;
layout?: SnapshotLayout;
}
export interface DocumentIndex<T extends DecodedNode = NodeFacts> {
readonly nodes: ReadonlyMap<number, T>;
readonly children: ReadonlyMap<number, readonly number[]>;
readonly ancestry: ReadonlyMap<number, AncestorState>;
readonly excludedBackendNodeIds: ReadonlySet<number>;
}
export interface DecodedDocument {
nodes: DecodedNode[];
}
export interface CaptureIssue { export interface CaptureIssue {
/** Owner failure or inherited blockage; absent for other geometry failures. */
projectionIssue?: FrameProjectionIssue;
target: CdpTarget; target: CdpTarget;
frameId?: string; frameId?: string;
stage: "dom" | "ax" | "identity" | "ownership" | "geometry" | "forms"; stage: "dom" | "ax" | "identity" | "ownership" | "geometry" | "forms";
reason: reason:
| "capture-unavailable"
| "document-changed" | "document-changed"
| "identity-unavailable"
| "identity-unverified" | "identity-unverified"
| "capture-unavailable"
| "frame-ownership-unresolved" | "frame-ownership-unresolved"
| "geometry-unavailable"; | "geometry-unavailable";
} }
@@ -131,91 +144,12 @@ export interface ObservationFacts<T extends FrameOwnedAxNode> {
readonly finishedAt: number; readonly finishedAt: number;
} }
export function throwCaptureAborted(signal?: AbortSignal): void { export interface CapturedSurfaceProbe {
if (signal?.aborted) throw new DOMException("observation aborted", "AbortError"); triggerBackendNodeId: number;
triggerPoint?: { x: number; y: number };
triggerAction: "hover" | "focus" | string;
subItems: string[];
confidence?: "high" | "medium" | "low";
} }
export function isCaptureAbort(error: unknown): boolean { export type { CapturedNode } from "./capture-types";
return error instanceof Error && error.name === "AbortError";
}
/** Called only at bounded block boundaries, not once per node. */
export async function captureCheckpoint(signal?: AbortSignal): Promise<void> {
throwCaptureAborted(signal);
await new Promise<void>((resolve) => {
const channel = new MessageChannel();
channel.port1.onmessage = () => {
channel.port1.close();
channel.port2.close();
resolve();
};
channel.port2.postMessage(null);
});
throwCaptureAborted(signal);
}
/** All snapshot nodes participate, including document and shadow roots. This
* preserves ancestry evidence even when the semantic adapter omits those nodes. */
export async function buildDocumentIndex<T extends DecodedNode>(
input: readonly T[],
signal?: AbortSignal,
): Promise<DocumentIndex<T>> {
const nodes = new Map<number, T>();
const children = new Map<number, number[]>();
const ancestry = new Map<number, AncestorState>();
const excludedBackendNodeIds = new Set<number>();
for (let i = 0; i < input.length; i++) {
if (i % 256 === 0) await captureCheckpoint(signal);
const node = input[i];
nodes.set(node.backendNodeId, node);
if (node.parentBackendNodeId !== null) {
const siblings = children.get(node.parentBackendNodeId);
if (siblings) siblings.push(node.backendNodeId);
else children.set(node.parentBackendNodeId, [node.backendNodeId]);
}
}
let work = 0;
for (const node of input) {
if (work++ % 256 === 0) await captureCheckpoint(signal);
if (ancestry.has(node.backendNodeId)) continue;
const path: DecodedNode[] = [];
const visiting = new Set<number>();
let current: DecodedNode | undefined = node;
let state: AncestorState = { complete: true, overlay: false };
while (current && !ancestry.has(current.backendNodeId)) {
if (work++ % 256 === 0) await captureCheckpoint(signal);
if (visiting.has(current.backendNodeId)) {
// No member of a malformed cycle supplies complete ancestry evidence.
state = {
complete: false,
overlay: path
.slice(path.findIndex((item) => item.backendNodeId === current!.backendNodeId))
.some((item) => isOverlayHostNode(item.tag, Object.keys(item.attrs))),
};
break;
}
visiting.add(current.backendNodeId);
path.push(current);
if (current.parentMissing) state = { complete: false, overlay: false };
if (current.parentBackendNodeId === null) {
current = undefined;
break;
}
current = nodes.get(current.parentBackendNodeId);
if (!current) state = { complete: false, overlay: false };
}
if (current && ancestry.has(current.backendNodeId))
state = ancestry.get(current.backendNodeId)!;
for (let i = path.length - 1; i >= 0; i--) {
if (work++ % 256 === 0) await captureCheckpoint(signal);
const item = path[i];
state = {
complete: state.complete,
overlay: state.overlay || isOverlayHostNode(item.tag, Object.keys(item.attrs)),
};
ancestry.set(item.backendNodeId, state);
if (state.overlay) excludedBackendNodeIds.add(item.backendNodeId);
}
}
return { nodes, children, ancestry, excludedBackendNodeIds };
}
+3 -2
View File
@@ -1,6 +1,6 @@
import type { CdpRunner } from "../shared"; import type { CdpRunner } from "../shared";
import type { CapturedNode } from "./facts"; import { isAbortError, throwIfAborted } from "./capture-abort";
import { isCaptureAbort as isAbortError, throwCaptureAborted as throwIfAborted } from "./facts"; import type { CapturedNode } from "./capture-types";
interface RuntimeEvaluateReply { interface RuntimeEvaluateReply {
result?: { deepSerializedValue?: DeepSerializedValue }; result?: { deepSerializedValue?: DeepSerializedValue };
@@ -150,5 +150,6 @@ export async function enrichFormControlStates(
return false; return false;
} finally { } finally {
await cdp.send(tabId, "Runtime.releaseObjectGroup", { objectGroup }).catch(() => undefined); await cdp.send(tabId, "Runtime.releaseObjectGroup", { objectGroup }).catch(() => undefined);
throwIfAborted(signal);
} }
} }
+28 -28
View File
@@ -4,8 +4,8 @@ import {
type CdpTarget, type CdpTarget,
cdpTargetKey, cdpTargetKey,
} from "@/browser-driver/frame-graph"; } from "@/browser-driver/frame-graph";
import type { CapturedNode } from "./facts"; import { createCaptureCheckpoint } from "./capture-abort";
import { captureCheckpoint } from "./facts"; import type { CapturedNode } from "./capture-types";
export interface FrameOwnedAxNode { export interface FrameOwnedAxNode {
nodeId: string; nodeId: string;
@@ -32,8 +32,7 @@ export interface FrameDomInput {
nodes: CapturedNode[]; nodes: CapturedNode[];
rootFrameId?: string; rootFrameId?: string;
frameNodes?: ReadonlyMap<string, CapturedNode[]>; frameNodes?: ReadonlyMap<string, CapturedNode[]>;
frameOwnerBackendNodeIds?: ReadonlyMap<string, number>; frameExcludedBackendNodeIds?: ReadonlyMap<string, ReadonlySet<number>>;
frameParentIds?: ReadonlyMap<string, string>;
} }
interface Ownership { interface Ownership {
@@ -58,23 +57,13 @@ function targetBackendKey(target: CdpTarget, backendNodeId: number): string {
function frameList<T extends FrameOwnedAxNode>( function frameList<T extends FrameOwnedAxNode>(
graph: CdpFrameGraph | null, graph: CdpFrameGraph | null,
batches: FrameAxBatch<T>[], batches: FrameAxBatch<T>[],
captured: FrameDomInput,
): CdpFrame[] { ): CdpFrame[] {
const frames = new Map<string, CdpFrame>(); const frames = new Map<string, CdpFrame>();
for (const frame of graph?.frames ?? []) frames.set(frame.frameId, frame); for (const frame of graph?.frames ?? []) frames.set(frame.frameId, frame);
for (const batch of batches) { for (const batch of batches) {
if (!frames.has(batch.frame.frameId)) frames.set(batch.frame.frameId, batch.frame); if (!frames.has(batch.frame.frameId)) frames.set(batch.frame.frameId, batch.frame);
} }
return [...frames.values()].map((frame) => { return [...frames.values()];
const ownerBackendNodeId =
frame.ownerBackendNodeId ?? captured.frameOwnerBackendNodeIds?.get(frame.frameId);
const parentFrameId = frame.parentFrameId ?? captured.frameParentIds?.get(frame.frameId);
return {
...frame,
...(ownerBackendNodeId !== undefined ? { ownerBackendNodeId } : {}),
...(parentFrameId ? { parentFrameId } : {}),
};
});
} }
export async function buildFrameDocuments<T extends FrameOwnedAxNode>( export async function buildFrameDocuments<T extends FrameOwnedAxNode>(
@@ -84,8 +73,9 @@ export async function buildFrameDocuments<T extends FrameOwnedAxNode>(
signal?: AbortSignal, signal?: AbortSignal,
unresolved?: (frame: CdpFrame) => void, unresolved?: (frame: CdpFrame) => void,
): Promise<FrameDocument<T>[]> { ): Promise<FrameDocument<T>[]> {
const checkpoint = createCaptureCheckpoint(signal);
let work = 0; let work = 0;
const frames = frameList(graph, batches, captured); const frames = frameList(graph, batches);
const frameById = new Map(frames.map((frame) => [frame.frameId, frame])); const frameById = new Map(frames.map((frame) => [frame.frameId, frame]));
const rootFrameId = graph?.rootFrameId ?? captured.rootFrameId ?? frames[0]?.frameId; const rootFrameId = graph?.rootFrameId ?? captured.rootFrameId ?? frames[0]?.frameId;
const domNodesForFrame = (frameId: string): CapturedNode[] => const domNodesForFrame = (frameId: string): CapturedNode[] =>
@@ -93,7 +83,10 @@ export async function buildFrameDocuments<T extends FrameOwnedAxNode>(
const backendOwner = new Map<string, string>(); const backendOwner = new Map<string, string>();
for (const frame of frames) { for (const frame of frames) {
for (const node of domNodesForFrame(frame.frameId)) { for (const node of domNodesForFrame(frame.frameId)) {
if (work++ % 256 === 0) await captureCheckpoint(signal); if (work++ % 256 === 0) {
const pending = checkpoint();
if (pending) await pending;
}
backendOwner.set(targetBackendKey(frame.target, node.backendNodeId), frame.frameId); backendOwner.set(targetBackendKey(frame.target, node.backendNodeId), frame.frameId);
} }
} }
@@ -102,18 +95,24 @@ export async function buildFrameDocuments<T extends FrameOwnedAxNode>(
for (const batch of batches) { for (const batch of batches) {
const nodeById = new Map<string, T>(); const nodeById = new Map<string, T>();
for (const node of batch.nodes) { for (const node of batch.nodes) {
if (work++ % 256 === 0) await captureCheckpoint(signal); if (work++ % 256 === 0) {
const pending = checkpoint();
if (pending) await pending;
}
nodeById.set(node.nodeId, node); nodeById.set(node.nodeId, node);
} }
let reported = false; let reported = false;
const ownershipByNodeId = new Map<string, Ownership>(); const ownershipByNodeId = new Map<string, Ownership>();
const resolveOwnership = async (node: T): Promise<Ownership> => { for (const node of batch.nodes) {
const path: T[] = []; const path: T[] = [];
const visiting = new Set<string>(); const visiting = new Set<string>();
let current: T | undefined = node; let current: T | undefined = node;
let ownership: Ownership = { frameId: batch.frame.frameId, strength: 1 }; let ownership: Ownership = { frameId: batch.frame.frameId, strength: 1 };
while (current) { while (current) {
if (work++ % 256 === 0) await captureCheckpoint(signal); if (work++ % 256 === 0) {
const pending = checkpoint();
if (pending) await pending;
}
const cached = ownershipByNodeId.get(current.nodeId); const cached = ownershipByNodeId.get(current.nodeId);
if (cached) { if (cached) {
ownership = cached; ownership = cached;
@@ -149,17 +148,14 @@ export async function buildFrameDocuments<T extends FrameOwnedAxNode>(
current = current.parentId ? nodeById.get(current.parentId) : undefined; current = current.parentId ? nodeById.get(current.parentId) : undefined;
} }
for (let i = path.length - 1; i >= 0; i--) { for (let i = path.length - 1; i >= 0; i--) {
if (work++ % 256 === 0) await captureCheckpoint(signal); if (work++ % 256 === 0) {
const pending = checkpoint();
if (pending) await pending;
}
ownership = { frameId: ownership.frameId, strength: Math.min(2, ownership.strength) }; ownership = { frameId: ownership.frameId, strength: Math.min(2, ownership.strength) };
ownershipByNodeId.set(path[i].nodeId, ownership); ownershipByNodeId.set(path[i].nodeId, ownership);
} }
if (!ownershipByNodeId.has(node.nodeId)) ownershipByNodeId.set(node.nodeId, ownership); if (!ownershipByNodeId.has(node.nodeId)) ownershipByNodeId.set(node.nodeId, ownership);
return ownership;
};
for (const node of batch.nodes) {
if (work++ % 256 === 0) await captureCheckpoint(signal);
const ownership = await resolveOwnership(node);
if (!ownership.frameId) continue; if (!ownership.frameId) continue;
const key = targetNodeKey(batch.frame.target, node.nodeId); const key = targetNodeKey(batch.frame.target, node.nodeId);
const existing = candidates.get(key); const existing = candidates.get(key);
@@ -174,7 +170,10 @@ export async function buildFrameDocuments<T extends FrameOwnedAxNode>(
); );
const axNodesByFrame = new Map<string, T[]>(); const axNodesByFrame = new Map<string, T[]>();
for (const candidate of candidates.values()) { for (const candidate of candidates.values()) {
if (work++ % 256 === 0) await captureCheckpoint(signal); if (work++ % 256 === 0) {
const pending = checkpoint();
if (pending) await pending;
}
const { node, target, ownership } = candidate; const { node, target, ownership } = candidate;
const parentFrameId = node.parentId const parentFrameId = node.parentId
? ownershipByTargetNode.get(targetNodeKey(target, node.parentId)) ? ownershipByTargetNode.get(targetNodeKey(target, node.parentId))
@@ -200,5 +199,6 @@ export async function buildFrameDocuments<T extends FrameOwnedAxNode>(
contextScopeId: frame.frameId, contextScopeId: frame.frameId,
axNodes: axNodesByFrame.get(frame.frameId) ?? [], axNodes: axNodesByFrame.get(frame.frameId) ?? [],
domNodes: domNodesForFrame(frame.frameId), domNodes: domNodesForFrame(frame.frameId),
excludedBackendNodeIds: captured.frameExcludedBackendNodeIds?.get(frame.frameId),
})); }));
} }
+241 -100
View File
@@ -1,21 +1,24 @@
import type { Viewport } from "@browser-skill/vom"; import type { Viewport } from "@browser-skill/vom";
import type { CdpFrame, CdpTarget } from "@/browser-driver/frame-graph"; import { type CdpFrame, type CdpTarget, cdpTargetKey } from "@/browser-driver/frame-graph";
import { type GeometryProjection, projectRectToViewport } from "../geometry"; import { type GeometryProjection, projectRectToViewport } from "../geometry";
import { import {
type FrameProjectionState,
projectSnapshotRect, projectSnapshotRect,
type SnapshotProjection, type SnapshotCoordinates,
snapshotCoordinates,
snapshotViewportRect, snapshotViewportRect,
} from "../geometry/coordinate-types"; } from "../geometry/coordinate-types";
import { cssViewport, GeometryContext } from "../geometry/frame-context";
import type { CapturedNode } from "./facts";
import { import {
buildDocumentIndex, GeometryContext,
type CaptureIssue, type LayoutMetrics,
captureCheckpoint, snapshotLayoutScale,
type DocumentIndex, } from "../geometry/frame-context";
isCaptureAbort, import {
type NodeFacts, createCaptureCheckpoint,
} from "./facts"; isAbortError as isCaptureAbort,
throwIfAborted as throwCaptureAborted,
} from "./capture-abort";
import { buildDocumentIndex, type CaptureIssue, type DocumentIndex, type NodeFacts } from "./facts";
import { import {
decodeDocument, decodeDocument,
type SnapshotDocument, type SnapshotDocument,
@@ -23,13 +26,92 @@ import {
snapshotFrameId, snapshotFrameId,
} from "./snapshot"; } from "./snapshot";
export interface FrameContext {
frameId?: string;
ownerFrameBackendNodeId: number | null;
projection: FrameProjectionState | null;
targetProjection?: GeometryProjection | null;
target: CdpTarget;
coordinates: SnapshotCoordinates | null;
}
export interface NormalizedDocument { export interface NormalizedDocument {
nodes: NodeFacts[];
index: DocumentIndex; index: DocumentIndex;
frame: CdpFrame;
domNodes: CapturedNode[];
documentElementBackendNodeId?: number; documentElementBackendNodeId?: number;
} }
/** Interpret one document using a supplied projection. No live reads or frame scheduling. */
export async function normalizeDocument(
doc: SnapshotDocument,
strings: string[],
context: FrameContext,
signal?: AbortSignal,
): Promise<NormalizedDocument> {
const decoded = await decodeDocument(doc, strings, signal);
const checkpoint = createCaptureCheckpoint(signal);
const nodes: NodeFacts[] = [];
for (let i = 0; i < decoded.nodes.length; i++) {
if (i % 256 === 0) {
const pending = checkpoint();
if (pending) await pending;
}
const node = decoded.nodes[i];
const bounds = node.layout?.bounds ?? [];
const input = snapshotViewportRect(
bounds,
{ target: context.target, frameId: context.frameId },
context.coordinates,
);
const local = input?.rect;
let rect =
input && context.projection?.status === "available"
? projectSnapshotRect(input, context.projection.projection)
: null;
if (rect && context.targetProjection !== undefined)
rect = context.targetProjection
? projectRectToViewport(
{ x: rect.x, y: rect.y, w: rect.width, h: rect.height },
context.targetProjection,
)
: null;
const visibility = node.layout?.styles.visibility || "visible";
const opacity = node.layout?.styles.opacity || "1";
nodes.push({
...node,
...(context.frameId ? { frameId: context.frameId } : {}),
ownerFrameBackendNodeId: context.ownerFrameBackendNodeId,
localRect: local ? { x: local.x, y: local.y, w: local.width, h: local.height } : null,
rect: rect ? { x: rect.x, y: rect.y, w: rect.width, h: rect.height } : null,
rendered:
bounds.length >= 4 &&
bounds.slice(0, 4).every(Number.isFinite) &&
bounds[2] > 0 &&
bounds[3] > 0 &&
visibility !== "hidden" &&
visibility !== "collapse" &&
(Number.parseFloat(opacity) || 0) > 0,
});
}
const index = await buildDocumentIndex(nodes, signal);
return {
nodes: nodes.filter(
(node) => !node.tag.startsWith("#") && !index.excludedBackendNodeIds.has(node.backendNodeId),
),
index,
documentElementBackendNodeId: nodes.find(
(node) =>
node.nodeType === 1 &&
node.parentBackendNodeId !== null &&
index.nodes.get(node.parentBackendNodeId)?.nodeType === 9,
)?.backendNodeId,
};
}
export interface NormalizedFrameDocument extends NormalizedDocument {
frame: CdpFrame;
}
/** One target's snapshot. Source ownership is checked before interpreting any /** One target's snapshot. Source ownership is checked before interpreting any
* coordinates. Snapshot and live quad coordinates are deliberately not conflated. */ * coordinates. Snapshot and live quad coordinates are deliberately not conflated. */
export async function normalizeSnapshot( export async function normalizeSnapshot(
@@ -39,9 +121,18 @@ export async function normalizeSnapshot(
geometry: GeometryContext, geometry: GeometryContext,
issues: CaptureIssue[], issues: CaptureIssue[],
signal?: AbortSignal, signal?: AbortSignal,
): Promise<NormalizedDocument[]> { rootFrameId?: string,
): Promise<NormalizedFrameDocument[]> {
const strings = snapshot.strings ?? []; const strings = snapshot.strings ?? [];
const raw = snapshot.documents ?? []; const raw = snapshot.documents ?? [];
const ownerIds = new Set<number>();
for (const document of raw) {
for (const index of document.nodes?.contentDocumentIndex?.index ?? []) {
const id = document.nodes?.backendNodeId?.[index];
if (id !== undefined) ownerIds.add(id);
}
}
geometry.registerSnapshotOwners(target, ownerIds);
const frameById = new Map(frames.map((frame) => [frame.frameId, frame])); const frameById = new Map(frames.map((frame) => [frame.frameId, frame]));
const sources = new Map<string, SnapshotDocument>(); const sources = new Map<string, SnapshotDocument>();
for (const doc of raw) { for (const doc of raw) {
@@ -57,17 +148,20 @@ export async function normalizeSnapshot(
} }
sources.set(id, doc); sources.set(id, doc);
} }
let viewport: Viewport = { width: 0, height: 0 }; let metrics: LayoutMetrics = {};
let scrollX = 0,
scrollY = 0;
try { try {
const measured = cssViewport(await geometry.layoutMetrics(target)); metrics = await geometry.layoutMetrics(target);
viewport = { width: measured.width, height: measured.height };
scrollX = measured.scrollX;
scrollY = measured.scrollY;
} catch (error) { } catch (error) {
if (isCaptureAbort(error)) throw error; if (isCaptureAbort(error)) throw error;
} }
const viewport: Viewport = {
width: metrics.cssLayoutViewport?.clientWidth ?? 0,
height: metrics.cssLayoutViewport?.clientHeight ?? 0,
};
const layoutUnitsPerCssPixel = snapshotLayoutScale(metrics);
const viewportAvailable = [viewport.width, viewport.height].every(
(size) => Number.isFinite(size) && size > 0,
);
const children = new Map<string, CdpFrame[]>(); const children = new Map<string, CdpFrame[]>();
const pending: CdpFrame[] = []; const pending: CdpFrame[] = [];
@@ -79,106 +173,153 @@ export async function normalizeSnapshot(
else children.set(frame.parentFrameId, [frame]); else children.set(frame.parentFrameId, [frame]);
} }
} }
const projections = new Map<string, SnapshotProjection | null>(); const projections = new Map<string, FrameProjectionState | null>();
const result: NormalizedDocument[] = []; const result: NormalizedFrameDocument[] = [];
let targetProjection: GeometryProjection | null = null; let targetProjection: GeometryProjection | null = null;
const targetRoot = pending[0]; const targetRoot = rootFrameId ? frameById.get(rootFrameId) : undefined;
if (target.sessionId && targetRoot) { if (target.sessionId && targetRoot) {
try { try {
targetProjection = await geometry.targetProjection(targetRoot.frameId); const known = await geometry.frame(targetRoot.frameId);
if (known && cdpTargetKey(known.target) === cdpTargetKey(target))
targetProjection = await geometry.targetProjection(targetRoot.frameId);
} catch (error) { } catch (error) {
if (isCaptureAbort(error)) throw error; if (isCaptureAbort(error)) throw error;
} }
} }
for (let cursor = 0; cursor < pending.length; cursor++) { const checkpoint = createCaptureCheckpoint(signal);
if (cursor % 256 === 0) await captureCheckpoint(signal); const visited = new Set<string>();
const remaining = frames[Symbol.iterator]();
for (let cursor = 0; visited.size < frames.length; cursor++) {
if (cursor >= pending.length) {
let next = remaining.next();
while (!next.done && visited.has(next.value.frameId)) next = remaining.next();
if (next.done) break;
pending.push(next.value);
}
if (cursor % 256 === 0) {
const pending = checkpoint();
if (pending) await pending;
}
const frame = pending[cursor]; const frame = pending[cursor];
if (visited.has(frame.frameId)) continue;
visited.add(frame.frameId);
for (const child of children.get(frame.frameId) ?? []) pending.push(child); for (const child of children.get(frame.frameId) ?? []) pending.push(child);
const doc = sources.get(frame.frameId); const doc = sources.get(frame.frameId);
if (!doc) continue; if (!doc) continue;
const source = { target, frameId: frame.frameId }; const source = { target, frameId: frame.frameId };
const parent = frame.parentFrameId ? frameById.get(frame.parentFrameId) : undefined; const coordinates = snapshotCoordinates(
let projection: SnapshotProjection | null = null; doc,
if (!parent) layoutUnitsPerCssPixel,
projection = { source, geometry: { sourceClips: [], edges: [], topViewport: viewport } }; frame.frameId === rootFrameId
else { ? { x: metrics.cssLayoutViewport?.pageX, y: metrics.cssLayoutViewport?.pageY }
const parentProjection = projections.get(parent.frameId)?.geometry; : undefined,
if (parentProjection && frame.ownerBackendNodeId !== undefined) { );
const edge = parentProjection.edges[0]; let state: FrameProjectionState | null =
const clips = edge frame.frameId === rootFrameId
? [edge.destinationQuad, ...(edge.destinationClips ?? [])] ? viewportAvailable && coordinates
: parentProjection.sourceClips; ? {
try { status: "available",
projection = await geometry.snapshotProjection( projection: {
source, source,
frame.ownerBackendNodeId, geometry: { sourceClips: [], edges: [], topViewport: viewport },
clips, },
viewport, }
); : { status: "unavailable", source, reason: "snapshot-coordinates-unavailable" }
} catch (error) { : (projections.get(frame.frameId) ?? null);
if (isCaptureAbort(error)) throw error; if (!state && frame.frameId !== rootFrameId)
} issues.push({
} target,
} frameId: frame.frameId,
projections.set(frame.frameId, projection); stage: "ownership",
reason: "frame-ownership-unresolved",
});
if (!coordinates && (!state || state.status === "available"))
state = { status: "unavailable", source, reason: "snapshot-coordinates-unavailable" };
projections.set(frame.frameId, state);
const projection = state?.status === "available" ? state.projection : null;
if (!projection || (target.sessionId && !targetProjection)) if (!projection || (target.sessionId && !targetProjection))
issues.push({ issues.push({
target, target,
frameId: frame.frameId, frameId: frame.frameId,
stage: "geometry", stage: "geometry",
reason: "geometry-unavailable", reason: "geometry-unavailable",
...(state && state.status !== "available" ? { projectionIssue: state } : {}),
}); });
const decoded = await decodeDocument(doc, strings, signal);
const nodes: NodeFacts[] = []; // Only direct siblings share this pool. Keep normalization in its existing
for (let i = 0; i < decoded.nodes.length; i++) { // breadth-first order, independent of owner read completion order.
if (i % 256 === 0) await captureCheckpoint(signal); const siblings = (children.get(frame.frameId) ?? []).filter((child) =>
const node = decoded.nodes[i]; sources.has(child.frameId),
const input = snapshotViewportRect(node.layout?.bounds ?? [], source, { );
x: doc.scrollOffsetX ?? (parent ? 0 : scrollX), const edge = projection?.geometry.edges[0];
y: doc.scrollOffsetY ?? (parent ? 0 : scrollY), const clips = edge
}); ? [edge.destinationQuad, ...(edge.destinationClips ?? [])]
const local = input?.rect; : (projection?.geometry.sourceClips ?? []);
let rect = input && projection ? projectSnapshotRect(input, projection) : null; let next = 0;
if (rect && target.sessionId) let failure: unknown;
rect = targetProjection await Promise.all(
? projectRectToViewport( Array.from({ length: Math.min(4, siblings.length) }, async () => {
{ x: rect.x, y: rect.y, w: rect.width, h: rect.height }, while (next < siblings.length && !signal?.aborted && !failure) {
targetProjection, const child = siblings[next++];
) const childSource = { target, frameId: child.frameId };
: null; if (!projection) {
const styles = node.layout?.styles; projections.set(
const visibility = styles?.visibility || "visible"; child.frameId,
const opacity = styles?.opacity || "1"; state && state.status !== "available"
nodes.push({ ? {
...node, status: "blocked",
source: childSource,
cause: state.status === "blocked" ? state.cause : state,
}
: null,
);
continue;
}
if (child.ownerBackendNodeId === undefined) {
projections.set(child.frameId, null);
continue;
}
if (!snapshotCoordinates(sources.get(child.frameId)!, layoutUnitsPerCssPixel)) {
projections.set(child.frameId, {
status: "unavailable",
source: childSource,
reason: "snapshot-coordinates-unavailable",
});
continue;
}
try {
projections.set(
child.frameId,
await geometry.snapshotProjection(
childSource,
child.ownerBackendNodeId,
clips,
viewport,
),
);
} catch (error) {
failure ??= error;
}
}
}),
);
// Join active reads and their cleanup before returning cancellation.
if (failure) throw failure;
throwCaptureAborted(signal);
const normalized = await normalizeDocument(
doc,
strings,
{
frameId: frame.frameId, frameId: frame.frameId,
ownerFrameBackendNodeId: frame.ownerBackendNodeId ?? null, ownerFrameBackendNodeId: frame.ownerBackendNodeId ?? null,
localRect: local ? { x: local.x, y: local.y, w: local.width, h: local.height } : null, projection: state,
rect: rect ? { x: rect.x, y: rect.y, w: rect.width, h: rect.height } : null, target,
rendered: ...(target.sessionId ? { targetProjection } : {}),
!!local && coordinates,
visibility !== "hidden" && },
visibility !== "collapse" && signal,
(Number.parseFloat(opacity) || 0) > 0, );
}); result.push({ ...normalized, frame });
}
const index = await buildDocumentIndex(nodes, signal);
const domNodes = nodes.filter((node) => !node.tag.startsWith("#"));
result.push({
index,
frame,
domNodes,
documentElementBackendNodeId: nodes.find(
(node) =>
node.nodeType === 1 &&
node.parentBackendNodeId !== null &&
index.nodes.get(node.parentBackendNodeId)?.nodeType === 9,
)?.backendNodeId,
});
}
for (const frameId of sources.keys()) {
if (!projections.has(frameId))
issues.push({ target, frameId, stage: "ownership", reason: "frame-ownership-unresolved" });
} }
return result; return result;
} }
+85 -24
View File
@@ -1,18 +1,13 @@
import type { CapturedNode } from "./facts"; import { type CdpFrame, type CdpTarget, cdpTargetKey } from "@/browser-driver/frame-graph";
import { captureCheckpoint, type DecodedDocument, type DecodedNode } from "./facts"; import { createCaptureCheckpoint } from "./capture-abort";
import type { CapturedNode } from "./capture-types";
import type { DecodedDocument, DecodedNode } from "./facts";
export const REQUESTED_STYLES = [ export const REQUESTED_STYLES = [
"position", "position",
"pointer-events", "pointer-events",
"cursor", "cursor",
"visibility", "visibility",
"opacity", "opacity",
"display",
"overflow-x",
"overflow-y",
"transform",
"zoom",
"clip-path",
"mask-image",
] as const; ] as const;
const STYLE_COL = Object.fromEntries( const STYLE_COL = Object.fromEntries(
REQUESTED_STYLES.map((name, index) => [name, index]), REQUESTED_STYLES.map((name, index) => [name, index]),
@@ -29,6 +24,7 @@ interface RareBooleanData {
export interface SnapshotDocument { export interface SnapshotDocument {
frameId?: string | number; frameId?: string | number;
documentURL?: number;
scrollOffsetX?: number; scrollOffsetX?: number;
scrollOffsetY?: number; scrollOffsetY?: number;
nodes?: { nodes?: {
@@ -54,9 +50,6 @@ export interface SnapshotDocument {
styles?: number[][]; styles?: number[][];
bounds?: number[][]; bounds?: number[][];
paintOrders?: number[]; paintOrders?: number[];
clientRects?: number[][];
offsetRects?: number[][];
scrollRects?: number[][];
}; };
} }
@@ -95,7 +88,7 @@ function str(strings: string[], idx: number | undefined): string {
return strings[idx] ?? ""; return strings[idx] ?? "";
} }
function sparseIndexMap(sparse: SparseArray | undefined): Map<number, number> { export function sparseIndexMap(sparse: SparseArray | undefined): Map<number, number> {
const out = new Map<number, number>(); const out = new Map<number, number>();
if (!sparse?.index || !sparse.value) return out; if (!sparse?.index || !sparse.value) return out;
for (let i = 0; i < sparse.index.length; i++) { for (let i = 0; i < sparse.index.length; i++) {
@@ -111,6 +104,7 @@ export async function decodeDocument(
strings: string[], strings: string[],
signal?: AbortSignal, signal?: AbortSignal,
): Promise<DecodedDocument> { ): Promise<DecodedDocument> {
const checkpoint = createCaptureCheckpoint(signal);
const dn = doc.nodes; const dn = doc.nodes;
const dl = doc.layout; const dl = doc.layout;
if (!dn?.backendNodeId) { if (!dn?.backendNodeId) {
@@ -120,7 +114,10 @@ export async function decodeDocument(
const count = dn.backendNodeId.length; const count = dn.backendNodeId.length;
const layoutByNode = new Map<number, number>(); const layoutByNode = new Map<number, number>();
for (let i = 0; i < (dl?.nodeIndex?.length ?? 0); i++) { for (let i = 0; i < (dl?.nodeIndex?.length ?? 0); i++) {
if (i % 256 === 0) await captureCheckpoint(signal); if (i % 256 === 0) {
const pending = checkpoint();
if (pending) await pending;
}
layoutByNode.set(dl!.nodeIndex![i], i); layoutByNode.set(dl!.nodeIndex![i], i);
} }
@@ -135,7 +132,10 @@ export async function decodeDocument(
const nodeTextContent = new Map<number, string[]>(); const nodeTextContent = new Map<number, string[]>();
if (dn.nodeValue) { if (dn.nodeValue) {
for (let n = 0; n < count; n++) { for (let n = 0; n < count; n++) {
if (n % 256 === 0) await captureCheckpoint(signal); if (n % 256 === 0) {
const pending = checkpoint();
if (pending) await pending;
}
const nvIdx = dn.nodeValue[n] ?? -1; const nvIdx = dn.nodeValue[n] ?? -1;
if (nvIdx < 0) continue; if (nvIdx < 0) continue;
const text = str(strings, nvIdx).trim(); const text = str(strings, nvIdx).trim();
@@ -152,7 +152,10 @@ export async function decodeDocument(
// Decode fields without assigning a coordinate projection or semantic policy. // Decode fields without assigning a coordinate projection or semantic policy.
const nodes: DecodedNode[] = []; const nodes: DecodedNode[] = [];
for (let n = 0; n < count; n++) { for (let n = 0; n < count; n++) {
if (n % 256 === 0) await captureCheckpoint(signal); if (n % 256 === 0) {
const pending = checkpoint();
if (pending) await pending;
}
const backendNodeId = dn.backendNodeId[n]; const backendNodeId = dn.backendNodeId[n];
const parentIdx = dn.parentIndex?.[n] ?? -1; const parentIdx = dn.parentIndex?.[n] ?? -1;
const parentBackendNodeId = parentIdx >= 0 ? (dn.backendNodeId[parentIdx] ?? null) : null; const parentBackendNodeId = parentIdx >= 0 ? (dn.backendNodeId[parentIdx] ?? null) : null;
@@ -172,12 +175,8 @@ export async function decodeDocument(
li === undefined li === undefined
? undefined ? undefined
: { : {
boundsSpace: "snapshot-document-css" as const, boundsSpace: "snapshot-document-layout" as const,
clientSpace: "unscaled-client-offset-css" as const,
bounds: dl?.bounds?.[li], bounds: dl?.bounds?.[li],
clientRect: dl?.clientRects?.[li],
offsetRect: dl?.offsetRects?.[li],
scrollRect: dl?.scrollRects?.[li],
styles, styles,
}; };
@@ -201,9 +200,6 @@ export async function decodeDocument(
backendNodeId, backendNodeId,
nodeType: dn.nodeType?.[n], nodeType: dn.nodeType?.[n],
parentBackendNodeId, parentBackendNodeId,
...(parentIdx >= 0 && dn.backendNodeId[parentIdx] === undefined
? { parentMissing: true }
: {}),
tag, tag,
attrs, attrs,
@@ -228,3 +224,68 @@ export async function decodeDocument(
} }
return { nodes }; return { nodes };
} }
/** Snapshot membership and same-target edges are current response evidence.
* The earlier graph only supplies compatible metadata and cross-target edges. */
export function describeSnapshotFrames(
snapshot: SnapshotReply,
target: CdpTarget,
hintById: ReadonlyMap<string, CdpFrame>,
pageUrl?: string,
): { frames: CdpFrame[]; ids: Set<string>; rootFrameId?: string } {
const raw = snapshot.documents ?? [];
const strings = snapshot.strings ?? [];
const ids = new Set<string>();
const invalid = new Set<string>();
const frames = raw.map((doc) => {
const frameId = snapshotFrameId(doc, strings);
if (!frameId) return undefined;
if (ids.has(frameId)) invalid.add(frameId);
ids.add(frameId);
const hint = hintById.get(frameId);
const compatible =
hint && cdpTargetKey(hint.target) === cdpTargetKey(target) ? hint : undefined;
const url =
(doc.documentURL === undefined ? undefined : strings[doc.documentURL]) || compatible?.url;
return { frameId, target, ...(url ? { url } : {}) } as CdpFrame;
});
const root = frames[0];
if (root) {
const hint = hintById.get(root.frameId);
const parent = hint?.parentFrameId ? hintById.get(hint.parentFrameId) : undefined;
if (
target.sessionId &&
hint &&
cdpTargetKey(hint.target) === cdpTargetKey(target) &&
parent &&
cdpTargetKey(parent.target) !== cdpTargetKey(target)
) {
root.parentFrameId = parent.frameId;
root.ownerBackendNodeId = hint.ownerBackendNodeId;
}
if (!target.sessionId && !root.url && pageUrl) root.url = pageUrl;
}
const owned = new Set<string>();
for (let i = 0; i < raw.length; i++) {
const edges = raw[i].nodes?.contentDocumentIndex;
const parent = frames[i];
if (!parent || !edges) continue;
for (let e = 0; e < edges.index.length; e++) {
const child = frames[edges.value[e]];
const owner = raw[i].nodes?.backendNodeId?.[edges.index[e]];
if (!child || owner === undefined) continue;
if (child === root || child === parent || owned.has(child.frameId)) {
invalid.add(child.frameId);
continue;
}
owned.add(child.frameId);
child.parentFrameId = parent.frameId;
child.ownerBackendNodeId = owner;
}
}
return {
frames: frames.filter((frame): frame is CdpFrame => !!frame && !invalid.has(frame.frameId)),
ids,
rootFrameId: root?.frameId,
};
}
+103 -6
View File
@@ -73,6 +73,14 @@ where
P: Serialize + Send + 'static, P: Serialize + Send + 'static,
R: DeserializeOwned + Send + 'static, R: DeserializeOwned + Send + 'static,
{ {
let parent_cancel = stdin_cancellation();
if parent_cancel.as_ref().is_some_and(|rx| *rx.borrow()) {
return Err(CliError::from_rpc(RpcError {
code: ErrorCode::Cancelled,
message: "parent closed the cancellation pipe".into(),
data: Some(serde_json::json!({"reason": "cancelled_before_dispatch"})),
}));
}
let rpc_id: RpcId = format!("{}-{}", rpc_id_prefix, random_short_id()); let rpc_id: RpcId = format!("{}-{}", rpc_id_prefix, random_short_id());
let mut client = IpcClient::connect(sock.clone()).await?; let mut client = IpcClient::connect(sock.clone()).await?;
let rpc_id_for_call = rpc_id.clone(); let rpc_id_for_call = rpc_id.clone();
@@ -88,16 +96,16 @@ where
let outcome = tokio::select! { let outcome = tokio::select! {
biased; biased;
res = &mut main_fut => res?, res = &mut main_fut => res?,
sig = wait_for_sigint() => { sig = wait_for_cancel(parent_cancel) => {
sig.context("install SIGINT handler").map_err(CliError::Local)?; sig.context("wait for cancellation").map_err(CliError::Local)?;
debug!(rpc_id = %rpc_id_for_cancel, "SIGINT: forwarding cancel to daemon"); debug!(rpc_id = %rpc_id_for_cancel, "forwarding cancel to daemon");
let _ = send_cancel(&sock, &rpc_id_for_cancel).await; let _ = send_cancel(&sock, &rpc_id_for_cancel).await;
match tokio::time::timeout(CANCEL_RESPONSE_GRACE, &mut main_fut).await { match tokio::time::timeout(CANCEL_RESPONSE_GRACE, &mut main_fut).await {
Ok(res) => res?, Ok(res) => res?,
Err(_) => { Err(_) => {
debug!( debug!(
rpc_id = %rpc_id_for_cancel, rpc_id = %rpc_id_for_cancel,
"SIGINT: cancel grace elapsed; synthesising cancelled error" "cancel grace elapsed; synthesising cancelled error"
); );
return Err(CliError::from_rpc(RpcError { return Err(CliError::from_rpc(RpcError {
code: ErrorCode::Cancelled, code: ErrorCode::Cancelled,
@@ -112,6 +120,46 @@ where
outcome.map_err(CliError::from_rpc) outcome.map_err(CliError::from_rpc)
} }
/// Release caller-owned staging even after cancellation. This deliberately
/// exposes only transfer.release, not a general bypass for business RPCs.
/// The entire batch, including connection setup, shares one bounded budget.
pub fn release_transfers<'a>(
sock: &Path,
transfer_ids: impl IntoIterator<Item = &'a str>,
) -> Result<(), CliError> {
use bsk_protocol::tools::{TransferIdParams, TransferReleaseResult};
const CLEANUP_TIMEOUT: Duration = Duration::from_secs(5);
let ids: Vec<_> = transfer_ids.into_iter().collect();
if ids.is_empty() {
return Ok(());
}
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.context("build runtime for transfer cleanup")?;
rt.block_on(async {
tokio::time::timeout(CLEANUP_TIMEOUT, async {
let mut client = IpcClient::connect(sock).await?;
for id in ids {
client
.call::<_, TransferReleaseResult>(
"transfer-release",
Method::TransferRelease,
Some(TransferIdParams {
transfer_id: id.to_string(),
}),
CLEANUP_TIMEOUT,
)
.await?
.map_err(CliError::from_rpc)?;
}
Ok::<_, CliError>(())
})
.await
.context("transfer cleanup exceeded its total time budget")?
})
}
/// Send a `cancel { rpc_id }` frame over a fresh connection so it /// Send a `cancel { rpc_id }` frame over a fresh connection so it
/// lands on the daemon while the original call is still parked. /// lands on the daemon while the original call is still parked.
/// ///
@@ -146,8 +194,57 @@ fn random_short_id() -> String {
bytes.iter().map(|b| format!("{b:02x}")).collect() bytes.iter().map(|b| format!("{b:02x}")).collect()
} }
/// Wait for either Ctrl-C or process SIGTERM. Returning here means /// Opt-in control pipe for parent processes that cannot send Ctrl-C (Node on
/// the user has asked us to stop. /// Windows). One reader survives the short-lived runtimes used by multi-RPC
/// commands such as upload. Ordinary CLI stdin and terminal Ctrl-C are unchanged.
fn stdin_cancellation() -> Option<tokio::sync::watch::Receiver<bool>> {
if std::env::var("BSK_CANCEL_ON_STDIN_CLOSE").as_deref() != Ok("1") {
return None;
}
static CANCEL: std::sync::OnceLock<tokio::sync::watch::Receiver<bool>> =
std::sync::OnceLock::new();
Some(
CANCEL
.get_or_init(|| {
let (tx, rx) = tokio::sync::watch::channel(false);
std::thread::spawn(move || {
use std::io::Read;
let mut stdin = std::io::stdin().lock();
let mut buf = [0; 64];
loop {
match stdin.read(&mut buf) {
Ok(0) => break,
Ok(_) => {}
Err(err) if err.kind() == std::io::ErrorKind::Interrupted => continue,
Err(_) => break,
}
}
let _ = tx.send(true);
});
rx
})
.clone(),
)
}
async fn wait_for_cancel(
parent_cancel: Option<tokio::sync::watch::Receiver<bool>>,
) -> anyhow::Result<()> {
let parent_closed = async move {
match parent_cancel {
Some(mut rx) => {
let _ = rx.wait_for(|closed| *closed).await;
}
None => std::future::pending::<()>().await,
}
};
tokio::select! {
result = wait_for_sigint() => result,
_ = parent_closed => Ok(()),
}
}
/// Wait for terminal Ctrl-C.
async fn wait_for_sigint() -> anyhow::Result<()> { async fn wait_for_sigint() -> anyhow::Result<()> {
tokio::signal::ctrl_c().await.context("listen for SIGINT")?; tokio::signal::ctrl_c().await.context("listen for SIGINT")?;
Ok(()) Ok(())
+2 -9
View File
@@ -9,8 +9,7 @@ use anyhow::Context;
use base64::Engine; use base64::Engine;
use bsk_protocol::Method; use bsk_protocol::Method;
use bsk_protocol::tools::{ use bsk_protocol::tools::{
DownloadParams, DownloadResult, TransferChunkParams, TransferChunkResult, TransferIdParams, DownloadParams, DownloadResult, TransferChunkParams, TransferChunkResult,
TransferReleaseResult,
}; };
use clap::Args; use clap::Args;
use uuid::Uuid; use uuid::Uuid;
@@ -70,13 +69,7 @@ pub fn dispatch(args: DownloadArgs, format: Format) -> Result<(), CliError> {
CliError::Local(anyhow::anyhow!("daemon returned no download transfer id")) CliError::Local(anyhow::anyhow!("daemon returned no download transfer id"))
})?; })?;
let write_result = write_transfer(&info.sock_path, &transfer_id, &args.out, args.overwrite); let write_result = write_transfer(&info.sock_path, &transfer_id, &args.out, args.overwrite);
let _: Result<TransferReleaseResult, CliError> = crate::cli::business_rpc::call( let _ = crate::cli::business_rpc::release_transfers(&info.sock_path, [transfer_id.as_str()]);
info.sock_path,
"transfer-release",
Method::TransferRelease,
Some(TransferIdParams { transfer_id }),
Duration::from_secs(5),
);
write_result?; write_result?;
match format { match format {
Format::Json => { Format::Json => {
+48 -51
View File
@@ -10,8 +10,7 @@ use base64::Engine;
use bsk_protocol::Method; use bsk_protocol::Method;
use bsk_protocol::tools::{ use bsk_protocol::tools::{
TransferBeginParams, TransferBeginResult, TransferChunkParams, TransferChunkResult, TransferBeginParams, TransferBeginResult, TransferChunkParams, TransferChunkResult,
TransferIdParams, TransferReadyResult, TransferReleaseResult, UploadFile, UploadMode, TransferIdParams, TransferReadyResult, UploadFile, UploadMode, UploadParams, UploadResult,
UploadParams, UploadResult,
}; };
use clap::{Args, ValueEnum}; use clap::{Args, ValueEnum};
@@ -69,14 +68,12 @@ pub fn dispatch(args: UploadArgs, format: Format) -> Result<(), CliError> {
let (ref_, selector) = split_target(args.target, args.ref_, args.selector)?; let (ref_, selector) = split_target(args.target, args.ref_, args.selector)?;
let mut staged = Vec::new(); let mut staged = Vec::new();
for path in &args.files { for path in &args.files {
match stage_file(&info.sock_path, &args.session, path) { if let Err(err) = stage_file(&info.sock_path, &args.session, path, &mut staged) {
Ok(file) => staged.push(file), let _ = crate::cli::business_rpc::release_transfers(
Err(err) => { &info.sock_path,
for (id, _) in &staged { staged.iter().map(|(id, _)| id.as_str()),
let _ = release(&info.sock_path, id); );
} return Err(err);
return Err(err);
}
} }
} }
let params = UploadParams { let params = UploadParams {
@@ -102,6 +99,17 @@ pub fn dispatch(args: UploadArgs, format: Format) -> Result<(), CliError> {
Some(params), Some(params),
ipc_timeout(args.timeout), ipc_timeout(args.timeout),
); );
// The helper can reject cancellation before handing any request to the
// daemon. In that case staging still belongs to this CLI.
if result.as_ref().is_err_and(|err| {
err.data()
.is_some_and(|data| data["reason"] == "cancelled_before_dispatch")
}) {
let _ = crate::cli::business_rpc::release_transfers(
&info.sock_path,
staged.iter().map(|(id, _)| id.as_str()),
);
}
// Once tool.upload is dispatched, staging ownership belongs to the // Once tool.upload is dispatched, staging ownership belongs to the
// session. A transport timeout cannot prove that Chrome did not attach // session. A transport timeout cannot prove that Chrome did not attach
// the file, so releasing here could invalidate a late successful attach. // the file, so releasing here could invalidate a late successful attach.
@@ -118,7 +126,12 @@ pub fn dispatch(args: UploadArgs, format: Format) -> Result<(), CliError> {
Ok(()) Ok(())
} }
fn stage_file(sock: &Path, session: &str, path: &PathBuf) -> Result<(String, String), CliError> { fn stage_file(
sock: &Path,
session: &str,
path: &PathBuf,
staged: &mut Vec<(String, String)>,
) -> Result<(), CliError> {
let mut file = File::open(path) let mut file = File::open(path)
.with_context(|| format!("open upload file {}", path.display())) .with_context(|| format!("open upload file {}", path.display()))
.map_err(CliError::Local)?; .map_err(CliError::Local)?;
@@ -149,55 +162,39 @@ fn stage_file(sock: &Path, session: &str, path: &PathBuf) -> Result<(String, Str
}), }),
Duration::from_secs(10), Duration::from_secs(10),
)?; )?;
let staged = (|| { // Record ownership immediately so one error path releases both completed
let mut offset = 0u64; // files and the current partially staged file.
let mut buf = vec![0u8; begin.chunk_size as usize]; staged.push((begin.transfer_id.clone(), name));
loop { let mut offset = 0u64;
let n = file.read(&mut buf).map_err(|e| CliError::Local(e.into()))?; let mut buf = vec![0u8; begin.chunk_size as usize];
if n == 0 { loop {
break; let n = file.read(&mut buf).map_err(|e| CliError::Local(e.into()))?;
} if n == 0 {
let reply: TransferChunkResult = crate::cli::business_rpc::call( break;
sock.to_path_buf(),
"transfer-chunk",
Method::TransferChunk,
Some(TransferChunkParams {
transfer_id: begin.transfer_id.clone(),
offset,
data_base64: base64::engine::general_purpose::STANDARD.encode(&buf[..n]),
}),
Duration::from_secs(30),
)?;
offset = reply.next_offset;
} }
let _: TransferReadyResult = crate::cli::business_rpc::call( let reply: TransferChunkResult = crate::cli::business_rpc::call(
sock.to_path_buf(), sock.to_path_buf(),
"transfer-finish", "transfer-chunk",
Method::TransferFinish, Method::TransferChunk,
Some(TransferIdParams { Some(TransferChunkParams {
transfer_id: begin.transfer_id.clone(), transfer_id: begin.transfer_id.clone(),
offset,
data_base64: base64::engine::general_purpose::STANDARD.encode(&buf[..n]),
}), }),
Duration::from_secs(10), Duration::from_secs(30),
)?; )?;
Ok::<_, CliError>(()) offset = reply.next_offset;
})();
if let Err(err) = staged {
let _ = release(sock, &begin.transfer_id);
return Err(err);
} }
Ok((begin.transfer_id, name)) let _: TransferReadyResult = crate::cli::business_rpc::call(
}
fn release(sock: &Path, id: &str) -> Result<TransferReleaseResult, CliError> {
crate::cli::business_rpc::call(
sock.to_path_buf(), sock.to_path_buf(),
"transfer-release", "transfer-finish",
Method::TransferRelease, Method::TransferFinish,
Some(TransferIdParams { Some(TransferIdParams {
transfer_id: id.to_string(), transfer_id: begin.transfer_id.clone(),
}), }),
Duration::from_secs(5), Duration::from_secs(10),
) )?;
Ok(())
} }
fn ipc_timeout(timeout_ms: u32) -> Duration { fn ipc_timeout(timeout_ms: u32) -> Duration {
@@ -0,0 +1,342 @@
//! Real Windows CLI cancellation, including caller-owned transfer cleanup.
#![cfg(windows)]
use std::path::Path;
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use bsk::daemon::{file_transfer::TransferRegistry, info::DaemonInfo};
use bsk_protocol::tools::{TransferBeginParams, TransferIdParams};
use bsk_protocol::{ErrorCode, Frame, Method, ResponseBody, ResponseFrame, RpcError};
use serde_json::json;
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use tokio::net::windows::named_pipe::ServerOptions;
use tokio::process::Command;
use tokio::sync::{Notify, watch};
fn response<T: serde::Serialize>(result: Result<T, RpcError>) -> ResponseBody {
match result {
Ok(value) => ResponseBody::Ok(serde_json::to_value(value).unwrap()),
Err(err) => ResponseBody::Err(err),
}
}
#[test]
fn stdin_close_cancels_business_but_releases_resources_with_a_bounded_budget() {
// Resolve the production registry's root before starting any runtime threads.
let registry_home = tempfile::tempdir().unwrap();
let original_home = std::env::var_os("BSK_HOME");
unsafe {
std::env::set_var("BSK_HOME", registry_home.path());
}
let transfers = Arc::new(TransferRegistry::new().unwrap());
unsafe {
match original_home {
Some(value) => std::env::set_var("BSK_HOME", value),
None => std::env::remove_var("BSK_HOME"),
}
}
transfers.initialize().unwrap();
tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap()
.block_on(exercise(
transfers,
&registry_home.path().join("run/transfers"),
));
}
async fn exercise(transfers: Arc<TransferRegistry>, staging_root: &Path) {
for mode in [
"wait",
"upload",
"upload-multi",
"upload-before-dispatch",
"download",
"upload-multi-stall",
"ordinary",
] {
let home = tempfile::tempdir().unwrap();
let pipe_name = format!(r"\\.\pipe\bsk-parent-cancel-{}", uuid::Uuid::new_v4());
let info = DaemonInfo::now(std::process::id(), (&pipe_name).into(), 0, "0.2.0");
std::fs::write(
home.path().join("daemon.json"),
serde_json::to_vec(&info).unwrap(),
)
.unwrap();
let mut pipe = ServerOptions::new()
.first_pipe_instance(true)
.create(&pipe_name)
.unwrap();
let registered = Arc::new(Notify::new());
let pending = Arc::new(Mutex::new(None::<String>));
let ids = Arc::new(Mutex::new(Vec::<String>::new()));
let released = Arc::new(Mutex::new(Vec::<String>::new()));
let received = Arc::new(Mutex::new(Vec::<Method>::new()));
let download_id = if mode == "download" {
let staging = transfers.begin_download("fixture").unwrap();
let browser_dir = home.path().join("BrowserSkill").join(&staging.transfer_id);
std::fs::create_dir_all(&browser_dir).unwrap();
let file = browser_dir.join("download.txt");
std::fs::write(&file, b"download fixture").unwrap();
transfers
.import_download(&staging.transfer_id, &file)
.unwrap();
ids.lock().unwrap().push(staging.transfer_id.clone());
Some(staging.transfer_id)
} else {
None
};
let (cancel, cancelled) = watch::channel(false);
let ready = Arc::clone(&registered);
let name = pipe_name.clone();
let registry = Arc::clone(&transfers);
let server_ids = Arc::clone(&ids);
let server_released = Arc::clone(&released);
let server_received = Arc::clone(&received);
let server = tokio::spawn(async move {
let mut connections = tokio::task::JoinSet::new();
loop {
pipe.connect().await.unwrap();
let next = ServerOptions::new().create(&name).unwrap();
let connection = std::mem::replace(&mut pipe, next);
let ready = Arc::clone(&ready);
let pending = Arc::clone(&pending);
let cancel = cancel.clone();
let mut cancelled = cancelled.clone();
let name = name.clone();
let registry = Arc::clone(&registry);
let ids = Arc::clone(&server_ids);
let released = Arc::clone(&server_released);
let received = Arc::clone(&server_received);
let download_id = download_id.clone();
connections.spawn(async move {
let (read, mut write) = tokio::io::split(connection);
let mut reader = BufReader::new(read);
let mut line = String::new();
while reader.read_line(&mut line).await.unwrap_or(0) > 0 {
let Frame::Request(req) = serde_json::from_str::<Frame>(&line).unwrap()
else {
panic!("expected request");
};
line.clear();
received.lock().unwrap().push(req.method.clone());
let params = req.params.clone().unwrap_or_default();
let body = match req.method {
Method::SystemStatus => ResponseBody::Ok(json!({
"daemon_version": "0.2.0", "protocol_version": "1.1",
"pid": std::process::id(), "uptime_secs": 0,
"ws_port": 0, "sock_path": name,
"browsers": [], "sessions": [], "version_skew_browsers": []
})),
Method::TransferBegin => {
let begin = registry
.begin_upload(serde_json::from_value(params).unwrap())
.unwrap();
ids.lock().unwrap().push(begin.transfer_id.clone());
response(Ok(begin))
}
Method::ToolDownload => ResponseBody::Ok(json!({
"tab_id": 1, "suggested_filename": "download.txt",
"byte_size": 16, "transfer_id": download_id,
})),
Method::ToolWaitMs if mode == "ordinary" => {
ResponseBody::Ok(json!({"waited_ms": 1}))
}
Method::TransferChunk
if mode == "upload-before-dispatch"
|| (mode.contains("multi")
&& ids.lock().unwrap().len() == 1) =>
{
response(
registry.write_chunk(serde_json::from_value(params).unwrap()),
)
}
Method::TransferFinish if mode != "upload-before-dispatch" => response(
registry.finish_upload(serde_json::from_value(params).unwrap()),
),
Method::ToolWaitMs
| Method::TransferChunk
| Method::TransferRead
| Method::TransferFinish => {
*pending.lock().unwrap() = Some(req.id.clone());
ready.notify_one();
cancelled.wait_for(|value| *value).await.unwrap();
if mode == "upload-before-dispatch" {
// Finish won the race with cancellation, but tool.upload must
// not be dispatched and the CLI still owns the staging.
response(
registry
.finish_upload(serde_json::from_value(params).unwrap()),
)
} else {
ResponseBody::Err(RpcError {
code: ErrorCode::Cancelled,
message: "parent cancellation reached daemon".into(),
data: None,
})
}
}
Method::Cancel => {
let target = params["rpc_id"].as_str().unwrap();
assert_eq!(pending.lock().unwrap().as_deref(), Some(target));
cancel.send(true).unwrap();
ResponseBody::Ok(json!({"cancelled": true}))
}
Method::TransferRelease => {
let params: TransferIdParams =
serde_json::from_value(params).unwrap();
released.lock().unwrap().push(params.transfer_id.clone());
if mode.contains("stall") {
std::future::pending::<()>().await;
}
response(Ok(registry.release(params)))
}
other => panic!("unexpected request after cancellation: {other:?}"),
};
let response = Frame::Response(ResponseFrame { id: req.id, body });
let mut bytes = serde_json::to_vec(&response).unwrap();
bytes.push(b'\n');
if write.write_all(&bytes).await.is_err() {
break;
}
}
});
}
});
let mut cmd = Command::new(env!("CARGO_BIN_EXE_bsk"));
cmd.env("BSK_HOME", home.path())
.env("BSK_AUTO_UPDATE", "off")
.env_remove("BSK_CANCEL_ON_STDIN_CLOSE")
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped())
.creation_flags(0x0800_0000)
.kill_on_drop(true);
if mode.starts_with("upload") {
let source = home.path().join("upload.txt");
// Declare 300 MiB without allocating or transferring that much data.
std::fs::File::create(&source)
.unwrap()
.set_len(if mode == "upload-before-dispatch" {
16
} else {
300 * 1024 * 1024
})
.unwrap();
cmd.args(["upload", "--session", "fixture", "--selector", "input"]);
if mode.contains("multi") {
let first = home.path().join("first.txt");
std::fs::write(&first, b"first file").unwrap();
cmd.arg("--file").arg(first);
}
cmd.arg("--file").arg(source);
} else if mode == "download" {
let out = home.path().join("out.txt");
std::fs::write(&out, b"existing output").unwrap();
cmd.args([
"download",
"--session",
"fixture",
"--selector",
"a",
"--overwrite",
"--out",
])
.arg(out);
} else {
cmd.args(["wait-ms", if mode == "ordinary" { "1ms" } else { "60s" }]);
}
cmd.arg("--json");
if mode == "ordinary" {
cmd.stdin(std::process::Stdio::null());
} else {
cmd.env("BSK_CANCEL_ON_STDIN_CLOSE", "1")
.stdin(std::process::Stdio::piped());
}
let mut child = cmd.spawn().unwrap();
if mode != "ordinary" {
tokio::time::timeout(Duration::from_secs(10), registered.notified())
.await
.expect("CLI registered business RPC");
child
.stdin
.as_mut()
.unwrap()
.write_all(b"still connected")
.await
.unwrap();
drop(child.stdin.take());
}
let start = Instant::now();
let output = tokio::time::timeout(Duration::from_secs(8), child.wait_with_output())
.await
.expect("CLI must settle promptly, including cleanup")
.unwrap();
server.abort();
let _ = server.await;
assert_eq!(
output.status.code(),
Some(if mode == "ordinary" { 0 } else { 2 }),
"{mode}: stdout={} stderr={}",
String::from_utf8_lossy(&output.stdout),
String::from_utf8_lossy(&output.stderr)
);
if mode != "ordinary" {
let body: serde_json::Value = serde_json::from_slice(&output.stdout).unwrap();
assert_eq!(body["code"], "cancelled");
if mode != "upload-before-dispatch" {
assert_eq!(body["message"], "parent cancellation reached daemon");
}
let received = received.lock().unwrap();
let cancel = received.iter().position(|m| *m == Method::Cancel).unwrap();
assert!(
received[cancel + 1..]
.iter()
.all(|m| *m == Method::TransferRelease),
"only cleanup may follow cancellation: {received:?}"
);
}
let ids = ids.lock().unwrap();
if mode.contains("stall") {
assert_eq!(released.lock().unwrap().len(), 1);
assert!(start.elapsed() >= Duration::from_secs(4));
// The CLI cannot force an unresponsive daemon to clean up.
transfers.release_session("fixture");
} else {
assert_eq!(
*released.lock().unwrap(),
*ids,
"{mode}: every allocation must be released"
);
for id in ids.iter() {
assert!(!staging_root.join(id).exists(), "staging leaked: {id}");
}
if mode.starts_with("upload") {
let retry = transfers
.begin_upload(TransferBeginParams {
session_id: "fixture".into(),
name: "retry.txt".into(),
byte_size: 300 * 1024 * 1024,
})
.expect("same session must have room for another 300 MiB upload");
transfers.release(TransferIdParams {
transfer_id: retry.transfer_id,
});
}
}
if mode == "download" {
assert_eq!(
std::fs::read(home.path().join("out.txt")).unwrap(),
b"existing output"
);
assert!(
!std::fs::read_dir(home.path())
.unwrap()
.flatten()
.any(|entry| entry.file_name().to_string_lossy().ends_with(".part"))
);
}
}
}
@@ -0,0 +1,41 @@
# OOPIF occupied-scrollbar regression
An iframe's content quad includes its child viewport's scrollbar space, while
`Page.getLayoutMetrics().cssLayoutViewport` excludes that space. Mapping the latter
onto the whole quad stretches positions and sizes. The full target-local viewport
must determine scale; the visible viewport must still clip content at every OOPIF
boundary and constrain target-local action points.
This fixture alternates loopback hostnames across two nested OOPIFs. It includes
scrolling, borders, padding, scaled iframe owners, a partially clipped button and
a fully clipped button. Custom scrollbars occupy different widths and heights.
The `scrollbars` query parameter accepts `both` (default), `vertical`, `horizontal`
or `none`.
Run the numerical regression with Node 22+ and a local Chrome executable:
```sh
BSK_GEOMETRY_CHROME=/path/to/chrome pnpm --filter @browser-skill/extension exec vitest run \
src/tools/__tests__/snapshot-coordinates.browser.test.ts
```
The existing browser runner is shared with the snapshot-unit regression. The test:
- Verifies actual OOPIF targets and occupied scrollbar dimensions.
- Compares snapshot and live geometry with independent DOM rectangles and clipping.
- Dispatches real root-target clicks and verifies the intended frame/button received them.
- Rejects fully clipped controls and checks snapshot measurement reuse per target.
- Covers five device-scale/browser-zoom combinations plus single-axis and no-scrollbar cases.
The runner creates and removes isolated browser profiles. It is opt-in and skipped
in normal unit runs when `BSK_GEOMETRY_CHROME` is unset.
Run the CLI fixture smoke with a connected test extension:
```sh
BSK_AUTO_UPDATE=off pnpm eval:browser smoke --case oopif-scrollbars --bsk ./target/debug/bsk
```
The smoke assertions verify both nested frames have occupied scrollbars, that the
marker is observed and that the session is closed. Numeric geometry and actual
click assertions are exercised by the browser test above, not by smoke alone.
@@ -0,0 +1,39 @@
{
"$schema": "../../../schemas/case.schema.json",
"schemaVersion": 1,
"id": "oopif-scrollbars",
"title": "Nested OOPIFs with occupied scrollbars and clipped controls",
"suite": "regression",
"tags": ["geometry", "iframe", "scroll"],
"fixture": { "startPath": "/oopif-scrollbars" },
"prompts": {
"en": "Open {url}, observe the page and its frames, report the OOPIF-SCROLLBARS marker, then close the session.",
"zh-CN": "打开 {url},观察页面及其 iframe,报告 OOPIF-SCROLLBARS 标记,然后关闭会话。"
},
"coverage": ["session.start", "session.stop", "page.navigate", "inspect.observe"],
"assertions": {
"site": [
{
"label": "nested frame fixture settled",
"type": "geometry.ready",
"where": { "data.root": true },
"minCount": 1
},
{
"label": "both child frames have occupied scrollbars",
"type": "geometry.scrollbars",
"where": { "data.vertical": true, "data.horizontal": true },
"minCount": 2
}
],
"response": [{ "label": "observed the fixture marker", "includes": "OOPIF-SCROLLBARS" }],
"adapter": [{ "label": "browser session was closed", "key": "sessionStopped" }]
},
"smoke": {
"steps": [
{ "action": "navigate" },
{ "action": "wait-site-event", "type": "geometry.ready", "where": { "root": true } },
{ "action": "observe" }
]
}
}
@@ -0,0 +1,56 @@
import { page, withRun } from "../../../lib/fixtures.mjs";
export default {
id: "oopif-scrollbars",
routes: ["/oopif-scrollbars", "/oopif-scrollbars/frame", "/oopif-scrollbars/nested"],
render({ pathname, runId, query }) {
const root = pathname === "/oopif-scrollbars";
const nested = pathname.endsWith("/nested");
const mode = query.get("scrollbars") ?? "both";
return page({
title: "OOPIF scrollbar geometry regression",
body: `
<style>
main { margin: 0; width: auto; }
body { width: 2000px; height: 2400px; }
html { overflow-anchor: none; }
${!root ? `html { overflow-x: ${mode === "vertical" || mode === "none" ? "hidden" : "scroll"}; overflow-y: ${mode === "horizontal" || mode === "none" ? "hidden" : "scroll"}; } html::-webkit-scrollbar { width: 17px; height: 11px; }` : ""}
button { position: absolute; padding: 0; border: 0; border-radius: 0; }
#probe { left: ${root ? 180 : nested ? 30 : 100}px; top: ${root ? 360 : nested ? 40 : 160}px; width: 60px; height: 20px; }
#edge { position: fixed; right: -20px; bottom: -15px; width: 80px; height: 60px; }
#outside { position: fixed; left: calc(100% + 2px); top: calc(100% + 2px); width: 10px; height: 10px; }
iframe { position: absolute; box-sizing: content-box; width: 420px; height: 320px;
left: 340px; top: 400px; border: 6px solid black; padding: 8px; transform: scale(1.2); transform-origin: 0 0; }
#nested { left: 150px; top: 210px; width: 200px; height: 120px; border-width: 4px; padding: 6px; transform: scale(0.8); }
</style>
<button id="probe" data-geometry-probe>OOPIF-SCROLLBARS</button>
${!root ? '<button id="edge" data-geometry-probe>EDGE</button><button id="outside" data-geometry-probe>OUTSIDE</button>' : ""}
${!nested ? `<iframe id="${root ? "cross" : "nested"}" name="${root ? "cross" : "nested"}" title="${root ? "outer" : "nested"} cross-site frame"></iframe>` : ""}
`,
script: `
history.scrollRestoration = "manual";
const child = document.querySelector("iframe");
if (child) {
const url = new URL(${JSON.stringify(withRun(root ? "/oopif-scrollbars/frame" : "/oopif-scrollbars/nested", runId, { scrollbars: mode }))}, location.href);
url.hostname = location.hostname === "127.0.0.1" ? "localhost" : "127.0.0.1";
child.src = url.href;
}
document.addEventListener("click", event => {
const probe = event.target.closest("[data-geometry-probe]");
if (probe) browserEval.send("geometry.clicked", { probe: probe.id });
});
window.addEventListener("load", () => {
scrollTo(${root ? "80, 240" : nested ? "10, 20" : "40, 100"});
requestAnimationFrame(() => requestAnimationFrame(() => {
document.documentElement.dataset.geometryReady = "true";
if (!${root}) browserEval.send("geometry.scrollbars", {
vertical: innerWidth > document.documentElement.clientWidth,
horizontal: innerHeight > document.documentElement.clientHeight,
});
browserEval.send("geometry.ready", { root: ${root} });
}));
});
`,
});
},
};
@@ -0,0 +1,41 @@
# DOMSnapshot layout-unit regression (#195)
Source: https://github.com/Tencent/BrowserSkill/pull/195
DOMSnapshot bounds and document scroll offsets retain Blink layout units. Treating them as CSS
pixels changes both positions and sizes under device scaling/browser zoom, before iframe
projection or viewport clipping even starts.
The fixture contains a scrolled root, a scrolled same-process iframe, an OOPIF served from the
other loopback hostname, and a nested iframe in each. Owners have borders, padding and CSS
transforms. The geometry test compares the production capture's local and top-level rectangles
against independent DOM border boxes, with at most 2 CSS pixels of layout rounding. It also
checks that each target reads layout metrics only once. No credentials or personal tabs are used.
Run the geometric assertions with Node 22+ and a local Chrome executable:
```sh
BSK_GEOMETRY_CHROME=/path/to/chrome pnpm --filter @browser-skill/extension exec vitest run \
src/tools/__tests__/snapshot-coordinates.browser.test.ts
```
The test launches and cleans up its own headless Chrome profiles. It explicitly verifies the
requested browser zoom and OOPIF topology, so unsupported setups fail rather than silently
testing a different configuration. It is skipped in ordinary unit runs unless the executable
is specified. No browser download or new package dependency is required.
The corpus smoke case checks fixture readiness and observable semantics through the real CLI:
```sh
BSK_AUTO_UPDATE=off pnpm eval:browser smoke --case snapshot-coordinates --bsk ./target/debug/bsk
```
Smoke alone does **not** certify coordinate accuracy; CLI observations do not expose raw boxes.
Use the geometric test above for the regression's numeric assertions.
## Related scrollbar regression
The unit-conversion fixture keeps non-occupying OOPIF scrollbars by default. The same
browser test also runs the dedicated [OOPIF scrollbar fixture](../oopif-scrollbars/README.md),
which covers occupied scrollbars, two OOPIF boundaries, clipping and real clicks.
Append `&classic-scrollbars` to this fixture's URL to enable its normal OOPIF scrollbars.
@@ -0,0 +1,99 @@
import { spawn } from "node:child_process";
import { once } from "node:events";
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
// This regression owns its browser/profile. It never attaches to a user's Chrome.
export async function withChrome({ executable, deviceScale, zoom }, run) {
const profile = await mkdtemp(join(tmpdir(), "bsk-snapshot-coordinates-"));
let chrome;
let socket;
const pending = new Map();
try {
await mkdir(join(profile, "Default"));
await writeFile(
join(profile, "Default", "Preferences"),
JSON.stringify({
partition: { default_zoom_level: { x: Math.log(zoom) / Math.log(1.2) } },
}),
);
chrome = spawn(
executable,
[
"--headless=new",
"--no-first-run",
"--no-default-browser-check",
"--disable-extensions",
"--site-per-process",
"--remote-debugging-port=0",
"--window-size=1600,1200",
`--force-device-scale-factor=${deviceScale}`,
`--user-data-dir=${profile}`,
"about:blank",
],
{ stdio: ["ignore", "ignore", "pipe"] },
);
const endpoint = await new Promise((resolve, reject) => {
let output = "";
const timeout = setTimeout(
() => reject(new Error(`Chrome startup timed out: ${output}`)),
15_000,
);
const fail = (error) => {
clearTimeout(timeout);
reject(error);
};
chrome.once("error", fail);
chrome.once("exit", (code) => fail(new Error(`Chrome exited (${code}): ${output}`)));
chrome.stderr.on("data", (chunk) => {
output += chunk;
const match = output.match(/DevTools listening on (ws:\/\/\S+)/);
if (match) {
clearTimeout(timeout);
resolve(match[1]);
}
});
});
socket = new WebSocket(endpoint);
await new Promise((resolve, reject) => {
socket.addEventListener("open", resolve, { once: true });
socket.addEventListener("error", reject, { once: true });
});
socket.addEventListener("message", ({ data }) => {
const reply = JSON.parse(data);
const request = pending.get(reply.id);
if (!request) return;
clearTimeout(request.timeout);
pending.delete(reply.id);
if (reply.error) request.reject(new Error(JSON.stringify(reply.error)));
else request.resolve(reply.result);
});
let sequence = 0;
const send = (method, params = {}, sessionId) =>
new Promise((resolve, reject) => {
const id = ++sequence;
const timeout = setTimeout(() => {
pending.delete(id);
reject(new Error(`CDP timed out: ${method}`));
}, 10_000);
pending.set(id, { resolve, reject, timeout });
socket.send(JSON.stringify({ id, method, params, sessionId }));
});
await run(send);
} finally {
for (const request of pending.values()) {
clearTimeout(request.timeout);
request.reject(new Error("regression browser closed"));
}
socket?.close();
if (chrome && chrome.exitCode === null && chrome.signalCode === null) {
const exited = once(chrome, "exit");
const timeout = setTimeout(() => chrome.kill("SIGKILL"), 5_000);
chrome.kill();
await exited;
clearTimeout(timeout);
}
await rm(profile, { recursive: true, force: true });
}
}
@@ -0,0 +1,33 @@
{
"$schema": "../../../schemas/case.schema.json",
"schemaVersion": 1,
"id": "snapshot-coordinates",
"title": "Scrolled and nested frame coordinate fixture",
"suite": "regression",
"tags": ["geometry", "iframe", "scroll"],
"fixture": { "startPath": "/snapshot-coordinates" },
"prompts": {
"en": "Open {url}, observe the page and its frames, report the GEOMETRY marker, then close the session.",
"zh-CN": "打开 {url},观察页面及其 iframe,报告 GEOMETRY 标记,然后关闭会话。"
},
"coverage": ["session.start", "session.stop", "page.navigate", "inspect.observe"],
"assertions": {
"site": [
{
"label": "root fixture scrolled and settled",
"type": "geometry.ready",
"where": { "data.root": true },
"minCount": 1
}
],
"response": [{ "label": "observed the fixture marker", "includes": "GEOMETRY-195" }],
"adapter": [{ "label": "browser session was closed", "key": "sessionStopped" }]
},
"smoke": {
"steps": [
{ "action": "navigate" },
{ "action": "wait-site-event", "type": "geometry.ready", "where": { "root": true } },
{ "action": "observe" }
]
}
}
@@ -0,0 +1,52 @@
import { page, withRun } from "../../../lib/fixtures.mjs";
export default {
id: "snapshot-coordinates",
routes: ["/snapshot-coordinates", "/snapshot-coordinates/frame", "/snapshot-coordinates/nested"],
render({ pathname, runId, query }) {
const root = pathname === "/snapshot-coordinates";
const nested = pathname.endsWith("/nested");
return page({
title: "Snapshot coordinate regression",
body: `
<style>
main { margin: 0; width: auto; }
body { width: 2000px; height: 2400px; }
html { overflow-anchor: none; }
${query.get("scrollbars") === "none" ? "html { scrollbar-width: none; }" : ""}
button { position: absolute; box-sizing: border-box; padding: 0; border: 2px solid black; border-radius: 0; }
#probe { left: ${root ? 180 : 100}px; top: ${root ? 360 : 160}px; width: 120px; height: 40px; }
iframe { position: absolute; box-sizing: content-box; width: 320px; height: 220px;
border: 6px solid black; padding: 8px; transform: scale(1.25); transform-origin: 0 0; }
#same { left: 340px; top: 400px; }
#cross { left: 820px; top: 400px; }
#nested { left: 150px; top: 210px; width: 130px; height: 70px; border-width: 2px; padding: 4px; transform: scale(0.8); }
${nested ? "#probe { left: 30px; top: 40px; width: 60px; height: 20px; }" : ""}
</style>
<button id="probe" data-geometry-probe>GEOMETRY-195</button>
${root ? `<iframe id="same" name="same" title="same-process frame" src="${withRun("/snapshot-coordinates/frame", runId)}"></iframe><iframe id="cross" name="cross" title="cross-site frame"></iframe>` : ""}
${!root && !nested ? `<iframe id="nested" name="nested" title="nested frame" src="${withRun("/snapshot-coordinates/nested", runId)}"></iframe>` : ""}
`,
script: `
history.scrollRestoration = "manual";
const cross = document.querySelector("#cross");
if (cross) {
const url = new URL(${JSON.stringify(withRun("/snapshot-coordinates/frame", runId))}, location.href);
url.hostname = location.hostname === "127.0.0.1" ? "localhost" : "127.0.0.1";
// Isolate unit conversion from the pre-existing OOPIF scrollbar-width
// projection issue. ?classic-scrollbars retains that separate reproduction.
if (!new URLSearchParams(location.search).has("classic-scrollbars")) url.searchParams.set("scrollbars", "none");
cross.src = url.href;
}
window.addEventListener("load", () => {
scrollTo(${root ? "80, 240" : nested ? "10, 20" : "40, 100"});
requestAnimationFrame(() => requestAnimationFrame(() => {
window.geometryReady = true;
document.documentElement.dataset.geometryReady = "true";
browserEval.send("geometry.ready", { root: ${root} });
}));
});
`,
});
},
};
+10 -4
View File
@@ -41,17 +41,23 @@ test("case manifests are discovered, ordered, and grouped into suites", () => {
"diagnostics", "diagnostics",
"mobile-emulation", "mobile-emulation",
"generated-form", "generated-form",
"oopif-scrollbars",
"snapshot-coordinates",
], ],
); );
assert.deepEqual(repositorySummary(cases, fixtureRegistry).suites, { core: 6, matrix: 1 }); assert.deepEqual(repositorySummary(cases, fixtureRegistry).suites, {
core: 6,
matrix: 1,
regression: 2,
});
}); });
test("repository validation links every case to a fixture and valid workflow evidence", () => { test("repository validation links every case to a fixture and valid workflow evidence", () => {
assert.deepEqual(validateRepositoryCases(cases, fixtureRegistry), []); assert.deepEqual(validateRepositoryCases(cases, fixtureRegistry), []);
const summary = repositorySummary(cases, fixtureRegistry); const summary = repositorySummary(cases, fixtureRegistry);
assert.equal(summary.cases, 7); assert.equal(summary.cases, 9);
assert.equal(summary.fixtureModules, 8); assert.equal(summary.fixtureModules, 10);
assert.equal(summary.fixtureRoutes, 12); assert.equal(summary.fixtureRoutes, 18);
}); });
test("manifest validation rejects unknown operations and incomplete workflow steps", () => { test("manifest validation rejects unknown operations and incomplete workflow steps", () => {
+41 -15
View File
@@ -21,6 +21,7 @@ $ErrorActionPreference = "Stop"
$Repo = if ($env:BSK_REPO) { $env:BSK_REPO } else { "Tencent/BrowserSkill" } $Repo = if ($env:BSK_REPO) { $env:BSK_REPO } else { "Tencent/BrowserSkill" }
$InstallDir = if ($env:BSK_INSTALL_DIR) { $env:BSK_INSTALL_DIR } else { Join-Path $HOME ".local\bin" } $InstallDir = if ($env:BSK_INSTALL_DIR) { $env:BSK_INSTALL_DIR } else { Join-Path $HOME ".local\bin" }
$InstallDir = $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath($InstallDir)
$GitHub = "https://github.com/${Repo}" $GitHub = "https://github.com/${Repo}"
function Write-Log { function Write-Log {
@@ -64,7 +65,7 @@ function Get-PlatformTriple {
function Add-ToUserPath { function Add-ToUserPath {
param([string]$Dir) param([string]$Dir)
$currentUserPath = [Environment]::GetEnvironmentVariable("PATH", "User") -split ";" | Where-Object { $_ } $currentUserPath = @([Environment]::GetEnvironmentVariable("PATH", "User") -split ";" | Where-Object { $_ })
if ($currentUserPath -contains $Dir) { if ($currentUserPath -contains $Dir) {
Write-Log "$Dir is already in your user PATH" Write-Log "$Dir is already in your user PATH"
@@ -90,30 +91,59 @@ function Add-ToSessionPath {
# ── Git Bash (bash environment) PATH helper ────────────────────────────────── # ── Git Bash (bash environment) PATH helper ──────────────────────────────────
function Add-ToBashProfile { function Add-ToBashProfile {
param([string]$Dir) param([string]$Dir, [string]$BashRc = (Join-Path $HOME ".bashrc"))
# Convert Windows path (e.g. C:\Users\foo\.local\bin) to Git-Bash Unix-style (/c/Users/foo/.local/bin) # Convert Windows path (e.g. C:\Users\foo\.local\bin) to Git-Bash Unix-style (/c/Users/foo/.local/bin)
$unixPath = $Dir -replace '\\', '/' $unixPath = $Dir -replace '\\', '/'
if ($unixPath -match '^([A-Z]):(.*)$') { if ($unixPath -match '^([A-Z]):(.*)$') {
$unixPath = '/' + $matches[1].ToLower() + $matches[2] $unixPath = '/' + $matches[1].ToLower() + $matches[2]
} }
$bashRc = Join-Path $HOME ".bashrc" # Single-quote the literal directory; only the existing PATH is expanded.
$exportLine = "export PATH=""${unixPath}:`$PATH"" # bsk CLI" $shellQuote = "'" + [char]34 + "'" + [char]34 + "'"
$quotedPath = "'" + $unixPath.Replace("'", $shellQuote) + "'"
$exportLine = "export PATH=${quotedPath}:`"`$PATH`" # bsk CLI"
if (Test-Path $bashRc) { if (Test-Path -LiteralPath $BashRc) {
$content = Get-Content $bashRc -Raw -ErrorAction SilentlyContinue $content = [System.IO.File]::ReadAllText($BashRc)
if ($content -match [regex]::Escape($unixPath)) { if ($content.Contains($exportLine)) {
Write-Log "$unixPath is already in ~/.bashrc" Write-Log "$unixPath is already in ~/.bashrc"
return return
} }
} }
Add-Content $bashRc "`n$exportLine" -Encoding ASCII # Explicit BOM-less UTF-8 also works in Windows PowerShell 5.1.
[System.IO.File]::AppendAllText($BashRc, "`n$exportLine`n", (New-Object System.Text.UTF8Encoding($false)))
Write-Log "added ${unixPath} to ~/.bashrc" Write-Log "added ${unixPath} to ~/.bashrc"
} }
# ── Main ────────────────────────────────────────────────────────────────────── # ── Main ──────────────────────────────────────────────────────────────────────
# Stage on the destination volume before stopping the daemon. Never truncate
# the installed executable: failed replacement must leave it usable.
function Install-Binary {
param([string]$Source, [string]$Target)
$staged = "$Target.install-$([Guid]::NewGuid().ToString('N'))"
try {
[System.IO.File]::Copy($Source, $staged)
if ([System.IO.File]::Exists($Target)) {
# Use the downloaded CLI: older versions have broken Windows
# liveness checks. Stop verifies daemon identity before terminating it.
& $Source daemon stop
if ($LASTEXITCODE -ne 0) { throw "could not stop bsk daemon; existing installation was not replaced" }
# PowerShell 5.1 converts $null to an empty path for string parameters.
[System.IO.File]::Replace($staged, $Target, [NullString]::Value)
Write-Log "daemon will restart automatically on the next browser command"
}
else {
[System.IO.File]::Move($staged, $Target)
}
}
finally {
if ([System.IO.File]::Exists($staged)) { [System.IO.File]::Delete($staged) }
}
}
function Main { function Main {
$platform = Get-PlatformTriple $platform = Get-PlatformTriple
@@ -184,7 +214,7 @@ function Main {
New-Item -ItemType Directory -Path $InstallDir -Force | Out-Null New-Item -ItemType Directory -Path $InstallDir -Force | Out-Null
} }
Copy-Item -Path (Join-Path $tempDir "bsk.exe") -Destination (Join-Path $InstallDir "bsk.exe") -Force Install-Binary -Source (Join-Path $tempDir "bsk.exe") -Target (Join-Path $InstallDir "bsk.exe")
Write-Log "installed bsk to $InstallDir\bsk.exe" Write-Log "installed bsk to $InstallDir\bsk.exe"
@@ -199,12 +229,8 @@ function Main {
# Verify # Verify
$bskPath = Join-Path $InstallDir "bsk.exe" $bskPath = Join-Path $InstallDir "bsk.exe"
if (Get-Command bsk -ErrorAction SilentlyContinue) { & $bskPath --version
& bsk --version if ($LASTEXITCODE -ne 0) { throw "installed bsk failed verification" }
}
else {
Write-Log "verify install: & ""$bskPath"" --version"
}
Write-Log "done" Write-Log "done"
Write-Host "" Write-Host ""
+138 -154
View File
@@ -1,13 +1,51 @@
# dsh-plugin-browserskill # BrowserSkill for DeepSeek Harness
npm: [`@wxg-prc-cpg/browser-skill-dsh-plugin`](https://www.npmjs.com/package/@wxg-prc-cpg/browser-skill-dsh-plugin) [![npm version](https://img.shields.io/npm/v/@wxg-prc-cpg/browser-skill-dsh-plugin)](https://www.npmjs.com/package/@wxg-prc-cpg/browser-skill-dsh-plugin)
A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (dsh) tool plugin that exposes Use [BrowserSkill](https://github.com/Tencent/BrowserSkill) in
[BrowserSkill](https://github.com/Tencent/BrowserSkill) (`bsk`) browser automation to the model. [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (`dsh`) to browse
websites, fill forms, and capture screenshots through native `browser_*` tools.
Browser tasks run in Agent Windows, with a live view in the dsh Web UI.
Each action maps to one `bsk <cmd> --json` invocation: the plugin spawns the bsk CLI, parses its ## Installation
structured JSON output, and returns a canonical typed value. The bsk daemon, browser, and browser
extension keep owning the actual browser control — this package is a thin, well-typed bridge. Before installing the plugin:
- Install [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) and
[pnpm](https://pnpm.io/installation), which dsh uses to manage plugins.
- Install the `bsk` CLI and connect the BrowserSkill extension in Chrome or Edge.
Follow the [BrowserSkill setup guide](https://github.com/Tencent/BrowserSkill#quick-start).
- Make sure `bsk` is on the `PATH` used to start dsh, or set `bskPath` in the plugin configuration.
Install the plugin into the `web` profile, then start it:
```sh
dsh plugin --profile web add @wxg-prc-cpg/browser-skill-dsh-plugin
dsh --profile web
```
Replace `web` with your profile name if you use a different profile. The plugin
includes the `browser-skill` skill; no separate `bsk install-skill` step is needed.
In a conversation, try:
```text
/browser-skill open example.com and summarize the page.
```
By default, the browser tools become available when the skill is invoked.
## Updating
Installed plugins do not update automatically. To upgrade this plugin to npm's
`latest` version, including versions outside the profile's saved dependency range:
```sh
dsh plugin --profile web update @wxg-prc-cpg/browser-skill-dsh-plugin --latest
```
Restart that dsh profile after upgrading. This command updates the plugin; update
the `bsk` CLI and browser extension separately when a release requires it.
## Tools ## Tools
@@ -20,23 +58,7 @@ extension keep owning the actual browser control — this package is a thin, wel
| `browser_tabs` | `list`, `create`, `select`, `close`, `borrow`, `return` | Manage Agent Window tabs and temporarily borrow user tabs. | | `browser_tabs` | `list`, `create`, `select`, `close`, `borrow`, `return` | Manage Agent Window tabs and temporarily borrow user tabs. |
| `browser_assist` | `resize`, `emulate`, `request-help` | Resize or emulate the browser and pause for human-only steps. | | `browser_assist` | `resize`, `emulate`, `request-help` | Resize or emulate the browser and pause for human-only steps. |
`bsk evaluate` and `bsk record` are intentionally not exposed by this plugin: arbitrary page Arbitrary page-script evaluation and interaction recording are not supported.
evaluation is a higher-risk capability, while recording is long-running and needs a dedicated
DeepSeek Harness lifecycle before it can be added safely.
## Agent skill (progressive disclosure)
Beyond the tools, the plugin publishes the **`browser-skill` agent skill** through the harness's
official skill seam (`ctx.skills.register`): the catalog entry (name + routing description) is
resident in `<available_skills>`, and the body is loaded only when the model invokes the `skill`
tool. Its single source is the DSH-specific `skill/SKILL.md`, which documents only structured
`browser_*` calls and their plugin semantics. The repository-root CLI skill is intentionally not
concatenated: its command examples belong to a different execution interface and would bypass
the plugin's ownership, live observation UI, cancellation, and cleanup path if followed directly.
The build rejects internal CLI-name leakage, command-line code blocks, unknown browser tools, and
missing supported browser tools before embedding the Markdown. Registration and every pre-step
catalog snapshot are pure in-memory reads (no disk/process/daemon); compositions without the skill
seam degrade silently.
## Multi-session model ## Multi-session model
@@ -55,136 +77,73 @@ an explicit `session` argument naming a foreign or unknown id is rejected, the `
`browser_session` shows plugin-created sessions only (no daemon-wide view), and stop/unload cleanup `browser_session` shows plugin-created sessions only (no daemon-wide view), and stop/unload cleanup
can never touch a session owned by another program. can never touch a session owned by another program.
## Installation
The plugin follows the standard dsh bundle layout (`dsh.bundle` manifest + `cordis.patch.yml`):
```sh
dsh plugin --profile <name> add @wxg-prc-cpg/browser-skill-dsh-plugin
dsh --profile <name>
```
Prerequisite: the `bsk` CLI must be installed and on `PATH`, and the BrowserSkill browser extension
(Chrome or Edge) must be connected — see the
[BrowserSkill README](https://github.com/Tencent/BrowserSkill). When bsk is missing, tool calls fail
with install guidance instead of a bare spawn error.
## Configuration ## Configuration
All fields are optional and validated through the plugin's Schemastery `Config`: After installing the plugin, edit your profile's `cordis.patch.yml`. For the `web`
profile, the default location is `~/.dsh/profiles/web/cordis.patch.yml`. If you set
`DSH_HOME`, use `$DSH_HOME/profiles/web/cordis.patch.yml` instead. Replace `web` with
your profile name as needed.
If the file contains only comments and `[]`, keep the comments and replace `[]`
with the YAML below. If it already contains patch entries, add this entry to the
existing list or edit its existing `id: browserskill` entry. Keep a single
top-level YAML list. This overrides the plugin registered by the installed bundle:
```yaml ```yaml
# cordis.patch.yml override example - id: browserskill
- insert: config:
- id: browserskill bskPath: bsk
name: "@wxg-prc-cpg/browser-skill-dsh-plugin" defaultTimeoutMs: 120000
config: maxSessions: 5
bskPath: bsk # path to the bsk binary (default: resolve from PATH) observationEnabled: true
defaultTimeoutMs: 120000 thumbnailIntervalMs: 1500
maxSessions: 5 idleIntervalMs: 8000
# observationEnabled: true # live PiP/overlay observation (below) lazyTools: true
# thumbnailIntervalMs: 1500 # frame cadence while a session is active
# idleIntervalMs: 8000 # idle cadence / recent-activity window
# lazyTools: true # reveal browser_* tools only after the skill is invoked
``` ```
- **`lazyTools` (default `true`)** — the final progressive-disclosure stage: the six Change `bskPath` to the full path of your CLI binary if it is not on dsh's `PATH`.
`browser_*` tool schemas stay OUT of the system prompt (zero schema tokens) until the A patch replaces the entry's entire `config` object, so keep all overrides you need
`browser-skill` skill is actually invoked — the skill catalog entry is the only together in that object.
advertisement. One successful invocation (model tool call, or a `/browser-skill` user
gesture) registers the whole suite for the rest of the process; repeated invocations are
no-ops, and sessions resumed with a past invocation in their durable log reveal the suite
on entry. Set `false` for the legacy always-on registration.
## Observation overlay (PiP mini-window) Configuration changes follow `dsh.profile.patchReload` in the profile's
`package.json`: `live` (the default for `web`) applies changes when you save the
patch file; `startup` requires restarting the profile. Restart after upgrading
the plugin in either case.
When the plugin runs inside the dsh Web UI, an **observation overlay** floats over the app All fields are optional; omitted fields use the defaults below:
(registered into the `shell.overlay` seat): a breathing thumbnail per owned session plus its
current action and elapsed time. The card docks at the top-right of the content area (clear of
the composer and the shell's header controls) and wears the **BrowserSkill product family's own
look**: the overlay reuses `@browser-skill/ui` components (`Button`, `cn`) and its oklch design
tokens (`--card`, `--primary`, `--destructive`, `--ring`, …), status dots spec'd after the
extension popup's connection indicator, and Remix icons. The BSK utility sheet is compiled
scoped under the `.bsk-obs` root class (`scripts/build-client-css.mjs`), so the overlay looks
like the BrowserSkill extension without leaking a single selector into the host shell — and the
shell's theme cannot bleed back in.
- **Lifecycle**: hidden while the plugin owns no sessions; appears on the first | Option | Default | Purpose |
`browser_session` start action; disappears when all sessions stop (or the plugin unloads). | --- | --- | --- |
- **Focus view**: status row (green/idle/red dot + session + action + mm:ss), the latest page | `bskPath` | `bsk` | Path to the CLI binary. |
frame (while viewed, refreshes every ~1.5s while active, ~8s when idle; the last good frame stays on | `defaultTimeoutMs` | `120000` | Default command timeout in milliseconds. |
stage while the next one loads, and is kept on errors so the card does not flash), | `maxSessions` | `5` | Maximum concurrent sessions started by this plugin. |
and a compact icon toolbar (Interrupt + Pop out, hover for the label). | `observationEnabled` | `true` | Enable live browser observation. |
- **Interrupt**: one click kills the in-flight bsk command of the focus session (same semantics | `thumbnailIntervalMs` | `1500` | Screenshot interval for active sessions, in milliseconds. |
as the chat Stop button — the current action fails, the agent run may continue). Strip items | `idleIntervalMs` | `8000` | Screenshot interval for idle sessions and the recent-activity window, in milliseconds. |
carry their own hover interrupt button. | `lazyTools` | `true` | Reveal the browser tools when the skill is invoked. Set `false` to register them at startup. |
- **Multi-session strip**: every session gets a tile (mini frame + id + status dot); focus
auto-follows the most recently active session; clicking a tile pins focus (pin badge, click
again to release); errored sessions get a red edge without stealing focus; sessions the daemon
lost are greyed out; prolonged daemon/browser outage shows "browser unavailable" and greys
the interrupt button until captures recover.
- **Drag & resize**: drag the header to move the card; drag any of the four
corners to resize (min 240×180, max 80% of the viewport; no visible grip).
Both are remembered for the page lifetime.
- **Pop out (PiP)**: upgrades the card into a native Document PiP window (requires a user
gesture, per browser rules), sized from the current card; closing the PiP falls back to the
in-page card with state intact. Browsers without Document PiP simply hide the button.
- **Screenshot demand**: the client requests periodic screenshots while an observation view is
visible, using the PiP document's visibility when popped out. Hidden or collapsed views can
keep subscribing to state without requesting screenshots. The first screenshot subscriber
starts capture scheduling; the last one's departure cancels timers and skips queued captures.
An already running capture may finish. Demand applies to the service's owned sessions, not just
the selected session.
- **Wire**: `GET /bsk-observation/events` is the live synchronization channel (SSE). Every connection,
including reconnects, starts with `{ type: "snapshot", sessions, available }`, followed by
`upsert`, `remove`, `reset`, and `availability` events on the same stream. Replace local state
with each snapshot before applying subsequent events. Use `?thumbnails=0` for state-only
subscriptions or `?thumbnails=1` to request screenshots; omitting the parameter defaults to
requesting screenshots for compatibility. Closing the stream releases its screenshot demand.
`GET /bsk-observation/state` remains a one-time `{ sessions, available }` read and does not
request screenshots. It has no ordering guarantee relative to a separate SSE connection;
live clients should initialize and resynchronize from the stream's snapshot instead.
The host also serves `POST /bsk-observation/interrupt`, `POST /bsk-observation/stop`, and
`GET /bsk-observation/thumbnail/<attachmentId>` through the dsh `webServer` route seam
(dsh 0.1's Typert Remote pipeline is closed to out-of-tree packages). All commands for one
session — tool calls and frame captures alike — run through a per-session FIFO, because the
daemon accepts only one unfinished command per session.
- **Programmatic subscription**: `ObservationService.subscribe(listener, { thumbnails: false })`
subscribes to state without requesting screenshots. On an active service, the listener receives
the initial `snapshot` synchronously, before `subscribe` returns, then receives subsequent
changes. `thumbnails` defaults to `true`, so callers should choose it explicitly. The returned
unsubscribe function is safe to call repeatedly and releases this subscription's screenshot
demand. Subscribing after service disposal is a no-op.
- **Trust model**: these routes expose live screenshots (and an interrupt write), so they
replicate the browser-trust fence dsh applies to its own `/api` routes: the request Host
must be a loopback authority (`localhost`, `127.0.0.0/8`, `[::1]`), a present Origin must
match the Host, `sec-fetch-site: cross-site` is refused, and POST requires an
`application/json` body (cross-site simple requests can never satisfy that). The channel is
therefore built for **loopback-only serving** — binding the dsh web server to `0.0.0.0` and
reaching it through a LAN address will (deliberately) fail the fence; do not put these
routes behind a non-loopback reverse proxy without adding your own authentication.
- Configure with `observationEnabled` / `thumbnailIntervalMs` / `idleIntervalMs`.
## Behavior notes With `lazyTools: true`, only the skill's catalog entry is initially advertised to
the model. The six `browser_*` tool schemas are added to the system prompt after
the `browser-skill` skill is successfully invoked, either by the model or through
`/browser-skill`. Set `lazyTools: false` to make the tools available immediately.
- **Cancellation**: aborting a tool call (`exec.signal`) kills the underlying bsk child process, ## Live browser view
matching BrowserSkill's cooperative tool-cancellation model.
- **UI cards**: calls render as terminal cards (command line as title, output as the completed The dsh Web UI shows the plugin's browser sessions in a floating panel. If your
card). Screenshots additionally attach the image itself when the host mounts an attachment store profile provides the `dsh-better-sidebar` integration, the view appears in a
and the active model route declares image input; otherwise the PNG path is returned. **Browser Skill** sidebar tab instead.
- **Web UI toolview (browser half)**: the package is dual-face. `dsh.client` (platform `web`) ships
`lib/client.cjs`, which registers a keyed `tool.call.toolview` view for `browser_inspect`. The - See the current action, elapsed time, and recent screenshot for each session.
custom view keeps a terminal block for every inspect action and, when a screenshot result carries - Select a session to focus on it. The sidebar view follows the current conversation.
an image block, resolves the durable attachment through the client session's authorized - Use **Interrupt** to cancel the current browser command. The agent may continue
`readAttachment` RPC and renders it with the shared `MessageImage` thumbnail/lightbox atoms. with another action afterward.
The other five tools keep the stock terminal card. The bundle follows the dsh client - Drag or resize the floating panel, or use **Pop out** to open a Picture-in-Picture
contract: a CJS closure factory handed to `window.__ModuleLoader__.load`, platform modules window in browsers that support it.
(`react`, `dsh-client-ui-*`) external, everything else inlined, CSS Modules compiled by - Periodic screenshots are requested while a browser observation view is visible.
lightningcss. Configure the active and idle intervals with the options above.
- **Errors**: non-zero bsk exits surface the CLI's JSON error envelope (`code`, `message`, `hint`)
so the model gets the daemon's actionable guidance. The observation endpoints require a loopback address such as `localhost` or
- **Long-running work** (e.g. `bsk record`) is not backgrounded via `ctx.jobs` yet — tracked as a `127.0.0.1`. Access through a LAN hostname or non-loopback reverse proxy is not supported.
follow-up.
## Development ## Development
@@ -195,22 +154,47 @@ pnpm --filter @wxg-prc-cpg/browser-skill-dsh-plugin test # unit tests mock b
pnpm --filter @wxg-prc-cpg/browser-skill-dsh-plugin build # tsdown -> lib/ pnpm --filter @wxg-prc-cpg/browser-skill-dsh-plugin build # tsdown -> lib/
``` ```
See the [development notes](https://github.com/Tencent/BrowserSkill/blob/main/packages/dsh-plugin-browserskill/docs/development.md)
for skill registration, tool results, and observation APIs in the current source.
## Publishing ## Publishing
The GitHub Actions workflow **Release dsh plugin** publishes this package to npm The [Release dsh plugin workflow](https://github.com/Tencent/BrowserSkill/blob/main/.github/workflows/release-dsh-plugin.yml)
as `@wxg-prc-cpg/browser-skill-dsh-plugin`. The Cordis plugin id, client publishes the package and this README to npm. Pushing a `dsh-plugin-vX.Y.Z` tag
bundle registration, and `cordis.patch.yml` name are the same specifier so triggers it; ordinary commits to `main` do not.
dsh can import and materialize the plugin without a ModuleLoader id mismatch.
Trigger it by pushing a tag that matches `package.json`'s `version`: 1. Commit this README and any other changes intended for the release.
2. From the repository root, run the
[release script](https://github.com/Tencent/BrowserSkill/blob/main/scripts/release.mjs)
with a new stable version, replacing `<version>` below. The script updates the
CLI, extension, and DSH plugin versions, commits the version changes, and creates
their release tags:
```sh ```sh
git tag dsh-plugin-v0.1.2 node scripts/release.mjs <version>
git push origin dsh-plugin-v0.1.2 ```
```
Or run the workflow from the Actions tab (`workflow_dispatch`). The job reads the 3. Push the version commit to the release branch, then push the DSH plugin tag
`NPM_TOKEN` secret from the `npm-publish` GitHub Environment. created by the script, using the same `<version>`:
```sh
git push origin HEAD
git push origin dsh-plugin-v<version>
```
This publishes the DSH plugin. Push the CLI and extension tags separately when
those components are ready for release.
The workflow checks the version, runs typechecks and tests, builds the package,
and publishes it to npm.
You can also run the workflow manually from GitHub Actions on the intended release
ref. Both triggers require an unpublished version and the `NPM_TOKEN` secret in the
`npm-publish` GitHub Environment.
npm updates the package README only when a new version is published, including
for documentation-only changes. Published versions cannot be overwritten. See
[npm's README update rules](https://docs.npmjs.com/about-package-readme-files/).
## License ## License
@@ -0,0 +1,90 @@
# DSH plugin development notes
For installation, usage, and configuration, see the [plugin README](../README.md).
These notes describe the current source on this branch; a published npm version may
not include later changes.
## Tool execution
The plugin runs the `bsk` CLI and parses its JSON output. The daemon and browser
extension perform the browser operations. The published package name is also the
Cordis plugin id, client bundle registration id, and `cordis.patch.yml` plugin name.
Keep these identifiers aligned so dsh can load both halves of the plugin.
## Agent skill
Beyond the tools, the plugin publishes the **`browser-skill` agent skill** through the harness's
official skill seam (`ctx.skills.register`): the catalog entry (name + routing description) is
resident in `<available_skills>`, and the body is loaded only when the model invokes the `skill`
tool. Its single source is the DSH-specific `skill/SKILL.md`, which documents only structured
`browser_*` calls and their plugin semantics. The repository-root CLI skill is intentionally not
concatenated: its command examples belong to a different execution interface and would bypass
the plugin's ownership, live observation UI, cancellation, and cleanup path if followed directly.
The build rejects internal CLI-name leakage, command-line code blocks, unknown browser tools, and
missing supported browser tools before embedding the Markdown. Registration and every pre-step
catalog snapshot are pure in-memory reads (no disk/process/daemon); compositions without the skill
seam degrade silently.
With `lazyTools: true`, the six tool schemas stay out of the system prompt until
the skill is successfully invoked. Registration happens once and lasts until the
plugin is unloaded. Repeated invocations do not register duplicate tools. Entering
a resumed conversation whose history contains a successful skill invocation also
registers the tools. Setting `lazyTools: false` registers them at plugin startup.
## Observation subscriptions and routes
- **Screenshot demand**: the client requests periodic screenshots while an observation view is
visible, using the PiP document's visibility when popped out. Hidden or collapsed views can
keep subscribing to state without requesting screenshots. The first screenshot subscriber
starts capture scheduling; the last one's departure cancels timers and skips queued captures.
An already running capture may finish. Demand applies to the service's owned sessions, not just
the selected session.
- **Wire**: `GET /bsk-observation/events` is the live synchronization channel (SSE). Every connection,
including reconnects, starts with `{ type: "snapshot", sessions, available }`, followed by
`upsert`, `remove`, `reset`, and `availability` events on the same stream. Replace local state
with each snapshot before applying subsequent events. Use `?thumbnails=0` for state-only
subscriptions or `?thumbnails=1` to request screenshots; omitting the parameter defaults to
requesting screenshots for compatibility. Closing the stream releases its screenshot demand.
`GET /bsk-observation/state` remains a one-time `{ sessions, available }` read and does not
request screenshots. It has no ordering guarantee relative to a separate SSE connection;
live clients should initialize and resynchronize from the stream's snapshot instead.
The host also serves `POST /bsk-observation/interrupt`, `POST /bsk-observation/stop`, and
`GET /bsk-observation/thumbnail/<attachmentId>` through the dsh `webServer` route seam
(dsh 0.1's Typert Remote pipeline is closed to out-of-tree packages). All commands for one
session — tool calls and frame captures alike — run through a per-session FIFO, because the
daemon accepts only one unfinished command per session.
- **Programmatic subscription**: `ObservationService.subscribe(listener, { thumbnails: false })`
subscribes to state without requesting screenshots. On an active service, the listener receives
the initial `snapshot` synchronously, before `subscribe` returns, then receives subsequent
changes. `thumbnails` defaults to `true`, so callers should choose it explicitly. The returned
unsubscribe function is safe to call repeatedly and releases this subscription's screenshot
demand. Subscribing after service disposal is a no-op.
- **Trust model**: these routes expose live screenshots (and an interrupt write), so they
replicate the browser-trust fence dsh applies to its own `/api` routes: the request Host
must be a loopback authority (`localhost`, `127.0.0.0/8`, `[::1]`), a present Origin must
match the Host, `sec-fetch-site: cross-site` is refused, and POST requires an
`application/json` body (cross-site simple requests can never satisfy that). The channel is
therefore built for **loopback-only serving** — binding the dsh web server to `0.0.0.0` and
reaching it through a LAN address will (deliberately) fail the fence; do not put these
routes behind a non-loopback reverse proxy without adding your own authentication.
## Tool results and cancellation
- **Cancellation**: aborting a tool call (`exec.signal`) kills the underlying bsk child process,
matching BrowserSkill's cooperative tool-cancellation model.
- **UI cards**: calls render as terminal cards (command line as title, output as the completed
card). Screenshots additionally attach the image itself when the host mounts an attachment store
and the active model route declares image input; otherwise the PNG path is returned.
- **Web UI toolview (browser half)**: the package is dual-face. `dsh.client` (platform `web`) ships
`lib/client.cjs`, which registers a keyed `tool.call.toolview` view for `browser_inspect`. The
custom view keeps a terminal block for every inspect action and, when a screenshot result carries
an image block, resolves the durable attachment through the client session's authorized
`readAttachment` RPC and renders it with the shared `MessageImage` thumbnail/lightbox atoms.
The other five tools keep the stock terminal card. The bundle follows the dsh client
contract: a CJS closure factory handed to `window.__ModuleLoader__.load`, platform modules
(`react`, `dsh-client-ui-*`) external, everything else inlined, CSS Modules compiled by
lightningcss.
- **Errors**: non-zero bsk exits surface the CLI's JSON error envelope (`code`, `message`, `hint`)
so the model gets the daemon's actionable guidance.
- **Long-running work** (e.g. `bsk record`) is not backgrounded via `ctx.jobs` yet — tracked as a
follow-up.
@@ -1,7 +1,7 @@
{ {
"name": "@wxg-prc-cpg/browser-skill-dsh-plugin", "name": "@wxg-prc-cpg/browser-skill-dsh-plugin",
"description": "DeepSeek Harness tool plugin that exposes BrowserSkill browser automation (browser_* tools) to the model", "description": "DeepSeek Harness tool plugin that exposes BrowserSkill browser automation (browser_* tools) to the model",
"version": "0.2.0", "version": "0.2.1",
"license": "MIT", "license": "MIT",
"publishConfig": { "publishConfig": {
"access": "public", "access": "public",
+51 -10
View File
@@ -5,7 +5,7 @@
* map the CLI's JSON error envelope onto a thrown `BskError`. * map the CLI's JSON error envelope onto a thrown `BskError`.
*/ */
import { type ChildProcess, spawn } from "node:child_process"; import { type ChildProcess, type SpawnOptionsWithoutStdio, spawn } from "node:child_process";
/** Shape of the JSON error envelope `bsk --json` prints on failure. */ /** Shape of the JSON error envelope `bsk --json` prints on failure. */
export interface BskErrorBody { export interface BskErrorBody {
@@ -53,7 +53,11 @@ export interface BskRunOptions {
} }
/** Minimal spawn signature so tests can substitute a fake child process. */ /** Minimal spawn signature so tests can substitute a fake child process. */
export type SpawnImpl = (command: string, args: string[]) => ChildProcess; export type SpawnImpl = (
command: string,
args: string[],
options?: SpawnOptionsWithoutStdio,
) => ChildProcess;
export interface BskRunner { export interface BskRunner {
/** Run `bsk <args...> --json` and collect its output. */ /** Run `bsk <args...> --json` and collect its output. */
@@ -64,29 +68,66 @@ export interface BskRunner {
killFor(tag: string): number; killFor(tag: string): number;
} }
// Business RPCs translate SIGINT into the daemon's cancel(rpc_id) protocol. // Business RPCs translate Ctrl-C / opt-in stdin EOF into cancel(rpc_id).
// Give that bounded reconciliation path time to settle before the hard kill. // Allow reconciliation before hard-killing an old or unresponsive CLI.
const KILL_GRACE_MS = 3000; const KILL_GRACE_MS = 3000;
// Windows IPC may spend 5s connecting, 2s cancelling, 2s settling,
// and up to 5s releasing the entire batch of caller-owned transfers.
const WINDOWS_KILL_GRACE_MS = 15_000;
const SESSION_BUSY_RETRY_DELAY_MS = 100; const SESSION_BUSY_RETRY_DELAY_MS = 100;
export function createBskRunner(bskPath: string, spawnImpl: SpawnImpl = spawn): BskRunner { export function createBskRunner(bskPath: string, spawnImpl: SpawnImpl = spawn): BskRunner {
const live = new Map<ChildProcess, string | undefined>(); const live = new Map<ChildProcess, string | undefined>();
const windows = process.platform === "win32";
const cancelling = new Set<ChildProcess>();
function killChild(child: ChildProcess): void { function killChild(child: ChildProcess): void {
if (child.exitCode !== null || child.signalCode !== null) return; if (child.exitCode !== null || child.signalCode !== null || cancelling.has(child)) return;
child.kill("SIGINT"); cancelling.add(child);
const force = setTimeout(() => { // Node kills Windows children outright for SIGINT. EOF asks the CLI to
if (child.exitCode === null && child.signalCode === null) child.kill("SIGKILL"); // send its existing cancel RPC and wait for browser reconciliation.
}, KILL_GRACE_MS); if (windows && child.stdin) child.stdin.end();
else child.kill("SIGINT");
const force = setTimeout(
() => {
if (child.exitCode === null && child.signalCode === null) child.kill("SIGKILL");
},
windows ? WINDOWS_KILL_GRACE_MS : KILL_GRACE_MS,
);
force.unref(); force.unref();
child.once("close", () => {
clearTimeout(force);
cancelling.delete(child);
});
} }
return { return {
run(args, options = {}) { run(args, options = {}) {
if (options.signal?.aborted) {
return Promise.resolve({
code: null,
stdout: "",
stderr: "",
timedOut: false,
aborted: true,
});
}
return new Promise<BskRunResult>((resolve, reject) => { return new Promise<BskRunResult>((resolve, reject) => {
let child: ChildProcess; let child: ChildProcess;
try { try {
child = spawnImpl(bskPath, [...args, "--json"]); child = spawnImpl(
bskPath,
[...args, "--json"],
windows
? {
windowsHide: true,
env: { ...process.env, BSK_CANCEL_ON_STDIN_CLOSE: "1" },
}
: undefined,
);
// A child exiting while cancellation closes stdin may report EPIPE.
// Its close/error event remains the authority for the run result.
child.stdin?.on("error", () => {});
} catch (error) { } catch (error) {
reject(error); reject(error);
return; return;
@@ -1,6 +1,7 @@
import type { ChildProcess } from "node:child_process"; import type { ChildProcess } from "node:child_process";
import { EventEmitter } from "node:events"; import { EventEmitter } from "node:events";
import { describe, expect, it } from "vitest"; import { PassThrough } from "node:stream";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { import {
type BskRunResult, type BskRunResult,
createBskRunner, createBskRunner,
@@ -92,6 +93,77 @@ describe("createBskRunner", () => {
}); });
}); });
describe("Windows parent cancellation", () => {
const originalPlatform = Object.getOwnPropertyDescriptor(process, "platform")!;
beforeEach(() => Object.defineProperty(process, "platform", { value: "win32" }));
afterEach(() => {
Object.defineProperty(process, "platform", originalPlatform);
vi.useRealTimers();
});
it("opts in to stdin cancellation and waits for CLI reconciliation", async () => {
const child = Object.assign(new FakeChild(), { stdin: new PassThrough() });
const spawn = vi.fn(() => child as unknown as ChildProcess);
const runner = createBskRunner("bsk", spawn);
const abort = new AbortController();
const result = runner.run(["snapshot"], { signal: abort.signal });
expect(spawn.mock.calls[0]).toEqual([
"bsk",
["snapshot", "--json"],
{
windowsHide: true,
env: { ...process.env, BSK_CANCEL_ON_STDIN_CLOSE: "1" },
},
]);
abort.abort();
expect(child.stdin.writableEnded).toBe(true);
expect(child.killedWith).toEqual([]);
// Closing an already-exited child's pipe must not crash the host.
child.stdin.emit("error", Object.assign(new Error("closed"), { code: "EPIPE" }));
child.finish(2, '{"code":"cancelled"}');
expect(await result).toMatchObject({ code: 2, aborted: true });
});
it("bounds cancellation of an old or unresponsive CLI", async () => {
vi.useFakeTimers();
const child = Object.assign(new FakeChild(), { stdin: new PassThrough() });
const runner = createBskRunner("bsk", fakeSpawn([child]));
const result = runner.run(["snapshot"], { timeoutMs: 5 });
await vi.advanceTimersByTimeAsync(5);
expect(child.stdin.writableEnded).toBe(true);
expect(child.killedWith).toEqual([]);
await vi.advanceTimersByTimeAsync(10_000);
expect(child.killedWith).toEqual([]);
await vi.advanceTimersByTimeAsync(5_000);
expect(child.killedWith).toEqual(["SIGKILL"]);
expect(await result).toMatchObject({ timedOut: true });
});
it("cancels only the requested tag and clears the fallback after exit", async () => {
vi.useFakeTimers();
const a = Object.assign(new FakeChild(), { stdin: new PassThrough() });
const b = Object.assign(new FakeChild(), { stdin: new PassThrough() });
const runner = createBskRunner("bsk", fakeSpawn([a, b]));
const first = runner.run(["snapshot"], { tag: "a" });
const second = runner.run(["snapshot"], { tag: "b" });
expect(runner.killFor("a")).toBe(1);
expect(a.stdin.writableEnded).toBe(true);
expect(b.stdin.writableEnded).toBe(false);
a.finish(2);
b.finish(0);
await Promise.all([first, second]);
expect(vi.getTimerCount()).toBe(0);
});
it("does not start work for an already-aborted call", async () => {
const spawn = vi.fn();
const runner = createBskRunner("bsk", spawn);
const signal = AbortSignal.abort();
expect(await runner.run(["snapshot"], { signal })).toMatchObject({ aborted: true });
expect(spawn).not.toHaveBeenCalled();
});
});
describe("parseBskJson", () => { describe("parseBskJson", () => {
const base: BskRunResult = { code: 0, stdout: "", stderr: "", timedOut: false, aborted: false }; const base: BskRunResult = { code: 0, stdout: "", stderr: "", timedOut: false, aborted: false };
+135
View File
@@ -0,0 +1,135 @@
#Requires -Version 5.1
param([string]$BskPath = (Join-Path $PSScriptRoot "../target/debug/bsk.exe"))
$ErrorActionPreference = "Stop"
# Load definitions, never Main: no downloads or changes to the real user PATH.
$installer = Join-Path $PSScriptRoot "../install.ps1"
$tokens = $null
$errors = $null
$ast = [System.Management.Automation.Language.Parser]::ParseFile($installer, [ref]$tokens, [ref]$errors)
if ($errors.Count) { throw ($errors | Out-String) }
foreach ($statement in $ast.EndBlock.Statements) {
if ($statement -is [System.Management.Automation.Language.FunctionDefinitionAst]) {
$definition = $statement.Extent.Text
if ($statement.Name -eq "Add-ToUserPath") {
$definition = $definition.Replace('[Environment]::GetEnvironmentVariable("PATH", "User")', '$script:UserPath')
$definition = $definition.Replace('[Environment]::SetEnvironmentVariable("PATH", $newUserPath, "User")', '$script:UserPath = $newUserPath')
}
Invoke-Expression $definition
}
}
function Assert-Equal($Actual, $Expected) {
if ($Actual -cne $Expected) { throw "expected [$Expected], got [$Actual]" }
}
function Assert-Fails([scriptblock]$Action) {
$failed = $false
try { & $Action } catch { $failed = $true }
if (-not $failed) { throw "expected failure" }
}
$dir = 'C:\Users\Alice\.local\bin'
foreach ($existing in @('', 'C:\WindowsApps', 'C:\A;C:\B')) {
$script:UserPath = $existing
Add-ToUserPath $dir
$expected = if ($existing) { "$existing;$dir" } else { $dir }
Assert-Equal $script:UserPath $expected
Add-ToUserPath $dir
Assert-Equal $script:UserPath $expected
}
$root = Join-Path ([IO.Path]::GetTempPath()) ("bsk-install-test-" + [Guid]::NewGuid().ToString('N'))
[IO.Directory]::CreateDirectory($root) | Out-Null
$oldBskHome = $env:BSK_HOME
$oldAutoUpdate = $env:BSK_AUTO_UPDATE
$daemon = $null
try {
$utf8 = New-Object System.Text.UTF8Encoding($false)
$bashRc = Join-Path $root ".bashrc"
$existing = "# existing 中文 configuration" + [char]10
[IO.File]::WriteAllText($bashRc, $existing, $utf8)
$special = 'C:\Users\张三 space $HOME $(echo unsafe) ! & [x] O''Brien\.local\bin'
Add-ToBashProfile $special $bashRc
$first = [IO.File]::ReadAllText($bashRc)
if (-not $first.StartsWith($existing)) { throw "existing bashrc content changed" }
if (-not $first.Contains('张三')) { throw "Unicode path was lost" }
Add-ToBashProfile $special $bashRc
Assert-Equal ([IO.File]::ReadAllText($bashRc)) $first
$bytes = [IO.File]::ReadAllBytes($bashRc)
if ($bytes[0] -eq 0xEF) { throw "unexpected UTF-8 BOM" }
$bash = [IO.Path]::GetFullPath((Join-Path (Split-Path (Get-Command git).Source -Parent) "../bin/bash.exe"))
if (-not (Test-Path -LiteralPath $bash)) { throw "Git Bash is required for quoting regression" }
$env:BSK_TEST_RC = $bashRc
$probe = Join-Path $root 'probe.sh'
[IO.File]::WriteAllText($probe, 'source "$BSK_TEST_RC"; printf "%s\n" "${PATH%%:*}"', $utf8)
$actual = & $bash --noprofile --norc $probe
if ($LASTEXITCODE -ne 0) { throw "Git Bash failed" }
Assert-Equal $actual ('/c' + $special.Substring(2).Replace('\', '/'))
$env:BSK_HOME = Join-Path $root "home"
$env:BSK_AUTO_UPDATE = "off"
[IO.Directory]::CreateDirectory($env:BSK_HOME) | Out-Null
$sourceDir = Join-Path $root "download"
[IO.Directory]::CreateDirectory($sourceDir) | Out-Null
$source = Join-Path $sourceDir "bsk.exe"
[IO.File]::Copy((Resolve-Path -LiteralPath $BskPath).Path, $source)
$targetDir = Join-Path $root "中文 space [x] & install"
[IO.Directory]::CreateDirectory($targetDir) | Out-Null
$target = Join-Path $targetDir "bsk.exe"
Install-Binary $source $target
Assert-Equal (Get-FileHash -LiteralPath $target).Hash (Get-FileHash -LiteralPath $source).Hash
$daemon = Start-Process -FilePath $target -ArgumentList @('daemon', 'start', '--foreground', '--port', '0') -WindowStyle Hidden -PassThru
$infoPath = Join-Path $env:BSK_HOME "daemon.json"
$deadline = [DateTime]::UtcNow.AddSeconds(15)
while (-not [IO.File]::Exists($infoPath)) {
if ($daemon.HasExited -or [DateTime]::UtcNow -gt $deadline) { throw "daemon failed to start" }
Start-Sleep -Milliseconds 50
}
# PE overlays distinguish old/new executables without another build.
$stream = [IO.File]::OpenWrite($source)
try {
$stream.Seek(0, [IO.SeekOrigin]::End) | Out-Null
$marker = $utf8.GetBytes("installer replacement regression")
$stream.Write($marker, 0, $marker.Length)
} finally { $stream.Dispose() }
Install-Binary $source $target
if (-not $daemon.WaitForExit(5000)) { throw "old daemon still running" }
Assert-Equal (Get-FileHash -LiteralPath $target).Hash (Get-FileHash -LiteralPath $source).Hash
& $target --version
if ($LASTEXITCODE -ne 0) { throw "replacement is not executable" }
# Refuse replacement when daemon identity cannot be verified.
[IO.File]::WriteAllText($infoPath, 'invalid daemon metadata')
$before = (Get-FileHash -LiteralPath $target).Hash
Assert-Fails { Install-Binary $source $target }
Assert-Equal (Get-FileHash -LiteralPath $target).Hash $before
[IO.File]::Delete($infoPath)
# A remaining lock must fail without truncation or staging debris.
$lock = [IO.File]::Open($target, [IO.FileMode]::Open, [IO.FileAccess]::Read, [IO.FileShare]::None)
try { Assert-Fails { Install-Binary $source $target } } finally { $lock.Dispose() }
Assert-Equal (Get-FileHash -LiteralPath $target).Hash $before
if (@(Get-ChildItem -LiteralPath $targetDir -Filter "*.install-*").Count) { throw "staging files leaked" }
Write-Host "Windows installer regressions passed ($($PSVersionTable.PSVersion))"
}
catch {
Write-Host ($_ | Out-String)
Write-Host $_.ScriptStackTrace
throw
}
finally {
if ($daemon -and -not $daemon.HasExited) {
Stop-Process -Id $daemon.Id -Force
$daemon.WaitForExit(5000) | Out-Null
}
$env:BSK_HOME = $oldBskHome
$env:BSK_AUTO_UPDATE = $oldAutoUpdate
Remove-Item Env:BSK_TEST_RC -ErrorAction SilentlyContinue
# Resolve and verify before recursive cleanup; only this fixture is removed.
$resolvedRoot = [IO.Path]::GetFullPath($root)
$tempRoot = [IO.Path]::GetFullPath([IO.Path]::GetTempPath()).TrimEnd('\') + '\'
if (-not $resolvedRoot.StartsWith($tempRoot, [StringComparison]::OrdinalIgnoreCase) -or
-not ([IO.Path]::GetFileName($resolvedRoot)).StartsWith('bsk-install-test-')) { throw "invalid cleanup path" }
Remove-Item -LiteralPath $resolvedRoot -Recurse -Force
}
+9 -3
View File
@@ -174,16 +174,22 @@ function updateChangelog(version) {
const today = new Date().toISOString().slice(0, 10); const today = new Date().toISOString().slice(0, 10);
const escapedVer = version.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); const escapedVer = version.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
const versionHeading = new RegExp(`^## \\[${escapedVer}\\](?:\\s|$)`, "m");
// First: fill in any date-placeholder line for this version (e.g. "## [0.2.0] - 2026-09-XX") // First: fill in any date-placeholder line for this version (e.g. "## [0.2.0] - 2026-09-XX")
const versionPlaceholder = new RegExp(`^## \\[${escapedVer}\\]\\s*-\\s*\\d{4}-\\d{2}-XX`, "m"); const versionPlaceholder = new RegExp(`^## \\[${escapedVer}\\]\\s*-\\s*\\d{4}-\\d{2}-XX`, "m");
if (versionPlaceholder.test(content)) { if (versionPlaceholder.test(content)) {
content = content.replace(versionPlaceholder, `## [${version}] - ${today}`); content = content.replace(versionPlaceholder, `## [${version}] - ${today}`);
} else { } else if (!versionHeading.test(content)) {
// Otherwise: insert a new section after [Unreleased] // Preserve prepared release entries; new versions go before the release history.
const heading = `## [${version}] - ${today}`;
const unreleased = /^## \[Unreleased\]/m; const unreleased = /^## \[Unreleased\]/m;
if (unreleased.test(content)) { if (unreleased.test(content)) {
content = content.replace(unreleased, `## [Unreleased]\n\n## [${version}] - ${today}`); content = content.replace(unreleased, `## [Unreleased]\n\n${heading}`);
} else if (/^## /m.test(content)) {
content = content.replace(/^## /m, `${heading}\n\n## `);
} else {
content = `${content.trimEnd()}\n\n${heading}\n`;
} }
} }