mirror of
https://github.com/Tencent/BrowserSkill.git
synced 2026-09-10 13:01:23 +08:00
Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8d7c0aa66c | ||
|
|
a9489282fa | ||
|
|
d4cddd6224 | ||
|
|
8e4de6cf4b | ||
|
|
fea66ce404 | ||
|
|
8bae85127a | ||
|
|
c8c5c4a7c4 |
@@ -13,7 +13,7 @@
|
|||||||
</p>
|
</p>
|
||||||
|
|
||||||
**BrowserSkill** connects Cursor, Claude Code, Codex, OpenClaw, CodeBuddy,
|
**BrowserSkill** connects Cursor, Claude Code, Codex, OpenClaw, CodeBuddy,
|
||||||
WorkBuddy, Pi, Hermes Agent, and other shell-capable AI agents to your already logged-in
|
WorkBuddy, Pi, Hermes Agent, DeepSeek Harness, and other AI agents to your already logged-in
|
||||||
browser.
|
browser.
|
||||||
|
|
||||||
Need the agent to touch a tab you already have open? It must borrow that tab
|
Need the agent to touch a tab you already have open? It must borrow that tab
|
||||||
@@ -123,7 +123,8 @@ internal variants and install paths.
|
|||||||
|
|
||||||
Other shell-capable agent harnesses are supported too. Copy
|
Other shell-capable agent harnesses are supported too. Copy
|
||||||
[`skill/SKILL.md`](skill/SKILL.md) into your harness's skills directory as
|
[`skill/SKILL.md`](skill/SKILL.md) into your harness's skills directory as
|
||||||
`browser-skill/SKILL.md` to install the skill manually.
|
`browser-skill/SKILL.md` to install the skill manually. DeepSeek Harness uses a
|
||||||
|
dedicated plugin instead — see [DeepSeek Harness plugin](#deepseek-harness-plugin).
|
||||||
|
|
||||||
</details>
|
</details>
|
||||||
|
|
||||||
@@ -133,6 +134,16 @@ Start a new Agent session and write a prompt that needs the browser, for example
|
|||||||
/browser-skill open example.com and summarize what is on the page.
|
/browser-skill open example.com and summarize what is on the page.
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## DeepSeek Harness plugin
|
||||||
|
|
||||||
|
A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) plugin that
|
||||||
|
injects native `browser_*` tools (no shelling out to `bsk`) and a live Web UI
|
||||||
|
overlay of each Agent Window.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
dsh plugin --profile web add @wxg-prc-cpg/browser-skill-dsh-plugin
|
||||||
|
```
|
||||||
|
|
||||||
## How It Works
|
## How It Works
|
||||||
|
|
||||||
BrowserSkill is a local bridge between your agent harness and your browser.
|
BrowserSkill is a local bridge between your agent harness and your browser.
|
||||||
@@ -166,7 +177,9 @@ flowchart TB
|
|||||||
|
|
||||||
The agent never talks to the browser directly. It asks the `bsk` CLI to perform a
|
The agent never talks to the browser directly. It asks the `bsk` CLI to perform a
|
||||||
browser task; the local daemon routes that request to the extension; the
|
browser task; the local daemon routes that request to the extension; the
|
||||||
extension runs it in an Agent Window.
|
extension runs it in an Agent Window. DeepSeek Harness takes the same path
|
||||||
|
through the [plugin](#deepseek-harness-plugin): the agent calls injected
|
||||||
|
`browser_*` tools, and the plugin invokes `bsk` on its behalf.
|
||||||
|
|
||||||
## For Developers
|
## For Developers
|
||||||
|
|
||||||
@@ -176,6 +189,7 @@ The repository is a Cargo + pnpm workspace:
|
|||||||
- `crates/bsk-protocol` — shared wire types and JSON schemas
|
- `crates/bsk-protocol` — shared wire types and JSON schemas
|
||||||
- `apps/extension` — browser extension
|
- `apps/extension` — browser extension
|
||||||
- `packages/ui` and `packages/i18n` — shared extension UI support
|
- `packages/ui` and `packages/i18n` — shared extension UI support
|
||||||
|
- `packages/dsh-plugin-browserskill` — DeepSeek Harness plugin (`@wxg-prc-cpg/browser-skill-dsh-plugin`)
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
|
|||||||
+12
-3
@@ -12,7 +12,7 @@
|
|||||||
<a href="README.md">English</a> · 中文
|
<a href="README.md">English</a> · 中文
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
**BrowserSkill** 把 Cursor、Claude Code、Codex、OpenClaw、CodeBuddy、WorkBuddy、Pi、Hermes Agent 等支持 Shell 的 AI Agent 连接到你已登录的浏览器。
|
**BrowserSkill** 把 Cursor、Claude Code、Codex、OpenClaw、CodeBuddy、WorkBuddy、Pi、Hermes Agent、DeepSeek Harness 等 AI Agent 连接到你已登录的浏览器。
|
||||||
|
|
||||||
需要 Agent 操作你已打开的标签页?必须显式借用该标签,任务结束后归还,其余浏览器窗口不受影响。
|
需要 Agent 操作你已打开的标签页?必须显式借用该标签,任务结束后归还,其余浏览器窗口不受影响。
|
||||||
|
|
||||||
@@ -102,7 +102,7 @@ bsk install-skill
|
|||||||
|
|
||||||
用 <kbd>Space</kbd> 选择需要安装的 Agent harness,然后按 <kbd>Enter</kbd> 安装 skill。运行 `bsk install-skill --list` 可查看 internal 变体及安装路径。
|
用 <kbd>Space</kbd> 选择需要安装的 Agent harness,然后按 <kbd>Enter</kbd> 安装 skill。运行 `bsk install-skill --list` 可查看 internal 变体及安装路径。
|
||||||
|
|
||||||
其他支持 Shell 的 Agent harness 也可使用 BrowserSkill,但需手动将 [`skill/SKILL.md`](skill/SKILL.md) 复制到对应 skills 目录下的 `browser-skill/SKILL.md`。
|
其他支持 Shell 的 Agent harness 也可使用 BrowserSkill,但需手动将 [`skill/SKILL.md`](skill/SKILL.md) 复制到对应 skills 目录下的 `browser-skill/SKILL.md`。DeepSeek Harness 走独立插件,见 [DeepSeek Harness 插件](#deepseek-harness-插件)。
|
||||||
|
|
||||||
</details>
|
</details>
|
||||||
|
|
||||||
@@ -112,6 +112,14 @@ bsk install-skill
|
|||||||
/browser-skill open example.com and summarize what is on the page.
|
/browser-skill open example.com and summarize what is on the page.
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## DeepSeek Harness 插件
|
||||||
|
|
||||||
|
[DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) 插件:注入原生 `browser_*` 工具(无需再通过 Shell 调用 `bsk`),并在 Web UI 中实时观察每个 Agent Window。
|
||||||
|
|
||||||
|
```sh
|
||||||
|
dsh plugin --profile web add @wxg-prc-cpg/browser-skill-dsh-plugin
|
||||||
|
```
|
||||||
|
|
||||||
## 工作原理
|
## 工作原理
|
||||||
|
|
||||||
BrowserSkill 是 Agent 运行时与浏览器之间的本地桥接层。
|
BrowserSkill 是 Agent 运行时与浏览器之间的本地桥接层。
|
||||||
@@ -143,7 +151,7 @@ flowchart TB
|
|||||||
style UserWindows fill:#f8fafc,stroke:#cbd5e1,color:#334155
|
style UserWindows fill:#f8fafc,stroke:#cbd5e1,color:#334155
|
||||||
```
|
```
|
||||||
|
|
||||||
Agent 不直接与浏览器通信。它通过 `bsk` CLI 下发浏览器任务;本地 daemon 把请求路由到扩展;扩展在 Agent Window 中执行。
|
Agent 不直接与浏览器通信。它通过 `bsk` CLI 下发浏览器任务;本地 daemon 把请求路由到扩展;扩展在 Agent Window 中执行。DeepSeek Harness 走同一条链路,只是经由 [插件](#deepseek-harness-插件):Agent 调用注入的 `browser_*` 工具,由插件代为执行 `bsk`。
|
||||||
|
|
||||||
## 面向开发者
|
## 面向开发者
|
||||||
|
|
||||||
@@ -153,6 +161,7 @@ Agent 不直接与浏览器通信。它通过 `bsk` CLI 下发浏览器任务;
|
|||||||
- `crates/bsk-protocol` — 共享协议类型与 JSON Schema
|
- `crates/bsk-protocol` — 共享协议类型与 JSON Schema
|
||||||
- `apps/extension` — 浏览器扩展
|
- `apps/extension` — 浏览器扩展
|
||||||
- `packages/ui` 和 `packages/i18n` — 扩展 UI 共享支持
|
- `packages/ui` 和 `packages/i18n` — 扩展 UI 共享支持
|
||||||
|
- `packages/dsh-plugin-browserskill` — DeepSeek Harness 插件(`@wxg-prc-cpg/browser-skill-dsh-plugin`)
|
||||||
|
|
||||||
## 许可证
|
## 许可证
|
||||||
|
|
||||||
|
|||||||
@@ -104,6 +104,27 @@ describe("reduceTraceSteps", () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("keeps committed empty fill steps", () => {
|
||||||
|
const { steps } = reduceTraceSteps(
|
||||||
|
[
|
||||||
|
{
|
||||||
|
op: "fill",
|
||||||
|
target: { tag: "input", role: "textbox", name: "Search query" },
|
||||||
|
value: "",
|
||||||
|
page_url: "https://example.com/search",
|
||||||
|
},
|
||||||
|
],
|
||||||
|
"https://example.com/search",
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(steps).toEqual([
|
||||||
|
expect.objectContaining({
|
||||||
|
op: "fill",
|
||||||
|
value: "",
|
||||||
|
}),
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
it("maps select navigated_to onto effect.navigated_to (page id)", () => {
|
it("maps select navigated_to onto effect.navigated_to (page id)", () => {
|
||||||
const { pages, steps } = reduceTraceSteps(
|
const { pages, steps } = reduceTraceSteps(
|
||||||
[
|
[
|
||||||
|
|||||||
@@ -18,7 +18,6 @@ export function shouldRecordPress(
|
|||||||
}
|
}
|
||||||
|
|
||||||
function shouldIncludeDraft(step: DraftTraceStep): boolean {
|
function shouldIncludeDraft(step: DraftTraceStep): boolean {
|
||||||
if (step.op === "fill" && !(step.value ?? "").trim() && !step.redacted) return false;
|
|
||||||
if (step.op === "press" && !shouldRecordPress(step.key, step.modifiers)) return false;
|
if (step.op === "press" && !shouldRecordPress(step.key, step.modifiers)) return false;
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import { OVERLAY_HOST_MARKER_ATTR, OVERLAY_HOST_NAME } from "@/lib/overlay-bridg
|
|||||||
import { SessionManager } from "@/session-manager/manager";
|
import { SessionManager } from "@/session-manager/manager";
|
||||||
import type { CdpRunner } from "@/tools/shared";
|
import type { CdpRunner } from "@/tools/shared";
|
||||||
import {
|
import {
|
||||||
|
buildFrameVomScene,
|
||||||
buildVomScene,
|
buildVomScene,
|
||||||
type CdpAxNode,
|
type CdpAxNode,
|
||||||
handleGetHtml,
|
handleGetHtml,
|
||||||
@@ -508,6 +509,284 @@ describe("handleScreenshot overlay suppression", () => {
|
|||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
describe("buildVomScene", () => {
|
describe("buildVomScene", () => {
|
||||||
|
it("merges multiple sibling and nested frames without mixing their targets", () => {
|
||||||
|
const node = (
|
||||||
|
backendNodeId: number,
|
||||||
|
parentBackendNodeId: number | null,
|
||||||
|
frameId: string,
|
||||||
|
tag: string,
|
||||||
|
attrs: Record<string, string> = {},
|
||||||
|
): CapturedNode => ({
|
||||||
|
backendNodeId,
|
||||||
|
parentBackendNodeId,
|
||||||
|
frameId,
|
||||||
|
tag,
|
||||||
|
attrs,
|
||||||
|
rect: { x: 0, y: 0, w: 100, h: 30 },
|
||||||
|
localRect: { x: 0, y: 0, w: 100, h: 30 },
|
||||||
|
paintOrder: 1,
|
||||||
|
position: "static",
|
||||||
|
pointerEvents: "auto",
|
||||||
|
});
|
||||||
|
const mainNodes = [
|
||||||
|
node(1, null, "main", "body"),
|
||||||
|
node(10, 1, "main", "iframe", { title: "First" }),
|
||||||
|
node(20, 1, "main", "iframe", { title: "Second" }),
|
||||||
|
];
|
||||||
|
const firstNodes = [
|
||||||
|
node(100, null, "first", "body"),
|
||||||
|
node(101, 100, "first", "button", { "aria-label": "First action" }),
|
||||||
|
];
|
||||||
|
const secondNodes = [
|
||||||
|
node(200, null, "second", "body"),
|
||||||
|
node(201, 200, "second", "h2"),
|
||||||
|
node(210, 200, "second", "iframe", { title: "Nested" }),
|
||||||
|
];
|
||||||
|
const nestedNodes = [
|
||||||
|
node(300, null, "nested", "body"),
|
||||||
|
node(301, 300, "nested", "input", { placeholder: "Nested field" }),
|
||||||
|
];
|
||||||
|
const captured: CapturedViewModel = {
|
||||||
|
viewport: { width: 1200, height: 800 },
|
||||||
|
nodes: mainNodes,
|
||||||
|
iframeNodes: new Map([
|
||||||
|
[10, firstNodes],
|
||||||
|
[20, secondNodes],
|
||||||
|
[210, nestedNodes],
|
||||||
|
]),
|
||||||
|
frameNodes: new Map([
|
||||||
|
["main", mainNodes],
|
||||||
|
["first", firstNodes],
|
||||||
|
["second", secondNodes],
|
||||||
|
["nested", nestedNodes],
|
||||||
|
]),
|
||||||
|
frameOwnerBackendNodeIds: new Map([
|
||||||
|
["first", 10],
|
||||||
|
["second", 20],
|
||||||
|
["nested", 210],
|
||||||
|
]),
|
||||||
|
rootFrameId: "main",
|
||||||
|
excludedBackendNodeIds: new Set(),
|
||||||
|
};
|
||||||
|
const axNode = (
|
||||||
|
nodeId: string,
|
||||||
|
backendDOMNodeId: number,
|
||||||
|
role: string,
|
||||||
|
parentId?: string,
|
||||||
|
name?: string,
|
||||||
|
): CdpAxNode => ({
|
||||||
|
nodeId,
|
||||||
|
backendDOMNodeId,
|
||||||
|
role: { type: "role", value: role },
|
||||||
|
...(parentId ? { parentId } : {}),
|
||||||
|
...(name ? { name: { type: "computedString", value: name } } : {}),
|
||||||
|
});
|
||||||
|
const result = buildFrameVomScene(
|
||||||
|
[
|
||||||
|
{
|
||||||
|
frameId: "main",
|
||||||
|
contextScopeId: "main",
|
||||||
|
domNodes: mainNodes,
|
||||||
|
axNodes: [
|
||||||
|
axNode("root", 1, "RootWebArea"),
|
||||||
|
axNode("frame-1", 10, "Iframe", "root", "First"),
|
||||||
|
axNode("frame-2", 20, "Iframe", "root", "Second"),
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
frameId: "first",
|
||||||
|
contextScopeId: "first",
|
||||||
|
parentFrameId: "main",
|
||||||
|
ownerBackendNodeId: 10,
|
||||||
|
domNodes: firstNodes,
|
||||||
|
axNodes: [
|
||||||
|
axNode("root", 100, "RootWebArea"),
|
||||||
|
axNode("button", 101, "button", "root", "First action"),
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
frameId: "second",
|
||||||
|
contextScopeId: "second",
|
||||||
|
parentFrameId: "main",
|
||||||
|
ownerBackendNodeId: 20,
|
||||||
|
domNodes: secondNodes,
|
||||||
|
axNodes: [
|
||||||
|
axNode("root", 200, "RootWebArea"),
|
||||||
|
axNode("heading", 201, "heading", "root", "Second heading"),
|
||||||
|
axNode("nested-frame", 210, "Iframe", "root", "Nested"),
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
frameId: "nested",
|
||||||
|
contextScopeId: "nested",
|
||||||
|
parentFrameId: "second",
|
||||||
|
ownerBackendNodeId: 210,
|
||||||
|
domNodes: nestedNodes,
|
||||||
|
axNodes: [
|
||||||
|
axNode("root", 300, "RootWebArea"),
|
||||||
|
axNode("input", 301, "textbox", "root", "Nested field"),
|
||||||
|
],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
captured,
|
||||||
|
);
|
||||||
|
|
||||||
|
const rendered = renderVom(result);
|
||||||
|
expect(rendered.text).toContain('Iframe "First"');
|
||||||
|
expect(rendered.text).toContain('@e1 button "First action"');
|
||||||
|
expect(rendered.text).toContain('heading "Second heading"');
|
||||||
|
expect(rendered.text).toContain('Iframe "Nested"');
|
||||||
|
expect(rendered.text).toContain('textbox "Nested field"');
|
||||||
|
expect(rendered.refs.find((ref) => ref.backendNodeId === 101)?.frameId).toBe("first");
|
||||||
|
expect(rendered.refs.find((ref) => ref.backendNodeId === 301)?.frameId).toBe("nested");
|
||||||
|
|
||||||
|
const firstAction = result.nodes.find((item) => item.backendNodeId === 101);
|
||||||
|
const nestedInput = result.nodes.find((item) => item.backendNodeId === 301);
|
||||||
|
expect(firstAction).toMatchObject({ frameId: "first", contextScopeId: "first" });
|
||||||
|
expect(nestedInput).toMatchObject({ frameId: "nested", contextScopeId: "nested" });
|
||||||
|
expect(nestedInput?.parentId).toBe(result.nodes.find((item) => item.backendNodeId === 210)?.id);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not place a child document at the page root when its frame boundary is unresolved", () => {
|
||||||
|
const childNode: CapturedNode = {
|
||||||
|
backendNodeId: 101,
|
||||||
|
parentBackendNodeId: null,
|
||||||
|
frameId: "child",
|
||||||
|
tag: "button",
|
||||||
|
attrs: {},
|
||||||
|
rect: { x: 0, y: 0, w: 100, h: 30 },
|
||||||
|
paintOrder: 1,
|
||||||
|
position: "static",
|
||||||
|
pointerEvents: "auto",
|
||||||
|
};
|
||||||
|
const captured: CapturedViewModel = {
|
||||||
|
viewport: { width: 800, height: 600 },
|
||||||
|
nodes: [],
|
||||||
|
iframeNodes: new Map(),
|
||||||
|
frameNodes: new Map([["child", [childNode]]]),
|
||||||
|
rootFrameId: "main",
|
||||||
|
excludedBackendNodeIds: new Set(),
|
||||||
|
};
|
||||||
|
|
||||||
|
const result = buildFrameVomScene(
|
||||||
|
[
|
||||||
|
{
|
||||||
|
frameId: "main",
|
||||||
|
contextScopeId: "main",
|
||||||
|
domNodes: [],
|
||||||
|
axNodes: [],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
frameId: "child",
|
||||||
|
contextScopeId: "child",
|
||||||
|
parentFrameId: "main",
|
||||||
|
domNodes: [childNode],
|
||||||
|
axNodes: [
|
||||||
|
{
|
||||||
|
nodeId: "button",
|
||||||
|
frameId: "child",
|
||||||
|
backendDOMNodeId: 101,
|
||||||
|
role: { type: "role", value: "button" },
|
||||||
|
name: { type: "computedString", value: "Child action" },
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
captured,
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(result.nodes).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps AX-only frame content attached to its iframe boundary", () => {
|
||||||
|
const mainNodes: CapturedNode[] = [
|
||||||
|
{
|
||||||
|
backendNodeId: 1,
|
||||||
|
parentBackendNodeId: null,
|
||||||
|
frameId: "main",
|
||||||
|
tag: "body",
|
||||||
|
attrs: {},
|
||||||
|
rect: { x: 0, y: 0, w: 800, h: 600 },
|
||||||
|
paintOrder: 0,
|
||||||
|
position: "static",
|
||||||
|
pointerEvents: "auto",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
backendNodeId: 10,
|
||||||
|
parentBackendNodeId: 1,
|
||||||
|
frameId: "main",
|
||||||
|
tag: "iframe",
|
||||||
|
attrs: { title: "Remote" },
|
||||||
|
rect: { x: 100, y: 100, w: 400, h: 300 },
|
||||||
|
paintOrder: 1,
|
||||||
|
position: "static",
|
||||||
|
pointerEvents: "auto",
|
||||||
|
},
|
||||||
|
];
|
||||||
|
const captured: CapturedViewModel = {
|
||||||
|
viewport: { width: 800, height: 600 },
|
||||||
|
nodes: mainNodes,
|
||||||
|
iframeNodes: new Map(),
|
||||||
|
frameNodes: new Map([
|
||||||
|
["main", mainNodes],
|
||||||
|
["child", []],
|
||||||
|
]),
|
||||||
|
rootFrameId: "main",
|
||||||
|
excludedBackendNodeIds: new Set(),
|
||||||
|
};
|
||||||
|
|
||||||
|
const result = buildFrameVomScene(
|
||||||
|
[
|
||||||
|
{
|
||||||
|
frameId: "main",
|
||||||
|
contextScopeId: "main",
|
||||||
|
domNodes: mainNodes,
|
||||||
|
axNodes: [
|
||||||
|
{
|
||||||
|
nodeId: "main-root",
|
||||||
|
backendDOMNodeId: 1,
|
||||||
|
role: { type: "role", value: "RootWebArea" },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
nodeId: "frame-owner",
|
||||||
|
parentId: "main-root",
|
||||||
|
backendDOMNodeId: 10,
|
||||||
|
role: { type: "role", value: "Iframe" },
|
||||||
|
name: { type: "computedString", value: "Remote" },
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
frameId: "child",
|
||||||
|
contextScopeId: "child",
|
||||||
|
parentFrameId: "main",
|
||||||
|
ownerBackendNodeId: 10,
|
||||||
|
domNodes: [],
|
||||||
|
axNodes: [
|
||||||
|
{
|
||||||
|
nodeId: "child-root",
|
||||||
|
backendDOMNodeId: 100,
|
||||||
|
role: { type: "role", value: "RootWebArea" },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
nodeId: "child-button",
|
||||||
|
parentId: "child-root",
|
||||||
|
backendDOMNodeId: 101,
|
||||||
|
role: { type: "role", value: "button" },
|
||||||
|
name: { type: "computedString", value: "AX fallback action" },
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
captured,
|
||||||
|
);
|
||||||
|
|
||||||
|
const childButton = result.nodes.find((node) => node.backendNodeId === 101);
|
||||||
|
expect(childButton).toMatchObject({ frameId: "child", contextScopeId: "child" });
|
||||||
|
expect(childButton?.parentId).toBe(result.nodes.find((node) => node.backendNodeId === 10)?.id);
|
||||||
|
expect(renderVom(result).text).toContain('@e1 button "AX fallback action"');
|
||||||
|
});
|
||||||
|
|
||||||
it("joins AX semantics with captured geometry by backendDOMNodeId", () => {
|
it("joins AX semantics with captured geometry by backendDOMNodeId", () => {
|
||||||
const axNodes: CdpAxNode[] = [
|
const axNodes: CdpAxNode[] = [
|
||||||
{
|
{
|
||||||
@@ -816,6 +1095,7 @@ describe("buildVomScene", () => {
|
|||||||
parentId: "2",
|
parentId: "2",
|
||||||
role: { type: "role", value: "textbox" },
|
role: { type: "role", value: "textbox" },
|
||||||
name: { type: "x", value: "输入密码" },
|
name: { type: "x", value: "输入密码" },
|
||||||
|
value: { value: "iframe-secret" },
|
||||||
backendDOMNodeId: 202,
|
backendDOMNodeId: 202,
|
||||||
properties: [{ name: "inputType", value: { value: "password" } }],
|
properties: [{ name: "inputType", value: { value: "password" } }],
|
||||||
},
|
},
|
||||||
@@ -870,6 +1150,10 @@ describe("buildVomScene", () => {
|
|||||||
|
|
||||||
expect(passwordNodes).toHaveLength(1);
|
expect(passwordNodes).toHaveLength(1);
|
||||||
expect(passwordNodes[0]).toEqual(expect.objectContaining({ id: 202, sensitive: true }));
|
expect(passwordNodes[0]).toEqual(expect.objectContaining({ id: 202, sensitive: true }));
|
||||||
|
expect(passwordNodes[0].value).toBeUndefined();
|
||||||
|
const rendered = renderVom(buildVomScene(axNodes, captured)).text;
|
||||||
|
expect(rendered).toContain('textbox "输入密码" [filled] ="•••"');
|
||||||
|
expect(rendered).not.toContain("iframe-secret");
|
||||||
});
|
});
|
||||||
|
|
||||||
it("keeps unnamed iframe controls but skips unnamed iframe links", () => {
|
it("keeps unnamed iframe controls but skips unnamed iframe links", () => {
|
||||||
|
|||||||
@@ -50,6 +50,7 @@ import {
|
|||||||
captureViewModel,
|
captureViewModel,
|
||||||
collectOverlayExcludedBackendIds,
|
collectOverlayExcludedBackendIds,
|
||||||
} from "./vom/capture";
|
} from "./vom/capture";
|
||||||
|
import { type CapturedFrameDocument, captureFrameData } from "./vom/frame-capture";
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// Shared helpers (legacy aliases — observation.ts kept exporting these
|
// Shared helpers (legacy aliases — observation.ts kept exporting these
|
||||||
@@ -352,6 +353,7 @@ export type CdpRunner = SharedCdpRunner;
|
|||||||
/** Subset of CDP `AXNode` we care about — see `Accessibility.AXNode`. */
|
/** Subset of CDP `AXNode` we care about — see `Accessibility.AXNode`. */
|
||||||
export interface CdpAxNode {
|
export interface CdpAxNode {
|
||||||
nodeId: string;
|
nodeId: string;
|
||||||
|
frameId?: string;
|
||||||
parentId?: string;
|
parentId?: string;
|
||||||
backendDOMNodeId?: number;
|
backendDOMNodeId?: number;
|
||||||
ignored?: boolean;
|
ignored?: boolean;
|
||||||
@@ -584,7 +586,7 @@ function iframeNameFor(node: CapturedNode): string | undefined {
|
|||||||
function isRenderableIframeControl(node: CapturedNode): boolean {
|
function isRenderableIframeControl(node: CapturedNode): boolean {
|
||||||
const tag = normalizeTag(node.tag);
|
const tag = normalizeTag(node.tag);
|
||||||
if (!IFRAME_RENDERABLE_TAGS.has(tag)) return false;
|
if (!IFRAME_RENDERABLE_TAGS.has(tag)) return false;
|
||||||
if (!node.rect) return false;
|
if (!node.localRect && !node.rect) return false;
|
||||||
if (node.pointerEvents === "none") return false;
|
if (node.pointerEvents === "none") return false;
|
||||||
if ((node.attrs.type ?? "").toLowerCase() === "hidden") return false;
|
if ((node.attrs.type ?? "").toLowerCase() === "hidden") return false;
|
||||||
|
|
||||||
@@ -756,7 +758,12 @@ function applyCapturedSignals(
|
|||||||
signals: VomNodeDomSignals,
|
signals: VomNodeDomSignals,
|
||||||
): VomNode {
|
): VomNode {
|
||||||
if (!capturedNode) return node;
|
if (!capturedNode) return node;
|
||||||
const attrs = capturedNode.attrs;
|
const attrs =
|
||||||
|
node.sensitive && Object.prototype.hasOwnProperty.call(capturedNode.attrs, "value")
|
||||||
|
? Object.fromEntries(
|
||||||
|
Object.entries(capturedNode.attrs).filter(([name]) => name.toLowerCase() !== "value"),
|
||||||
|
)
|
||||||
|
: capturedNode.attrs;
|
||||||
const text = cleanAttr(capturedNode.textContent);
|
const text = cleanAttr(capturedNode.textContent);
|
||||||
const nearbyText = nearbyTextFor(capturedNode, signals.childrenByParentId);
|
const nearbyText = nearbyTextFor(capturedNode, signals.childrenByParentId);
|
||||||
const placeholder = cleanAttr(capturedNode.formPlaceholder) ?? cleanAttr(attrs.placeholder);
|
const placeholder = cleanAttr(capturedNode.formPlaceholder) ?? cleanAttr(attrs.placeholder);
|
||||||
@@ -823,8 +830,7 @@ function vomNodeFromCaptured(
|
|||||||
signals: VomNodeDomSignals,
|
signals: VomNodeDomSignals,
|
||||||
): VomNode {
|
): VomNode {
|
||||||
const sensitive = isSensitive(capturedNode);
|
const sensitive = isSensitive(capturedNode);
|
||||||
const value =
|
const value = sensitive ? undefined : capturedNode.formValue;
|
||||||
capturedNode.formValue ?? (sensitive ? (capturedNode.attrs.value ?? "") : undefined);
|
|
||||||
const tag = normalizeTag(capturedNode.tag);
|
const tag = normalizeTag(capturedNode.tag);
|
||||||
const node = applyCapturedSignals(
|
const node = applyCapturedSignals(
|
||||||
{
|
{
|
||||||
@@ -860,8 +866,17 @@ function axVomNode(
|
|||||||
): VomNode {
|
): VomNode {
|
||||||
const role = axString(axNode.role);
|
const role = axString(axNode.role);
|
||||||
const capturedValue = capturedNode?.formValue;
|
const capturedValue = capturedNode?.formValue;
|
||||||
const value = capturedValue ?? axValue(axNode.value);
|
const sourceValue = capturedValue ?? axValue(axNode.value);
|
||||||
const signalsForNode = axSignals.get(axNode.nodeId);
|
const signalsForNode = axSignals.get(axNode.nodeId);
|
||||||
|
const sensitive = isSensitive(capturedNode) || signalsForNode?.sensitive === true;
|
||||||
|
const value = sensitive ? undefined : sourceValue;
|
||||||
|
const inputState =
|
||||||
|
inputStateFor(capturedNode, sourceValue) ??
|
||||||
|
(sensitive && ["textbox", "searchbox"].includes(role?.toLowerCase() ?? "")
|
||||||
|
? sourceValue === undefined || sourceValue === ""
|
||||||
|
? "empty"
|
||||||
|
: "filled"
|
||||||
|
: undefined);
|
||||||
let name = FORM_CONTROL_TAGS.has(normalizeTag(capturedNode?.tag))
|
let name = FORM_CONTROL_TAGS.has(normalizeTag(capturedNode?.tag))
|
||||||
? formControlName(capturedNode, axString(axNode.name), axSiblingLabel(axNode, axById), signals)
|
? formControlName(capturedNode, axString(axNode.name), axSiblingLabel(axNode, axById), signals)
|
||||||
: (axString(axNode.name) ?? signalsForNode?.aggregatedText);
|
: (axString(axNode.name) ?? signalsForNode?.aggregatedText);
|
||||||
@@ -877,8 +892,8 @@ function axVomNode(
|
|||||||
position: capturedNode?.position || "static",
|
position: capturedNode?.position || "static",
|
||||||
pointerEvents: capturedNode?.pointerEvents || "auto",
|
pointerEvents: capturedNode?.pointerEvents || "auto",
|
||||||
modal: isModalSignal(axNode, capturedNode),
|
modal: isModalSignal(axNode, capturedNode),
|
||||||
sensitive: isSensitive(capturedNode) || signalsForNode?.sensitive === true,
|
sensitive,
|
||||||
inputState: inputStateFor(capturedNode, value),
|
inputState,
|
||||||
};
|
};
|
||||||
if (role) node.role = role;
|
if (role) node.role = role;
|
||||||
if (name) node.name = name;
|
if (name) node.name = name;
|
||||||
@@ -1141,6 +1156,8 @@ export interface BuildVomSceneOptions {
|
|||||||
pageUrl?: string;
|
pageUrl?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export type VomFrameDocument = CapturedFrameDocument<CdpAxNode>;
|
||||||
|
|
||||||
function externalHrefHost(
|
function externalHrefHost(
|
||||||
href: string | undefined,
|
href: string | undefined,
|
||||||
pageUrl: string | undefined,
|
pageUrl: string | undefined,
|
||||||
@@ -1234,6 +1251,119 @@ export function buildVomScene(
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export function buildFrameVomScene(
|
||||||
|
documents: VomFrameDocument[],
|
||||||
|
captured: CapturedViewModel,
|
||||||
|
options: BuildVomSceneOptions = {},
|
||||||
|
): VomScene {
|
||||||
|
const documentByFrameId = new Map(documents.map((document) => [document.frameId, document]));
|
||||||
|
const rootFrameId =
|
||||||
|
captured.rootFrameId ?? documents.find((document) => !document.parentFrameId)?.frameId;
|
||||||
|
const orderedFrameIds = [
|
||||||
|
...(rootFrameId ? [rootFrameId] : []),
|
||||||
|
...documents.map((document) => document.frameId),
|
||||||
|
].filter((frameId, index, all) => all.indexOf(frameId) === index);
|
||||||
|
|
||||||
|
const localScenes = new Map<string, VomScene>();
|
||||||
|
for (const frameId of orderedFrameIds) {
|
||||||
|
const document = documentByFrameId.get(frameId);
|
||||||
|
if (!document) continue;
|
||||||
|
localScenes.set(
|
||||||
|
frameId,
|
||||||
|
buildVomScene(
|
||||||
|
document.axNodes,
|
||||||
|
{
|
||||||
|
viewport: captured.viewport,
|
||||||
|
nodes: document.domNodes,
|
||||||
|
iframeNodes: new Map(),
|
||||||
|
surfaceProbes: frameId === rootFrameId ? captured.surfaceProbes : [],
|
||||||
|
excludedBackendNodeIds:
|
||||||
|
frameId === rootFrameId ? captured.excludedBackendNodeIds : new Set(),
|
||||||
|
},
|
||||||
|
{ pageUrl: options.pageUrl },
|
||||||
|
),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const sceneIdByFrameAndBackend = new Map<string, number>();
|
||||||
|
let nextSceneId = 1;
|
||||||
|
for (const [frameId, scene] of localScenes) {
|
||||||
|
for (const node of scene.nodes) {
|
||||||
|
sceneIdByFrameAndBackend.set(`${frameId}:${node.id}`, nextSceneId);
|
||||||
|
nextSceneId += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const nodes: VomNode[] = [];
|
||||||
|
const surfaces: CondSurface[] = [];
|
||||||
|
const activeScopeBlocks: ActiveScopeBlock[] = [];
|
||||||
|
|
||||||
|
for (const [frameId, scene] of localScenes) {
|
||||||
|
const document = documentByFrameId.get(frameId);
|
||||||
|
const ownerBackendNodeId = document?.ownerBackendNodeId;
|
||||||
|
const parentFrameId = document?.parentFrameId;
|
||||||
|
const ownerSceneId =
|
||||||
|
ownerBackendNodeId !== undefined && parentFrameId
|
||||||
|
? sceneIdByFrameAndBackend.get(`${parentFrameId}:${ownerBackendNodeId}`)
|
||||||
|
: undefined;
|
||||||
|
if (frameId !== rootFrameId && ownerSceneId === undefined) continue;
|
||||||
|
|
||||||
|
const suppressedRoots = new Set(
|
||||||
|
frameId === rootFrameId
|
||||||
|
? []
|
||||||
|
: scene.nodes
|
||||||
|
.filter(
|
||||||
|
(node) =>
|
||||||
|
node.parentId === null &&
|
||||||
|
["rootwebarea", "webarea"].includes(node.role?.toLowerCase() ?? ""),
|
||||||
|
)
|
||||||
|
.map((node) => node.id),
|
||||||
|
);
|
||||||
|
|
||||||
|
for (const node of scene.nodes) {
|
||||||
|
if (suppressedRoots.has(node.id)) continue;
|
||||||
|
const sceneNodeId = sceneIdByFrameAndBackend.get(`${frameId}:${node.id}`) as number;
|
||||||
|
let parentId =
|
||||||
|
node.parentId !== null
|
||||||
|
? sceneIdByFrameAndBackend.get(`${frameId}:${node.parentId}`)
|
||||||
|
: ownerSceneId;
|
||||||
|
if (node.parentId !== null && suppressedRoots.has(node.parentId)) parentId = ownerSceneId;
|
||||||
|
const remapId = (id: number) => sceneIdByFrameAndBackend.get(`${frameId}:${id}`);
|
||||||
|
nodes.push({
|
||||||
|
...node,
|
||||||
|
id: sceneNodeId,
|
||||||
|
backendNodeId: node.id,
|
||||||
|
frameId,
|
||||||
|
contextScopeId: document?.contextScopeId ?? frameId,
|
||||||
|
parentId: parentId ?? null,
|
||||||
|
...(node.domParentId !== undefined
|
||||||
|
? { domParentId: node.domParentId === null ? null : (remapId(node.domParentId) ?? null) }
|
||||||
|
: {}),
|
||||||
|
...(node.domAncestorIds
|
||||||
|
? { domAncestorIds: node.domAncestorIds.flatMap((id) => remapId(id) ?? []) }
|
||||||
|
: {}),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const surface of scene.surfaces ?? []) {
|
||||||
|
const triggerId = sceneIdByFrameAndBackend.get(`${frameId}:${surface.triggerId}`);
|
||||||
|
if (triggerId !== undefined) surfaces.push({ ...surface, triggerId });
|
||||||
|
}
|
||||||
|
for (const block of scene.activeScopeBlocks ?? []) {
|
||||||
|
const triggerId = sceneIdByFrameAndBackend.get(`${frameId}:${block.triggerId}`);
|
||||||
|
if (triggerId !== undefined) activeScopeBlocks.push({ ...block, triggerId });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
viewport: captured.viewport,
|
||||||
|
nodes,
|
||||||
|
...(rootFrameId ? { rootFrameId } : {}),
|
||||||
|
...(surfaces.length > 0 ? { surfaces } : {}),
|
||||||
|
...(activeScopeBlocks.length > 0 ? { activeScopeBlocks } : {}),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
export interface SnapshotDeps {
|
export interface SnapshotDeps {
|
||||||
cdp: CdpRunner;
|
cdp: CdpRunner;
|
||||||
tabsApi: {
|
tabsApi: {
|
||||||
@@ -1447,15 +1577,6 @@ async function handleVomObservation(
|
|||||||
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
|
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
|
||||||
await deps.cdp.ensureAttachedToUrl?.(target.tabId, target.url);
|
await deps.cdp.ensureAttachedToUrl?.(target.tabId, target.url);
|
||||||
throwIfAborted(signal, toolName);
|
throwIfAborted(signal, toolName);
|
||||||
await deps.cdp.send<unknown>(target.tabId, "Accessibility.enable", {});
|
|
||||||
throwIfAborted(signal, toolName);
|
|
||||||
const result = await deps.cdp.send<{ nodes: CdpAxNode[] }>(
|
|
||||||
target.tabId,
|
|
||||||
"Accessibility.getFullAXTree",
|
|
||||||
{},
|
|
||||||
);
|
|
||||||
throwIfAborted(signal, toolName);
|
|
||||||
const axNodes = result.nodes ?? [];
|
|
||||||
const effectiveConditionalSurfaceProbe =
|
const effectiveConditionalSurfaceProbe =
|
||||||
deps.conditionalSurfaceProbe ?? conditionalSurfaceProbe;
|
deps.conditionalSurfaceProbe ?? conditionalSurfaceProbe;
|
||||||
const captured = await captureForVom(
|
const captured = await captureForVom(
|
||||||
@@ -1466,7 +1587,17 @@ async function handleVomObservation(
|
|||||||
signal,
|
signal,
|
||||||
);
|
);
|
||||||
throwIfAborted(signal, toolName);
|
throwIfAborted(signal, toolName);
|
||||||
const scene = buildVomScene(axNodes, captured, { pageUrl: target.url });
|
const frameDocuments = await captureFrameData<CdpAxNode>(
|
||||||
|
deps.cdp,
|
||||||
|
target.tabId,
|
||||||
|
captured,
|
||||||
|
signal,
|
||||||
|
);
|
||||||
|
throwIfAborted(signal, toolName);
|
||||||
|
const scene =
|
||||||
|
captured.rootFrameId && captured.frameNodes?.size
|
||||||
|
? buildFrameVomScene(frameDocuments, captured, { pageUrl: target.url })
|
||||||
|
: buildVomScene(frameDocuments[0]?.axNodes ?? [], captured, { pageUrl: target.url });
|
||||||
const rendered = renderVom(scene, {
|
const rendered = renderVom(scene, {
|
||||||
maxDepth: params.max_depth,
|
maxDepth: params.max_depth,
|
||||||
maxTokens: params.max_tokens,
|
maxTokens: params.max_tokens,
|
||||||
@@ -1475,7 +1606,7 @@ async function handleVomObservation(
|
|||||||
throwIfAborted(signal, toolName);
|
throwIfAborted(signal, toolName);
|
||||||
ctx.refStore.replace(
|
ctx.refStore.replace(
|
||||||
rendered.refs.map(
|
rendered.refs.map(
|
||||||
(r) => [r.ref, { backendNodeId: r.backendNodeId, tabId: target.tabId }] as const,
|
(ref) => [ref.ref, { backendNodeId: ref.backendNodeId, tabId: target.tabId }] as const,
|
||||||
),
|
),
|
||||||
);
|
);
|
||||||
return attachDialogs(deps.cdp, target.tabId, dialogCursor, {
|
return attachDialogs(deps.cdp, target.tabId, dialogCursor, {
|
||||||
|
|||||||
@@ -18,6 +18,8 @@ function fakeSnapshotReply() {
|
|||||||
"auto",
|
"auto",
|
||||||
"type",
|
"type",
|
||||||
"password",
|
"password",
|
||||||
|
"value",
|
||||||
|
"hunter2",
|
||||||
];
|
];
|
||||||
const i = (s: string) => S.indexOf(s);
|
const i = (s: string) => S.indexOf(s);
|
||||||
return {
|
return {
|
||||||
@@ -29,7 +31,8 @@ function fakeSnapshotReply() {
|
|||||||
nodeType: [1, 1, 1, 1],
|
nodeType: [1, 1, 1, 1],
|
||||||
nodeName: [i("html"), i("body"), i("div"), i("input")],
|
nodeName: [i("html"), i("body"), i("div"), i("input")],
|
||||||
backendNodeId: [10, 11, 12, 13],
|
backendNodeId: [10, 11, 12, 13],
|
||||||
attributes: [[], [], [], [i("type"), i("password")]],
|
attributes: [[], [], [], [i("type"), i("password"), i("value"), i("hunter2")]],
|
||||||
|
inputValue: { index: [3], value: [i("hunter2")] },
|
||||||
},
|
},
|
||||||
layout: {
|
layout: {
|
||||||
nodeIndex: [1, 2, 3],
|
nodeIndex: [1, 2, 3],
|
||||||
@@ -217,7 +220,8 @@ function makeCdp(snapshot: unknown) {
|
|||||||
{
|
{
|
||||||
state: "filled",
|
state: "filled",
|
||||||
sensitive: true,
|
sensitive: true,
|
||||||
defaultValue: "",
|
defaultValue: "hunter2",
|
||||||
|
value: "hunter2",
|
||||||
placeholder: "secret",
|
placeholder: "secret",
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
@@ -262,10 +266,11 @@ describe("captureViewModel", () => {
|
|||||||
|
|
||||||
expect(input).toMatchObject({
|
expect(input).toMatchObject({
|
||||||
formState: "filled",
|
formState: "filled",
|
||||||
formDefaultValue: "",
|
|
||||||
formPlaceholder: "secret",
|
formPlaceholder: "secret",
|
||||||
|
attrs: { type: "password" },
|
||||||
});
|
});
|
||||||
expect(input?.formValue).toBeUndefined();
|
expect(input?.formValue).toBeUndefined();
|
||||||
|
expect(input?.formDefaultValue).toBeUndefined();
|
||||||
expect(cdp.send).toHaveBeenCalledWith(
|
expect(cdp.send).toHaveBeenCalledWith(
|
||||||
4,
|
4,
|
||||||
"Runtime.evaluate",
|
"Runtime.evaluate",
|
||||||
@@ -692,10 +697,8 @@ describe("captureViewModel", () => {
|
|||||||
};
|
};
|
||||||
const { nodes } = await captureViewModel(cdp, 4);
|
const { nodes } = await captureViewModel(cdp, 4);
|
||||||
const div = nodes.find((n) => n.backendNodeId === 12);
|
const div = nodes.find((n) => n.backendNodeId === 12);
|
||||||
// local rect preserves viewport-relative scroll math.
|
|
||||||
expect(div?.localRect?.y).toBe(-200);
|
expect(div?.localRect?.y).toBe(-200);
|
||||||
// exported rect is clipped to the top-level viewport before VOM consumes it.
|
expect(div?.rect).toMatchObject({ y: -200, h: 800 });
|
||||||
expect(div?.rect).toMatchObject({ y: 0, h: 600 });
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("normalizes device-pixel bounds by devicePixelRatio", async () => {
|
it("normalizes device-pixel bounds by devicePixelRatio", async () => {
|
||||||
@@ -839,6 +842,7 @@ describe("captureViewModel", () => {
|
|||||||
strings: S,
|
strings: S,
|
||||||
documents: [
|
documents: [
|
||||||
{
|
{
|
||||||
|
frameId: "main-frame",
|
||||||
scrollOffsetX: 0,
|
scrollOffsetX: 0,
|
||||||
scrollOffsetY: 0,
|
scrollOffsetY: 0,
|
||||||
nodes: {
|
nodes: {
|
||||||
@@ -866,6 +870,7 @@ describe("captureViewModel", () => {
|
|||||||
},
|
},
|
||||||
// documents[1]: the iframe's sub-document with a login input
|
// documents[1]: the iframe's sub-document with a login input
|
||||||
{
|
{
|
||||||
|
frameId: "child-frame",
|
||||||
scrollOffsetX: 0,
|
scrollOffsetX: 0,
|
||||||
scrollOffsetY: 0,
|
scrollOffsetY: 0,
|
||||||
nodes: {
|
nodes: {
|
||||||
@@ -895,7 +900,14 @@ describe("captureViewModel", () => {
|
|||||||
throw new Error(method);
|
throw new Error(method);
|
||||||
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
|
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
|
||||||
};
|
};
|
||||||
const { nodes, iframeNodes } = await captureViewModel(cdp, 4);
|
const {
|
||||||
|
nodes,
|
||||||
|
iframeNodes,
|
||||||
|
frameNodes,
|
||||||
|
frameOwnerBackendNodeIds,
|
||||||
|
frameParentIds,
|
||||||
|
rootFrameId,
|
||||||
|
} = await captureViewModel(cdp, 4);
|
||||||
// Main frame has 4 nodes; iframe is included
|
// Main frame has 4 nodes; iframe is included
|
||||||
expect(nodes.find((n) => n.backendNodeId === 13)?.tag).toBe("iframe");
|
expect(nodes.find((n) => n.backendNodeId === 13)?.tag).toBe("iframe");
|
||||||
// iframeNodes keyed by the <iframe> element's backendNodeId=13
|
// iframeNodes keyed by the <iframe> element's backendNodeId=13
|
||||||
@@ -907,10 +919,14 @@ describe("captureViewModel", () => {
|
|||||||
expect(input?.attrs.type).toBe("text");
|
expect(input?.attrs.type).toBe("text");
|
||||||
expect(input?.ownerFrameBackendNodeId).toBe(13);
|
expect(input?.ownerFrameBackendNodeId).toBe(13);
|
||||||
expect(input?.localRect).toEqual({ x: 0, y: 0, w: 200, h: 40 });
|
expect(input?.localRect).toEqual({ x: 0, y: 0, w: 200, h: 40 });
|
||||||
expect(input?.rect).toEqual({ x: 100, y: 300, w: 200, h: 40 });
|
expect(input?.rect).toBeNull();
|
||||||
|
expect(rootFrameId).toBe("main-frame");
|
||||||
|
expect(frameNodes?.get("child-frame")).toBe(subNodes);
|
||||||
|
expect(frameOwnerBackendNodeIds?.get("child-frame")).toBe(13);
|
||||||
|
expect(frameParentIds?.get("child-frame")).toBe("main-frame");
|
||||||
});
|
});
|
||||||
|
|
||||||
it("clips iframe sub-document rects to the iframe viewport in top coordinates", async () => {
|
it("captures iframe documents without requiring owner-frame geometry", async () => {
|
||||||
const S = [
|
const S = [
|
||||||
"html",
|
"html",
|
||||||
"body",
|
"body",
|
||||||
@@ -938,16 +954,10 @@ describe("captureViewModel", () => {
|
|||||||
contentDocumentIndex: { index: [2], value: [1] },
|
contentDocumentIndex: { index: [2], value: [1] },
|
||||||
},
|
},
|
||||||
layout: {
|
layout: {
|
||||||
nodeIndex: [1, 2],
|
nodeIndex: [1],
|
||||||
styles: [
|
styles: [[i("static"), i("auto")]],
|
||||||
[i("static"), i("auto")],
|
bounds: [[0, 0, 1000, 800]],
|
||||||
[i("static"), i("auto")],
|
paintOrders: [0],
|
||||||
],
|
|
||||||
bounds: [
|
|
||||||
[0, 0, 1000, 800],
|
|
||||||
[950, 760, 200, 100],
|
|
||||||
],
|
|
||||||
paintOrders: [0, 1],
|
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -985,7 +995,7 @@ describe("captureViewModel", () => {
|
|||||||
const input = iframeNodes.get(13)?.find((n) => n.backendNodeId === 101);
|
const input = iframeNodes.get(13)?.find((n) => n.backendNodeId === 101);
|
||||||
|
|
||||||
expect(input?.localRect).toEqual({ x: 20, y: 20, w: 120, h: 60 });
|
expect(input?.localRect).toEqual({ x: 20, y: 20, w: 120, h: 60 });
|
||||||
expect(input?.rect).toEqual({ x: 970, y: 780, w: 30, h: 20 });
|
expect(input?.rect).toBeNull();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("recursively normalizes nested iframe sub-documents", async () => {
|
it("recursively normalizes nested iframe sub-documents", async () => {
|
||||||
@@ -1080,9 +1090,11 @@ describe("captureViewModel", () => {
|
|||||||
const nestedIframe = iframeNodes.get(13)?.find((n) => n.backendNodeId === 23);
|
const nestedIframe = iframeNodes.get(13)?.find((n) => n.backendNodeId === 23);
|
||||||
const input = iframeNodes.get(23)?.find((n) => n.backendNodeId === 31);
|
const input = iframeNodes.get(23)?.find((n) => n.backendNodeId === 31);
|
||||||
|
|
||||||
expect(nestedIframe?.rect).toEqual({ x: 15, y: 26, w: 100, h: 80 });
|
expect(nestedIframe?.localRect).toEqual({ x: 5, y: 6, w: 100, h: 80 });
|
||||||
|
expect(nestedIframe?.rect).toBeNull();
|
||||||
expect(input?.ownerFrameBackendNodeId).toBe(23);
|
expect(input?.ownerFrameBackendNodeId).toBe(23);
|
||||||
expect(input?.rect).toEqual({ x: 16, y: 28, w: 40, h: 20 });
|
expect(input?.localRect).toEqual({ x: 1, y: 2, w: 40, h: 20 });
|
||||||
|
expect(input?.rect).toBeNull();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("normalizes bounds then subtracts CSS scroll at dpr>1", async () => {
|
it("normalizes bounds then subtracts CSS scroll at dpr>1", async () => {
|
||||||
|
|||||||
@@ -0,0 +1,63 @@
|
|||||||
|
import { describe, expect, it, vi } from "vitest";
|
||||||
|
import type { CdpRunner } from "../../shared";
|
||||||
|
import type { CapturedNode, CapturedViewModel } from "../capture";
|
||||||
|
import { captureFrameData } from "../frame-capture";
|
||||||
|
|
||||||
|
function node(frameId: string, backendNodeId: number): CapturedNode {
|
||||||
|
return {
|
||||||
|
backendNodeId,
|
||||||
|
parentBackendNodeId: null,
|
||||||
|
frameId,
|
||||||
|
tag: "div",
|
||||||
|
attrs: {},
|
||||||
|
rect: null,
|
||||||
|
localRect: { x: 0, y: 0, w: 20, h: 20 },
|
||||||
|
paintOrder: 0,
|
||||||
|
position: "static",
|
||||||
|
pointerEvents: "auto",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("captureFrameData", () => {
|
||||||
|
it("captures AX trees for every same-target frame found in the DOM snapshot", async () => {
|
||||||
|
const captured: CapturedViewModel = {
|
||||||
|
nodes: [node("main", 1)],
|
||||||
|
viewport: { width: 1000, height: 800 },
|
||||||
|
iframeNodes: new Map([[10, [node("child", 101)]]]),
|
||||||
|
frameNodes: new Map([
|
||||||
|
["main", [node("main", 1)]],
|
||||||
|
["child", [node("child", 101)]],
|
||||||
|
]),
|
||||||
|
frameOwnerBackendNodeIds: new Map([["child", 10]]),
|
||||||
|
frameParentIds: new Map([["child", "main"]]),
|
||||||
|
rootFrameId: "main",
|
||||||
|
excludedBackendNodeIds: new Set(),
|
||||||
|
};
|
||||||
|
const send = vi.fn(async (_tabId: number, method: string, params?: object) => {
|
||||||
|
if (method === "Accessibility.enable") return {};
|
||||||
|
if (method === "Accessibility.getFullAXTree") {
|
||||||
|
const frameId = (params as { frameId?: string })?.frameId ?? "main";
|
||||||
|
return {
|
||||||
|
nodes: [
|
||||||
|
{
|
||||||
|
nodeId: `${frameId}-root`,
|
||||||
|
frameId,
|
||||||
|
backendDOMNodeId: frameId === "main" ? 1 : 101,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
throw new Error(`unexpected ${method}`);
|
||||||
|
});
|
||||||
|
|
||||||
|
const documents = await captureFrameData({ send } as CdpRunner, 4, captured);
|
||||||
|
|
||||||
|
expect(send).toHaveBeenCalledWith(4, "Accessibility.enable", {});
|
||||||
|
expect(documents.map((document) => document.frameId)).toEqual(["main", "child"]);
|
||||||
|
expect(documents.find((document) => document.frameId === "child")).toMatchObject({
|
||||||
|
parentFrameId: "main",
|
||||||
|
ownerBackendNodeId: 10,
|
||||||
|
axNodes: [{ nodeId: "child-root", frameId: "child", backendDOMNodeId: 101 }],
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,110 @@
|
|||||||
|
import { describe, expect, it } from "vitest";
|
||||||
|
import type { CapturedNode, CapturedViewModel } from "../capture";
|
||||||
|
import { buildFrameDocuments, type FrameAxBatch, type FrameOwnedAxNode } from "../frame-document";
|
||||||
|
|
||||||
|
function domNode(frameId: string, backendNodeId: number): CapturedNode {
|
||||||
|
return {
|
||||||
|
backendNodeId,
|
||||||
|
parentBackendNodeId: null,
|
||||||
|
frameId,
|
||||||
|
tag: "div",
|
||||||
|
attrs: {},
|
||||||
|
rect: null,
|
||||||
|
localRect: { x: 0, y: 0, w: 20, h: 20 },
|
||||||
|
paintOrder: 0,
|
||||||
|
position: "static",
|
||||||
|
pointerEvents: "auto",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function captured(frameNodes: Map<string, CapturedNode[]>): CapturedViewModel {
|
||||||
|
return {
|
||||||
|
nodes: frameNodes.get("main") ?? [],
|
||||||
|
viewport: { width: 800, height: 600 },
|
||||||
|
iframeNodes: new Map(),
|
||||||
|
frameNodes,
|
||||||
|
rootFrameId: "main",
|
||||||
|
excludedBackendNodeIds: new Set(),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("buildFrameDocuments", () => {
|
||||||
|
it("partitions mixed AX results by node ownership and cuts cross-frame parent edges", () => {
|
||||||
|
const batches: FrameAxBatch<FrameOwnedAxNode>[] = [
|
||||||
|
{
|
||||||
|
frameId: "main",
|
||||||
|
nodes: [
|
||||||
|
{ nodeId: "root", frameId: "main", backendDOMNodeId: 1, childIds: ["h"] },
|
||||||
|
{
|
||||||
|
nodeId: "h",
|
||||||
|
frameId: "child",
|
||||||
|
parentId: "root",
|
||||||
|
backendDOMNodeId: 100,
|
||||||
|
childIds: ["virtual"],
|
||||||
|
},
|
||||||
|
{ nodeId: "virtual", parentId: "h" },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
const documents = buildFrameDocuments(
|
||||||
|
batches,
|
||||||
|
captured(
|
||||||
|
new Map([
|
||||||
|
["main", [domNode("main", 1)]],
|
||||||
|
["child", [domNode("child", 100)]],
|
||||||
|
]),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(documents.find((document) => document.frameId === "main")?.axNodes).toHaveLength(1);
|
||||||
|
const childNodes = documents.find((document) => document.frameId === "child")?.axNodes ?? [];
|
||||||
|
expect(childNodes.map((node) => node.nodeId)).toEqual(["h", "virtual"]);
|
||||||
|
expect(childNodes.find((node) => node.nodeId === "h")?.parentId).toBeUndefined();
|
||||||
|
expect(childNodes.find((node) => node.nodeId === "virtual")?.frameId).toBe("child");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("deduplicates overlapping AX results using the strongest ownership", () => {
|
||||||
|
const childNode = { nodeId: "shared", backendDOMNodeId: 100 };
|
||||||
|
const batches: FrameAxBatch<FrameOwnedAxNode>[] = [
|
||||||
|
{ frameId: "main", nodes: [{ ...childNode, frameId: "child" }] },
|
||||||
|
{ frameId: "child", nodes: [childNode] },
|
||||||
|
];
|
||||||
|
|
||||||
|
const documents = buildFrameDocuments(
|
||||||
|
batches,
|
||||||
|
captured(
|
||||||
|
new Map([
|
||||||
|
["main", [domNode("main", 1)]],
|
||||||
|
["child", [domNode("child", 100)]],
|
||||||
|
]),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(documents.find((document) => document.frameId === "main")?.axNodes).toEqual([]);
|
||||||
|
expect(documents.find((document) => document.frameId === "child")?.axNodes).toHaveLength(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("uses captured frame parent identity instead of guessing from backend ids", () => {
|
||||||
|
const frameNodes = new Map([
|
||||||
|
["main", [domNode("main", 1)]],
|
||||||
|
["left", [domNode("left", 7)]],
|
||||||
|
["right", [domNode("right", 8)]],
|
||||||
|
["child", [domNode("child", 9)]],
|
||||||
|
]);
|
||||||
|
const capture = captured(frameNodes);
|
||||||
|
capture.frameOwnerBackendNodeIds = new Map([["child", 7]]);
|
||||||
|
capture.frameParentIds = new Map([["child", "left"]]);
|
||||||
|
const batches = ["main", "left", "right", "child"].map((frameId) => ({
|
||||||
|
frameId,
|
||||||
|
nodes: [],
|
||||||
|
}));
|
||||||
|
|
||||||
|
const documents = buildFrameDocuments(batches, capture);
|
||||||
|
|
||||||
|
expect(documents.find((document) => document.frameId === "child")).toMatchObject({
|
||||||
|
parentFrameId: "left",
|
||||||
|
ownerBackendNodeId: 7,
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -17,13 +17,14 @@ const STYLE_COL = Object.fromEntries(
|
|||||||
export interface CapturedNode {
|
export interface CapturedNode {
|
||||||
backendNodeId: number;
|
backendNodeId: number;
|
||||||
parentBackendNodeId: number | null;
|
parentBackendNodeId: number | null;
|
||||||
|
frameId?: string;
|
||||||
/** Owning iframe backend node id; `null` for the top-level document. */
|
/** Owning iframe backend node id; `null` for the top-level document. */
|
||||||
ownerFrameBackendNodeId?: number | null;
|
ownerFrameBackendNodeId?: number | null;
|
||||||
tag: string;
|
tag: string;
|
||||||
attrs: Record<string, string>;
|
attrs: Record<string, string>;
|
||||||
/** Top-level viewport-relative CSS px, clipped to the owning frame viewport. */
|
/** Top-level viewport geometry when this node belongs to the root document. */
|
||||||
rect: Rect | null;
|
rect: Rect | null;
|
||||||
/** Frame-local viewport-relative CSS px before top-level translation/clipping. */
|
/** Owning document viewport-relative CSS px. */
|
||||||
localRect?: Rect | null;
|
localRect?: Rect | null;
|
||||||
paintOrder: number;
|
paintOrder: number;
|
||||||
position: string;
|
position: string;
|
||||||
@@ -57,6 +58,11 @@ export interface CapturedViewModel {
|
|||||||
nodes: CapturedNode[];
|
nodes: CapturedNode[];
|
||||||
viewport: Viewport;
|
viewport: Viewport;
|
||||||
iframeNodes: CapturedIframeNodes;
|
iframeNodes: CapturedIframeNodes;
|
||||||
|
frameNodes?: Map<string, CapturedNode[]>;
|
||||||
|
frameOwnerBackendNodeIds?: Map<string, number>;
|
||||||
|
/** Explicit DOMSnapshot frame ancestry; never inferred from backend node ids. */
|
||||||
|
frameParentIds?: Map<string, string>;
|
||||||
|
rootFrameId?: string;
|
||||||
surfaceProbes?: CapturedSurfaceProbe[];
|
surfaceProbes?: CapturedSurfaceProbe[];
|
||||||
/** Backend node ids belonging to the agent overlay host + its shadow subtree. */
|
/** Backend node ids belonging to the agent overlay host + its shadow subtree. */
|
||||||
excludedBackendNodeIds: Set<number>;
|
excludedBackendNodeIds: Set<number>;
|
||||||
@@ -92,7 +98,12 @@ interface SparseArray {
|
|||||||
value: number[];
|
value: number[];
|
||||||
}
|
}
|
||||||
|
|
||||||
|
interface RareBooleanData {
|
||||||
|
index: number[];
|
||||||
|
}
|
||||||
|
|
||||||
interface SnapshotDocument {
|
interface SnapshotDocument {
|
||||||
|
frameId?: string | number;
|
||||||
scrollOffsetX?: number;
|
scrollOffsetX?: number;
|
||||||
scrollOffsetY?: number;
|
scrollOffsetY?: number;
|
||||||
nodes?: {
|
nodes?: {
|
||||||
@@ -107,6 +118,10 @@ interface SnapshotDocument {
|
|||||||
nodeValue?: number[];
|
nodeValue?: number[];
|
||||||
/** Maps node array index → index into `documents[]` for frame content. */
|
/** Maps node array index → index into `documents[]` for frame content. */
|
||||||
contentDocumentIndex?: SparseArray;
|
contentDocumentIndex?: SparseArray;
|
||||||
|
inputValue?: SparseArray;
|
||||||
|
textValue?: SparseArray;
|
||||||
|
inputChecked?: RareBooleanData;
|
||||||
|
optionSelected?: RareBooleanData;
|
||||||
};
|
};
|
||||||
layout?: {
|
layout?: {
|
||||||
nodeIndex?: number[];
|
nodeIndex?: number[];
|
||||||
@@ -116,6 +131,12 @@ interface SnapshotDocument {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function snapshotFrameId(document: SnapshotDocument, strings: string[]): string | undefined {
|
||||||
|
if (typeof document.frameId === "string") return document.frameId || undefined;
|
||||||
|
if (typeof document.frameId === "number") return str(strings, document.frameId) || undefined;
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
|
||||||
interface SnapshotReply {
|
interface SnapshotReply {
|
||||||
strings?: string[];
|
strings?: string[];
|
||||||
documents?: SnapshotDocument[];
|
documents?: SnapshotDocument[];
|
||||||
@@ -135,6 +156,25 @@ function isFormControlTag(tag: string): boolean {
|
|||||||
return tag === "input" || tag === "textarea" || tag === "select";
|
return tag === "input" || tag === "textarea" || tag === "select";
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function isSensitiveFormControl(node: Pick<CapturedNode, "tag" | "attrs">): boolean {
|
||||||
|
return node.tag === "input" && (node.attrs.type ?? "").toLowerCase() === "password";
|
||||||
|
}
|
||||||
|
|
||||||
|
function snapshotFormState(
|
||||||
|
value: string | undefined,
|
||||||
|
defaultValue: string,
|
||||||
|
hasDefaultValue: boolean,
|
||||||
|
sensitive: boolean,
|
||||||
|
): CapturedNode["formState"] {
|
||||||
|
if (sensitive) {
|
||||||
|
if (value === undefined && !hasDefaultValue) return undefined;
|
||||||
|
return (value ?? defaultValue) === "" ? "empty" : "filled";
|
||||||
|
}
|
||||||
|
if (value === undefined) return undefined;
|
||||||
|
if (value === "") return "empty";
|
||||||
|
return value === defaultValue ? "default" : "filled";
|
||||||
|
}
|
||||||
|
|
||||||
const MAX_FORM_ENRICH_CONTROLS = 250;
|
const MAX_FORM_ENRICH_CONTROLS = 250;
|
||||||
|
|
||||||
interface CapturedFormState {
|
interface CapturedFormState {
|
||||||
@@ -166,9 +206,8 @@ function formStateBatchExpression(maxControls: number): string {
|
|||||||
return {
|
return {
|
||||||
state,
|
state,
|
||||||
sensitive,
|
sensitive,
|
||||||
defaultValue,
|
|
||||||
placeholder,
|
placeholder,
|
||||||
...(sensitive ? {} : { value: rawValue }),
|
...(sensitive ? {} : { value: rawValue, defaultValue }),
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
const collect = (doc) => {
|
const collect = (doc) => {
|
||||||
@@ -210,10 +249,14 @@ function applyFormStates(nodes: CapturedNode[], states: CapturedFormState[]): vo
|
|||||||
index += 1;
|
index += 1;
|
||||||
if (!state) continue;
|
if (!state) continue;
|
||||||
node.formState = state.state;
|
node.formState = state.state;
|
||||||
node.formDefaultValue = state.defaultValue ?? "";
|
|
||||||
node.formPlaceholder = state.placeholder ?? "";
|
node.formPlaceholder = state.placeholder ?? "";
|
||||||
if (!state.sensitive && state.value !== undefined) {
|
if (state.sensitive || isSensitiveFormControl(node)) {
|
||||||
node.formValue = state.value;
|
delete node.formValue;
|
||||||
|
delete node.formDefaultValue;
|
||||||
|
delete node.attrs.value;
|
||||||
|
} else {
|
||||||
|
node.formDefaultValue = state.defaultValue ?? "";
|
||||||
|
if (state.value !== undefined) node.formValue = state.value;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -272,9 +315,8 @@ interface ParseDocumentResult {
|
|||||||
}
|
}
|
||||||
|
|
||||||
interface FrameContext {
|
interface FrameContext {
|
||||||
|
frameId?: string;
|
||||||
ownerFrameBackendNodeId: number | null;
|
ownerFrameBackendNodeId: number | null;
|
||||||
originInTop: { x: number; y: number };
|
|
||||||
clipRectInTop: Rect;
|
|
||||||
scrollX: number;
|
scrollX: number;
|
||||||
scrollY: number;
|
scrollY: number;
|
||||||
}
|
}
|
||||||
@@ -293,38 +335,6 @@ function devicePixelRatio(metrics: LayoutMetricsReply): number {
|
|||||||
return dpr >= 1 ? dpr : 1;
|
return dpr >= 1 ? dpr : 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
function viewportRect(viewport: Viewport): Rect {
|
|
||||||
return { x: 0, y: 0, w: Math.max(0, viewport.width), h: Math.max(0, viewport.height) };
|
|
||||||
}
|
|
||||||
|
|
||||||
function intersectRects(a: Rect, b: Rect): Rect | null {
|
|
||||||
const x1 = Math.max(a.x, b.x);
|
|
||||||
const y1 = Math.max(a.y, b.y);
|
|
||||||
const x2 = Math.min(a.x + a.w, b.x + b.w);
|
|
||||||
const y2 = Math.min(a.y + a.h, b.y + b.h);
|
|
||||||
if (x2 <= x1 || y2 <= y1) return null;
|
|
||||||
return { x: x1, y: y1, w: x2 - x1, h: y2 - y1 };
|
|
||||||
}
|
|
||||||
|
|
||||||
function rectInTop(localRect: Rect, context: FrameContext): Rect | null {
|
|
||||||
const translated = {
|
|
||||||
x: context.originInTop.x + localRect.x,
|
|
||||||
y: context.originInTop.y + localRect.y,
|
|
||||||
w: localRect.w,
|
|
||||||
h: localRect.h,
|
|
||||||
};
|
|
||||||
return intersectRects(translated, context.clipRectInTop);
|
|
||||||
}
|
|
||||||
|
|
||||||
function unclipRectInTop(localRect: Rect, context: FrameContext): Rect {
|
|
||||||
return {
|
|
||||||
x: context.originInTop.x + localRect.x,
|
|
||||||
y: context.originInTop.y + localRect.y,
|
|
||||||
w: localRect.w,
|
|
||||||
h: localRect.h,
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
function sparseIndexMap(sparse: SparseArray | undefined): Map<number, number> {
|
function sparseIndexMap(sparse: SparseArray | undefined): Map<number, number> {
|
||||||
const out = new Map<number, number>();
|
const out = new Map<number, number>();
|
||||||
if (!sparse?.index || !sparse.value) return out;
|
if (!sparse?.index || !sparse.value) return out;
|
||||||
@@ -805,6 +815,10 @@ function parseDocumentNodes(
|
|||||||
const posCol = STYLE_COL.position;
|
const posCol = STYLE_COL.position;
|
||||||
const peCol = STYLE_COL["pointer-events"];
|
const peCol = STYLE_COL["pointer-events"];
|
||||||
const cursorCol = STYLE_COL.cursor;
|
const cursorCol = STYLE_COL.cursor;
|
||||||
|
const inputValues = sparseIndexMap(dn.inputValue);
|
||||||
|
const textValues = sparseIndexMap(dn.textValue);
|
||||||
|
const checkedInputs = new Set(dn.inputChecked?.index ?? []);
|
||||||
|
const selectedOptions = new Set(dn.optionSelected?.index ?? []);
|
||||||
|
|
||||||
// Collect visible text from #text child nodes so element CapturedNodes
|
// Collect visible text from #text child nodes so element CapturedNodes
|
||||||
// carry a textContent value usable as a button/link label fallback.
|
// carry a textContent value usable as a button/link label fallback.
|
||||||
@@ -855,7 +869,7 @@ function parseDocumentNodes(
|
|||||||
w: b[2] / dpr,
|
w: b[2] / dpr,
|
||||||
h: b[3] / dpr,
|
h: b[3] / dpr,
|
||||||
};
|
};
|
||||||
rect = rectInTop(localRect, context);
|
rect = context.ownerFrameBackendNodeId === null ? localRect : null;
|
||||||
}
|
}
|
||||||
paintOrder = dl?.paintOrders?.[li] ?? 0;
|
paintOrder = dl?.paintOrders?.[li] ?? 0;
|
||||||
const styleRow = dl?.styles?.[li] ?? [];
|
const styleRow = dl?.styles?.[li] ?? [];
|
||||||
@@ -870,9 +884,24 @@ function parseDocumentNodes(
|
|||||||
|
|
||||||
const textContent = nodeTextContent.get(n);
|
const textContent = nodeTextContent.get(n);
|
||||||
|
|
||||||
|
const rawFormValueIndex = tag === "textarea" ? textValues.get(n) : inputValues.get(n);
|
||||||
|
const rawFormValue =
|
||||||
|
rawFormValueIndex !== undefined ? str(strings, rawFormValueIndex) : undefined;
|
||||||
|
const sensitive = isSensitiveFormControl({ tag, attrs });
|
||||||
|
const formDefaultValue = attrs.value ?? "";
|
||||||
|
const formValue = sensitive ? undefined : rawFormValue;
|
||||||
|
const formState = snapshotFormState(
|
||||||
|
rawFormValue,
|
||||||
|
formDefaultValue,
|
||||||
|
Object.prototype.hasOwnProperty.call(attrs, "value"),
|
||||||
|
sensitive,
|
||||||
|
);
|
||||||
|
if (sensitive) delete attrs.value;
|
||||||
|
|
||||||
nodes.push({
|
nodes.push({
|
||||||
backendNodeId,
|
backendNodeId,
|
||||||
parentBackendNodeId,
|
parentBackendNodeId,
|
||||||
|
...(context.frameId ? { frameId: context.frameId } : {}),
|
||||||
ownerFrameBackendNodeId: context.ownerFrameBackendNodeId,
|
ownerFrameBackendNodeId: context.ownerFrameBackendNodeId,
|
||||||
tag,
|
tag,
|
||||||
attrs,
|
attrs,
|
||||||
@@ -883,6 +912,16 @@ function parseDocumentNodes(
|
|||||||
pointerEvents,
|
pointerEvents,
|
||||||
cursor,
|
cursor,
|
||||||
textContent,
|
textContent,
|
||||||
|
...(formValue !== undefined ? { formValue } : {}),
|
||||||
|
...(tag === "input" || tag === "textarea"
|
||||||
|
? {
|
||||||
|
formPlaceholder: attrs.placeholder ?? "",
|
||||||
|
...(!sensitive ? { formDefaultValue } : {}),
|
||||||
|
...(formState ? { formState } : {}),
|
||||||
|
}
|
||||||
|
: {}),
|
||||||
|
...(checkedInputs.has(n) ? { formValue: "true", formState: "filled" } : {}),
|
||||||
|
...(selectedOptions.has(n) ? { formValue: attrs.value ?? textContent ?? "" } : {}),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
return { nodes, excludedBackendNodeIds };
|
return { nodes, excludedBackendNodeIds };
|
||||||
@@ -890,6 +929,8 @@ function parseDocumentNodes(
|
|||||||
|
|
||||||
interface ParseFrameDocumentsResult {
|
interface ParseFrameDocumentsResult {
|
||||||
iframeNodes: CapturedIframeNodes;
|
iframeNodes: CapturedIframeNodes;
|
||||||
|
frameOwnerBackendNodeIds: Map<string, number>;
|
||||||
|
frameParentIds: Map<string, string>;
|
||||||
excludedBackendNodeIds: Set<number>;
|
excludedBackendNodeIds: Set<number>;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -898,18 +939,20 @@ function parseChildFrameDocuments(
|
|||||||
strings: string[],
|
strings: string[],
|
||||||
dpr: number,
|
dpr: number,
|
||||||
parentDocIndex: number,
|
parentDocIndex: number,
|
||||||
parentNodes: CapturedNode[],
|
|
||||||
parentContext: FrameContext,
|
parentContext: FrameContext,
|
||||||
visited = new Set<number>(),
|
visited = new Set<number>(),
|
||||||
): ParseFrameDocumentsResult {
|
): ParseFrameDocumentsResult {
|
||||||
const iframeNodes: CapturedIframeNodes = new Map();
|
const iframeNodes: CapturedIframeNodes = new Map();
|
||||||
|
const frameOwnerBackendNodeIds = new Map<string, number>();
|
||||||
|
const frameParentIds = new Map<string, string>();
|
||||||
const excludedBackendNodeIds = new Set<number>();
|
const excludedBackendNodeIds = new Set<number>();
|
||||||
const parentDoc = documents[parentDocIndex];
|
const parentDoc = documents[parentDocIndex];
|
||||||
const cdi = sparseIndexMap(parentDoc?.nodes?.contentDocumentIndex);
|
const cdi = sparseIndexMap(parentDoc?.nodes?.contentDocumentIndex);
|
||||||
if (cdi.size === 0) return { iframeNodes, excludedBackendNodeIds };
|
if (cdi.size === 0) {
|
||||||
|
return { iframeNodes, frameOwnerBackendNodeIds, frameParentIds, excludedBackendNodeIds };
|
||||||
|
}
|
||||||
|
|
||||||
const parentBackendIds = parentDoc?.nodes?.backendNodeId ?? [];
|
const parentBackendIds = parentDoc?.nodes?.backendNodeId ?? [];
|
||||||
const parentNodeByBackendId = new Map(parentNodes.map((node) => [node.backendNodeId, node]));
|
|
||||||
|
|
||||||
for (const [nodeArrayIdx, childDocIndex] of cdi) {
|
for (const [nodeArrayIdx, childDocIndex] of cdi) {
|
||||||
if (visited.has(childDocIndex)) continue;
|
if (visited.has(childDocIndex)) continue;
|
||||||
@@ -917,20 +960,10 @@ function parseChildFrameDocuments(
|
|||||||
if (!childDoc) continue;
|
if (!childDoc) continue;
|
||||||
const iframeBackendId = parentBackendIds[nodeArrayIdx];
|
const iframeBackendId = parentBackendIds[nodeArrayIdx];
|
||||||
if (iframeBackendId === undefined) continue;
|
if (iframeBackendId === undefined) continue;
|
||||||
const iframeNode = parentNodeByBackendId.get(iframeBackendId);
|
|
||||||
if (!iframeNode?.localRect) continue;
|
|
||||||
|
|
||||||
const iframeRectInTop = unclipRectInTop(iframeNode.localRect, parentContext);
|
|
||||||
const childClip = intersectRects(iframeRectInTop, parentContext.clipRectInTop);
|
|
||||||
if (!childClip) {
|
|
||||||
iframeNodes.set(iframeBackendId, []);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
const childContext: FrameContext = {
|
const childContext: FrameContext = {
|
||||||
|
frameId: snapshotFrameId(childDoc, strings),
|
||||||
ownerFrameBackendNodeId: iframeBackendId,
|
ownerFrameBackendNodeId: iframeBackendId,
|
||||||
originInTop: { x: iframeRectInTop.x, y: iframeRectInTop.y },
|
|
||||||
clipRectInTop: childClip,
|
|
||||||
scrollX: childDoc.scrollOffsetX ?? 0,
|
scrollX: childDoc.scrollOffsetX ?? 0,
|
||||||
scrollY: childDoc.scrollOffsetY ?? 0,
|
scrollY: childDoc.scrollOffsetY ?? 0,
|
||||||
};
|
};
|
||||||
@@ -938,6 +971,10 @@ function parseChildFrameDocuments(
|
|||||||
nextVisited.add(childDocIndex);
|
nextVisited.add(childDocIndex);
|
||||||
const parsed = parseDocumentNodes(childDoc, strings, dpr, childContext);
|
const parsed = parseDocumentNodes(childDoc, strings, dpr, childContext);
|
||||||
iframeNodes.set(iframeBackendId, parsed.nodes);
|
iframeNodes.set(iframeBackendId, parsed.nodes);
|
||||||
|
if (childContext.frameId) {
|
||||||
|
frameOwnerBackendNodeIds.set(childContext.frameId, iframeBackendId);
|
||||||
|
if (parentContext.frameId) frameParentIds.set(childContext.frameId, parentContext.frameId);
|
||||||
|
}
|
||||||
for (const id of parsed.excludedBackendNodeIds) excludedBackendNodeIds.add(id);
|
for (const id of parsed.excludedBackendNodeIds) excludedBackendNodeIds.add(id);
|
||||||
|
|
||||||
const nested = parseChildFrameDocuments(
|
const nested = parseChildFrameDocuments(
|
||||||
@@ -945,17 +982,22 @@ function parseChildFrameDocuments(
|
|||||||
strings,
|
strings,
|
||||||
dpr,
|
dpr,
|
||||||
childDocIndex,
|
childDocIndex,
|
||||||
parsed.nodes,
|
|
||||||
childContext,
|
childContext,
|
||||||
nextVisited,
|
nextVisited,
|
||||||
);
|
);
|
||||||
for (const [nestedFrameId, nestedNodes] of nested.iframeNodes) {
|
for (const [nestedFrameId, nestedNodes] of nested.iframeNodes) {
|
||||||
iframeNodes.set(nestedFrameId, nestedNodes);
|
iframeNodes.set(nestedFrameId, nestedNodes);
|
||||||
}
|
}
|
||||||
|
for (const [frameId, ownerBackendNodeId] of nested.frameOwnerBackendNodeIds) {
|
||||||
|
frameOwnerBackendNodeIds.set(frameId, ownerBackendNodeId);
|
||||||
|
}
|
||||||
|
for (const [frameId, parentFrameId] of nested.frameParentIds) {
|
||||||
|
frameParentIds.set(frameId, parentFrameId);
|
||||||
|
}
|
||||||
for (const id of nested.excludedBackendNodeIds) excludedBackendNodeIds.add(id);
|
for (const id of nested.excludedBackendNodeIds) excludedBackendNodeIds.add(id);
|
||||||
}
|
}
|
||||||
|
|
||||||
return { iframeNodes, excludedBackendNodeIds };
|
return { iframeNodes, frameOwnerBackendNodeIds, frameParentIds, excludedBackendNodeIds };
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function captureViewModel(
|
export async function captureViewModel(
|
||||||
@@ -998,9 +1040,8 @@ export async function captureViewModel(
|
|||||||
}
|
}
|
||||||
|
|
||||||
const topContext: FrameContext = {
|
const topContext: FrameContext = {
|
||||||
|
frameId: snapshotFrameId(doc0, strings),
|
||||||
ownerFrameBackendNodeId: null,
|
ownerFrameBackendNodeId: null,
|
||||||
originInTop: { x: 0, y: 0 },
|
|
||||||
clipRectInTop: viewportRect(viewport),
|
|
||||||
scrollX,
|
scrollX,
|
||||||
scrollY,
|
scrollY,
|
||||||
};
|
};
|
||||||
@@ -1008,7 +1049,7 @@ export async function captureViewModel(
|
|||||||
const nodes = mainParsed.nodes;
|
const nodes = mainParsed.nodes;
|
||||||
const excludedBackendNodeIds = new Set(mainParsed.excludedBackendNodeIds);
|
const excludedBackendNodeIds = new Set(mainParsed.excludedBackendNodeIds);
|
||||||
|
|
||||||
const frameParsed = parseChildFrameDocuments(documents, strings, dpr, 0, nodes, topContext);
|
const frameParsed = parseChildFrameDocuments(documents, strings, dpr, 0, topContext);
|
||||||
const iframeNodes = frameParsed.iframeNodes;
|
const iframeNodes = frameParsed.iframeNodes;
|
||||||
for (const id of frameParsed.excludedBackendNodeIds) {
|
for (const id of frameParsed.excludedBackendNodeIds) {
|
||||||
excludedBackendNodeIds.add(id);
|
excludedBackendNodeIds.add(id);
|
||||||
@@ -1022,5 +1063,27 @@ export async function captureViewModel(
|
|||||||
: [];
|
: [];
|
||||||
throwIfAborted(options.signal);
|
throwIfAborted(options.signal);
|
||||||
|
|
||||||
return { nodes, viewport, iframeNodes, surfaceProbes, excludedBackendNodeIds };
|
const frameNodes = new Map<string, CapturedNode[]>();
|
||||||
|
if (topContext.frameId) frameNodes.set(topContext.frameId, nodes);
|
||||||
|
for (const iframeFrameNodes of iframeNodes.values()) {
|
||||||
|
const frameId = iframeFrameNodes.find((node) => node.frameId)?.frameId;
|
||||||
|
if (frameId) frameNodes.set(frameId, iframeFrameNodes);
|
||||||
|
}
|
||||||
|
for (const [frameId, ownerBackendNodeId] of frameParsed.frameOwnerBackendNodeIds) {
|
||||||
|
if (!frameNodes.has(frameId)) {
|
||||||
|
frameNodes.set(frameId, iframeNodes.get(ownerBackendNodeId) ?? []);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
nodes,
|
||||||
|
viewport,
|
||||||
|
iframeNodes,
|
||||||
|
frameNodes,
|
||||||
|
frameOwnerBackendNodeIds: frameParsed.frameOwnerBackendNodeIds,
|
||||||
|
frameParentIds: frameParsed.frameParentIds,
|
||||||
|
...(topContext.frameId ? { rootFrameId: topContext.frameId } : {}),
|
||||||
|
surfaceProbes,
|
||||||
|
excludedBackendNodeIds,
|
||||||
|
};
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,75 @@
|
|||||||
|
import type { CdpRunner } from "../shared";
|
||||||
|
import type { CapturedViewModel } from "./capture";
|
||||||
|
import {
|
||||||
|
buildFrameDocuments,
|
||||||
|
type FrameAxBatch,
|
||||||
|
type FrameDocument,
|
||||||
|
type FrameOwnedAxNode,
|
||||||
|
} from "./frame-document";
|
||||||
|
|
||||||
|
export type FrameAxNode = FrameOwnedAxNode;
|
||||||
|
export type CapturedFrameDocument<T extends FrameAxNode> = FrameDocument<T>;
|
||||||
|
|
||||||
|
function throwIfAborted(signal?: AbortSignal): void {
|
||||||
|
if (signal?.aborted) throw new DOMException("observation aborted", "AbortError");
|
||||||
|
}
|
||||||
|
|
||||||
|
function isAbortError(err: unknown): boolean {
|
||||||
|
return err instanceof Error && err.name === "AbortError";
|
||||||
|
}
|
||||||
|
|
||||||
|
function capturedFrameIds(captured: CapturedViewModel): string[] {
|
||||||
|
const frameIds = new Set<string>();
|
||||||
|
if (captured.rootFrameId) frameIds.add(captured.rootFrameId);
|
||||||
|
for (const frameId of captured.frameNodes?.keys() ?? []) frameIds.add(frameId);
|
||||||
|
if (frameIds.size === 0) frameIds.add("root");
|
||||||
|
return [...frameIds];
|
||||||
|
}
|
||||||
|
|
||||||
|
async function captureAxTrees<T extends FrameAxNode>(
|
||||||
|
cdp: CdpRunner,
|
||||||
|
tabId: number,
|
||||||
|
captured: CapturedViewModel,
|
||||||
|
signal?: AbortSignal,
|
||||||
|
): Promise<FrameAxBatch<T>[]> {
|
||||||
|
const frameIds = capturedFrameIds(captured);
|
||||||
|
let firstFailure: unknown;
|
||||||
|
await cdp.send(tabId, "Accessibility.enable", {});
|
||||||
|
const trees = await Promise.all(
|
||||||
|
frameIds.map(async (frameId): Promise<FrameAxBatch<T>> => {
|
||||||
|
try {
|
||||||
|
throwIfAborted(signal);
|
||||||
|
const result = await cdp.send<{ nodes?: T[] }>(
|
||||||
|
tabId,
|
||||||
|
"Accessibility.getFullAXTree",
|
||||||
|
frameId === "root" ? {} : { frameId },
|
||||||
|
);
|
||||||
|
throwIfAborted(signal);
|
||||||
|
return { frameId, nodes: result.nodes ?? [] };
|
||||||
|
} catch (err) {
|
||||||
|
if (isAbortError(err)) throw err;
|
||||||
|
firstFailure ??= err;
|
||||||
|
console.debug("[bsk observation] frame AX capture failed", { frameId, err });
|
||||||
|
return { frameId, nodes: [] };
|
||||||
|
}
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
const capturedNodeCount = [...captured.iframeNodes.values()].reduce(
|
||||||
|
(count, nodes) => count + nodes.length,
|
||||||
|
captured.nodes.length,
|
||||||
|
);
|
||||||
|
if (firstFailure && capturedNodeCount === 0 && trees.every((tree) => tree.nodes.length === 0)) {
|
||||||
|
throw firstFailure;
|
||||||
|
}
|
||||||
|
return trees;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function captureFrameData<T extends FrameAxNode>(
|
||||||
|
cdp: CdpRunner,
|
||||||
|
tabId: number,
|
||||||
|
captured: CapturedViewModel,
|
||||||
|
signal?: AbortSignal,
|
||||||
|
): Promise<CapturedFrameDocument<T>[]> {
|
||||||
|
const batches = await captureAxTrees<T>(cdp, tabId, captured, signal);
|
||||||
|
return buildFrameDocuments(batches, captured);
|
||||||
|
}
|
||||||
@@ -0,0 +1,142 @@
|
|||||||
|
import type { CapturedNode, CapturedViewModel } from "./capture";
|
||||||
|
|
||||||
|
export interface FrameOwnedAxNode {
|
||||||
|
nodeId: string;
|
||||||
|
frameId?: string;
|
||||||
|
parentId?: string;
|
||||||
|
childIds?: string[];
|
||||||
|
backendDOMNodeId?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface FrameAxBatch<T extends FrameOwnedAxNode> {
|
||||||
|
frameId: string;
|
||||||
|
nodes: T[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface FrameDocument<T extends FrameOwnedAxNode> {
|
||||||
|
frameId: string;
|
||||||
|
parentFrameId?: string;
|
||||||
|
ownerBackendNodeId?: number;
|
||||||
|
contextScopeId: string;
|
||||||
|
axNodes: T[];
|
||||||
|
domNodes: CapturedNode[];
|
||||||
|
}
|
||||||
|
|
||||||
|
interface Ownership {
|
||||||
|
frameId: string;
|
||||||
|
strength: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface OwnedCandidate<T> {
|
||||||
|
node: T;
|
||||||
|
ownership: Ownership;
|
||||||
|
}
|
||||||
|
|
||||||
|
function frameIds<T extends FrameOwnedAxNode>(
|
||||||
|
batches: FrameAxBatch<T>[],
|
||||||
|
captured: CapturedViewModel,
|
||||||
|
): string[] {
|
||||||
|
const ids = new Set<string>();
|
||||||
|
if (captured.rootFrameId) ids.add(captured.rootFrameId);
|
||||||
|
for (const frameId of captured.frameNodes?.keys() ?? []) ids.add(frameId);
|
||||||
|
for (const batch of batches) ids.add(batch.frameId);
|
||||||
|
return [...ids];
|
||||||
|
}
|
||||||
|
|
||||||
|
export function buildFrameDocuments<T extends FrameOwnedAxNode>(
|
||||||
|
batches: FrameAxBatch<T>[],
|
||||||
|
captured: CapturedViewModel,
|
||||||
|
): FrameDocument<T>[] {
|
||||||
|
const frames = frameIds(batches, captured);
|
||||||
|
const knownFrames = new Set(frames);
|
||||||
|
const backendOwner = new Map<number, string>();
|
||||||
|
for (const frameId of frames) {
|
||||||
|
for (const node of captured.frameNodes?.get(frameId) ?? []) {
|
||||||
|
backendOwner.set(node.backendNodeId, frameId);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const candidates = new Map<string, OwnedCandidate<T>>();
|
||||||
|
for (const batch of batches) {
|
||||||
|
const nodeById = new Map(batch.nodes.map((node) => [node.nodeId, node]));
|
||||||
|
const ownershipByNodeId = new Map<string, Ownership>();
|
||||||
|
const resolving = new Set<string>();
|
||||||
|
const resolveOwnership = (node: T): Ownership => {
|
||||||
|
const cached = ownershipByNodeId.get(node.nodeId);
|
||||||
|
if (cached) return cached;
|
||||||
|
if (node.frameId && knownFrames.has(node.frameId)) {
|
||||||
|
const ownership = { frameId: node.frameId, strength: 4 };
|
||||||
|
ownershipByNodeId.set(node.nodeId, ownership);
|
||||||
|
return ownership;
|
||||||
|
}
|
||||||
|
if (typeof node.backendDOMNodeId === "number") {
|
||||||
|
const owner = backendOwner.get(node.backendDOMNodeId);
|
||||||
|
if (owner) {
|
||||||
|
const ownership = { frameId: owner, strength: 3 };
|
||||||
|
ownershipByNodeId.set(node.nodeId, ownership);
|
||||||
|
return ownership;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (node.parentId && !resolving.has(node.nodeId)) {
|
||||||
|
const parent = nodeById.get(node.parentId);
|
||||||
|
if (parent) {
|
||||||
|
resolving.add(node.nodeId);
|
||||||
|
const inherited = resolveOwnership(parent);
|
||||||
|
resolving.delete(node.nodeId);
|
||||||
|
const ownership = {
|
||||||
|
frameId: inherited.frameId,
|
||||||
|
strength: Math.min(2, inherited.strength),
|
||||||
|
};
|
||||||
|
ownershipByNodeId.set(node.nodeId, ownership);
|
||||||
|
return ownership;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const ownership = { frameId: batch.frameId, strength: 1 };
|
||||||
|
ownershipByNodeId.set(node.nodeId, ownership);
|
||||||
|
return ownership;
|
||||||
|
};
|
||||||
|
|
||||||
|
for (const node of batch.nodes) {
|
||||||
|
const ownership = resolveOwnership(node);
|
||||||
|
const existing = candidates.get(node.nodeId);
|
||||||
|
if (!existing || ownership.strength > existing.ownership.strength) {
|
||||||
|
candidates.set(node.nodeId, { node, ownership });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const ownershipByNodeId = new Map(
|
||||||
|
[...candidates].map(([nodeId, candidate]) => [nodeId, candidate.ownership.frameId]),
|
||||||
|
);
|
||||||
|
const axNodesByFrame = new Map<string, T[]>();
|
||||||
|
for (const { node, ownership } of candidates.values()) {
|
||||||
|
const parentFrameId = node.parentId ? ownershipByNodeId.get(node.parentId) : undefined;
|
||||||
|
const childIds = node.childIds?.filter(
|
||||||
|
(childId) => ownershipByNodeId.get(childId) === ownership.frameId,
|
||||||
|
);
|
||||||
|
const ownedNode = {
|
||||||
|
...node,
|
||||||
|
frameId: ownership.frameId,
|
||||||
|
...(parentFrameId === ownership.frameId
|
||||||
|
? { parentId: node.parentId }
|
||||||
|
: { parentId: undefined }),
|
||||||
|
...(childIds ? { childIds } : {}),
|
||||||
|
};
|
||||||
|
const nodes = axNodesByFrame.get(ownership.frameId) ?? [];
|
||||||
|
nodes.push(ownedNode);
|
||||||
|
axNodesByFrame.set(ownership.frameId, nodes);
|
||||||
|
}
|
||||||
|
|
||||||
|
return frames.map((frameId) => ({
|
||||||
|
frameId,
|
||||||
|
...(captured.frameParentIds?.get(frameId)
|
||||||
|
? { parentFrameId: captured.frameParentIds.get(frameId) }
|
||||||
|
: {}),
|
||||||
|
...(captured.frameOwnerBackendNodeIds?.get(frameId) !== undefined
|
||||||
|
? { ownerBackendNodeId: captured.frameOwnerBackendNodeIds.get(frameId) }
|
||||||
|
: {}),
|
||||||
|
contextScopeId: frameId,
|
||||||
|
axNodes: axNodesByFrame.get(frameId) ?? [],
|
||||||
|
domNodes: captured.frameNodes?.get(frameId) ?? [],
|
||||||
|
}));
|
||||||
|
}
|
||||||
@@ -20,7 +20,7 @@ export interface EventSourceLike {
|
|||||||
export interface ObservationClientDeps {
|
export interface ObservationClientDeps {
|
||||||
fetchFn: (
|
fetchFn: (
|
||||||
url: string,
|
url: string,
|
||||||
init?: { method?: string; body?: string },
|
init?: { method?: string; body?: string; headers?: Record<string, string> },
|
||||||
) => Promise<{
|
) => Promise<{
|
||||||
ok: boolean;
|
ok: boolean;
|
||||||
json(): Promise<unknown>;
|
json(): Promise<unknown>;
|
||||||
@@ -282,6 +282,7 @@ export class ObservationClientStore {
|
|||||||
try {
|
try {
|
||||||
const res = await this.deps.fetchFn(INTERRUPT_URL, {
|
const res = await this.deps.fetchFn(INTERRUPT_URL, {
|
||||||
method: "POST",
|
method: "POST",
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
body: JSON.stringify(sessionId === undefined ? {} : { sessionId }),
|
body: JSON.stringify(sessionId === undefined ? {} : { sessionId }),
|
||||||
});
|
});
|
||||||
if (!res.ok) return false;
|
if (!res.ok) return false;
|
||||||
|
|||||||
@@ -1,16 +1,18 @@
|
|||||||
/**
|
/**
|
||||||
* ObservationService: per-owned-session live observation state for the PiP
|
* ObservationService: per-owned-session live observation state for the PiP
|
||||||
* overlay — current action, page url, and a breathing thumbnail carried by
|
* overlay — current action, page url, and a breathing thumbnail identified by
|
||||||
* attachment reference. Strict ownership boundary applies throughout: only
|
* an ephemeral frame id. Frames live in a per-session 2-slot ring in process
|
||||||
* sessions in the plugin's SessionRegistry (owned by construction) ever get
|
* memory (never the durable attachment store). Strict ownership boundary
|
||||||
* an observation entry, and interrupt/kill paths can only reach children this
|
* applies throughout: only sessions in the plugin's SessionRegistry (owned by
|
||||||
* plugin spawned.
|
* construction) ever get an observation entry, and interrupt/kill paths can
|
||||||
|
* only reach children this plugin spawned.
|
||||||
*
|
*
|
||||||
* Observation traffic isolation: thumbnail captures run through the runner
|
* Observation traffic isolation: thumbnail captures run through the runner
|
||||||
* directly (never through the tool-level instrumentation), emit no action
|
* directly (never through the tool-level instrumentation), emit no action
|
||||||
* events, and never move the registry's current pointer.
|
* events, and never move the registry's current pointer.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
import { createHash } from "node:crypto";
|
||||||
import { readFile, unlink } from "node:fs/promises";
|
import { readFile, unlink } from "node:fs/promises";
|
||||||
import { tmpdir } from "node:os";
|
import { tmpdir } from "node:os";
|
||||||
import { join } from "node:path";
|
import { join } from "node:path";
|
||||||
@@ -28,7 +30,7 @@ export interface SessionObservation {
|
|||||||
action: string;
|
action: string;
|
||||||
/** Epoch ms when the current action started (elapsed time is client-side). */
|
/** Epoch ms when the current action started (elapsed time is client-side). */
|
||||||
since: number;
|
since: number;
|
||||||
/** Latest thumbnail, by attachment-store reference (never bytes on the wire). */
|
/** Latest thumbnail id (ephemeral, plugin-owned; never bytes on the wire). */
|
||||||
thumbnailAttachmentId?: string;
|
thumbnailAttachmentId?: string;
|
||||||
/** Summary of the most recent failed action (drives the red status dot). */
|
/** Summary of the most recent failed action (drives the red status dot). */
|
||||||
lastError?: string;
|
lastError?: string;
|
||||||
@@ -52,28 +54,6 @@ export interface ObservationOptions {
|
|||||||
idleIntervalMs: number;
|
idleIntervalMs: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Structural view of the optional attachment seam (absent in headless compositions). */
|
|
||||||
interface AttachmentLike {
|
|
||||||
saveImage(input: {
|
|
||||||
data: Uint8Array;
|
|
||||||
mediaType: string;
|
|
||||||
name?: string;
|
|
||||||
}): Promise<ImageAttachmentRefLike>;
|
|
||||||
readImage(
|
|
||||||
ref: ImageAttachmentRefLike,
|
|
||||||
): Promise<{ data: Uint8Array; attachment: ImageAttachmentRefLike }>;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** The attachment reference fields the store verifies reads against. */
|
|
||||||
export interface ImageAttachmentRefLike {
|
|
||||||
attachmentId: unknown;
|
|
||||||
mediaType: string;
|
|
||||||
bytes: number;
|
|
||||||
width: number;
|
|
||||||
height: number;
|
|
||||||
name?: string;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Injectable clock/scheduler bits for tests. */
|
/** Injectable clock/scheduler bits for tests. */
|
||||||
export interface ObservationScheduler {
|
export interface ObservationScheduler {
|
||||||
setTimeout: (fn: () => void, ms: number) => unknown;
|
setTimeout: (fn: () => void, ms: number) => unknown;
|
||||||
@@ -94,6 +74,8 @@ const DEFAULT_SCHEDULER: ObservationScheduler = {
|
|||||||
|
|
||||||
/** Max consecutive capture failures before a session drops to the idle cadence. */
|
/** Max consecutive capture failures before a session drops to the idle cadence. */
|
||||||
const FAILURE_BACKOFF_THRESHOLD = 3;
|
const FAILURE_BACKOFF_THRESHOLD = 3;
|
||||||
|
/** Current + previous frame, so an in-flight HTTP fetch of the old id still lands. */
|
||||||
|
const FRAME_RING_SIZE = 2;
|
||||||
|
|
||||||
export class ObservationService {
|
export class ObservationService {
|
||||||
private readonly observations = new Map<string, SessionObservation>();
|
private readonly observations = new Map<string, SessionObservation>();
|
||||||
@@ -102,8 +84,14 @@ export class ObservationService {
|
|||||||
private readonly captureInFlight = new Set<string>();
|
private readonly captureInFlight = new Set<string>();
|
||||||
private readonly captureFailures = new Map<string, number>();
|
private readonly captureFailures = new Map<string, number>();
|
||||||
private readonly lastActivity = new Map<string, number>();
|
private readonly lastActivity = new Map<string, number>();
|
||||||
/** attachmentId → full reference (the store requires it for verified reads). */
|
/** Ephemeral frame id → PNG bytes. Only the live ring members are present. */
|
||||||
private readonly thumbRefs = new Map<string, ImageAttachmentRefLike>();
|
private readonly frames = new Map<string, { data: Uint8Array; mediaType: string }>();
|
||||||
|
/** sessionId → oldest-first frame ids, length ≤ FRAME_RING_SIZE. */
|
||||||
|
private readonly rings = new Map<string, string[]>();
|
||||||
|
/** sessionId → sha256 of the current frame (skip republish when unchanged). */
|
||||||
|
private readonly hashes = new Map<string, string>();
|
||||||
|
/** sessionId → last issued sequence number. */
|
||||||
|
private readonly seqs = new Map<string, number>();
|
||||||
/** Consecutive capture failures across ALL sessions (daemon-level signal). */
|
/** Consecutive capture failures across ALL sessions (daemon-level signal). */
|
||||||
private globalFailures = 0;
|
private globalFailures = 0;
|
||||||
private available = true;
|
private available = true;
|
||||||
@@ -179,10 +167,7 @@ export class ObservationService {
|
|||||||
this.cancelCapture(sessionId);
|
this.cancelCapture(sessionId);
|
||||||
this.captureFailures.delete(sessionId);
|
this.captureFailures.delete(sessionId);
|
||||||
this.lastActivity.delete(sessionId);
|
this.lastActivity.delete(sessionId);
|
||||||
const entry = this.observations.get(sessionId);
|
this.dropSessionFrames(sessionId);
|
||||||
if (entry?.thumbnailAttachmentId !== undefined) {
|
|
||||||
this.thumbRefs.delete(entry.thumbnailAttachmentId);
|
|
||||||
}
|
|
||||||
if (this.observations.delete(sessionId)) {
|
if (this.observations.delete(sessionId)) {
|
||||||
this.emit({
|
this.emit({
|
||||||
type: "remove",
|
type: "remove",
|
||||||
@@ -243,24 +228,17 @@ export class ObservationService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Read one captured thumbnail back through the store's verified path.
|
* Read one captured thumbnail from the in-process ring. Powers the plugin's
|
||||||
* Powers the plugin's own HTTP thumbnail route — frames are plugin-owned
|
* own HTTP thumbnail route — frames are plugin-owned runtime data, never
|
||||||
* runtime data, never referenced by any session log, so the
|
* referenced by any session log, so the session-authorized client RPC
|
||||||
* session-authorized client RPC cannot serve them.
|
* cannot serve them.
|
||||||
*/
|
*/
|
||||||
async readThumbnail(
|
async readThumbnail(
|
||||||
attachmentId: string,
|
attachmentId: string,
|
||||||
): Promise<{ data: Uint8Array; mediaType: string } | undefined> {
|
): Promise<{ data: Uint8Array; mediaType: string } | undefined> {
|
||||||
const ref = this.thumbRefs.get(attachmentId);
|
const frame = this.frames.get(attachmentId);
|
||||||
if (ref === undefined) return undefined;
|
if (frame === undefined) return undefined;
|
||||||
const attachments = this.deps.ctx.get("attachments") as AttachmentLike | undefined;
|
return { data: frame.data, mediaType: frame.mediaType };
|
||||||
if (attachments === undefined) return undefined;
|
|
||||||
try {
|
|
||||||
const stored = await attachments.readImage(ref);
|
|
||||||
return { data: stored.data, mediaType: ref.mediaType };
|
|
||||||
} catch {
|
|
||||||
return undefined;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Tear down all state and timers (plugin dispose). */
|
/** Tear down all state and timers (plugin dispose). */
|
||||||
@@ -271,7 +249,10 @@ export class ObservationService {
|
|||||||
this.captureInFlight.clear();
|
this.captureInFlight.clear();
|
||||||
this.captureFailures.clear();
|
this.captureFailures.clear();
|
||||||
this.lastActivity.clear();
|
this.lastActivity.clear();
|
||||||
this.thumbRefs.clear();
|
this.frames.clear();
|
||||||
|
this.rings.clear();
|
||||||
|
this.hashes.clear();
|
||||||
|
this.seqs.clear();
|
||||||
this.observations.clear();
|
this.observations.clear();
|
||||||
this.emit({ type: "reset" });
|
this.emit({ type: "reset" });
|
||||||
this.listeners.clear();
|
this.listeners.clear();
|
||||||
@@ -313,8 +294,8 @@ export class ObservationService {
|
|||||||
|
|
||||||
/**
|
/**
|
||||||
* Capture one frame: `bsk screenshot --json` through the runner, bytes into
|
* Capture one frame: `bsk screenshot --json` through the runner, bytes into
|
||||||
* the attachment store, reference onto the observation. Runs OUTSIDE the
|
* the in-process ring, id onto the observation. Runs OUTSIDE the tool
|
||||||
* tool instrumentation on purpose — no action events, no registry writes.
|
* instrumentation on purpose — no action events, no registry writes.
|
||||||
* Failures keep the previous frame and back off silently.
|
* Failures keep the previous frame and back off silently.
|
||||||
*/
|
*/
|
||||||
private async capture(sessionId: string): Promise<void> {
|
private async capture(sessionId: string): Promise<void> {
|
||||||
@@ -322,11 +303,6 @@ export class ObservationService {
|
|||||||
const current = this.observations.get(sessionId);
|
const current = this.observations.get(sessionId);
|
||||||
if (current === undefined || current.dead === true) return;
|
if (current === undefined || current.dead === true) return;
|
||||||
if (this.captureInFlight.has(sessionId)) return;
|
if (this.captureInFlight.has(sessionId)) return;
|
||||||
const attachments = this.deps.ctx.get("attachments") as AttachmentLike | undefined;
|
|
||||||
if (attachments === undefined) {
|
|
||||||
// No attachment store (headless composition): thumbnails stay absent.
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
this.captureInFlight.add(sessionId);
|
this.captureInFlight.add(sessionId);
|
||||||
// One fixed scratch path per session (overwritten each frame) and always
|
// One fixed scratch path per session (overwritten each frame) and always
|
||||||
// unlinked after the bytes are read — /tmp must not grow without bound.
|
// unlinked after the bytes are read — /tmp must not grow without bound.
|
||||||
@@ -355,23 +331,12 @@ export class ObservationService {
|
|||||||
const reply = JSON.parse(result.stdout) as { path?: string };
|
const reply = JSON.parse(result.stdout) as { path?: string };
|
||||||
writtenPath = reply.path ?? outPath;
|
writtenPath = reply.path ?? outPath;
|
||||||
const data = await readFile(writtenPath);
|
const data = await readFile(writtenPath);
|
||||||
const ref = await attachments.saveImage({
|
|
||||||
data,
|
|
||||||
mediaType: "image/png",
|
|
||||||
name: `observation-${sessionId}.png`,
|
|
||||||
});
|
|
||||||
const entry = this.observations.get(sessionId);
|
const entry = this.observations.get(sessionId);
|
||||||
if (entry === undefined) return;
|
if (entry === undefined) return;
|
||||||
this.captureFailures.delete(sessionId);
|
this.captureFailures.delete(sessionId);
|
||||||
this.globalFailures = 0;
|
this.globalFailures = 0;
|
||||||
this.setAvailable(true);
|
this.setAvailable(true);
|
||||||
// Drop the replaced frame's reference — the store keeps the bytes, but
|
this.publishFrame(sessionId, entry, data);
|
||||||
// this map (and the old id's read path) must not grow without bound.
|
|
||||||
if (entry.thumbnailAttachmentId !== undefined) {
|
|
||||||
this.thumbRefs.delete(entry.thumbnailAttachmentId);
|
|
||||||
}
|
|
||||||
this.thumbRefs.set(String(ref.attachmentId), ref);
|
|
||||||
this.put({ ...entry, thumbnailAttachmentId: String(ref.attachmentId) });
|
|
||||||
} catch {
|
} catch {
|
||||||
// Silent by design: keep the previous frame, count toward backoff.
|
// Silent by design: keep the previous frame, count toward backoff.
|
||||||
this.captureFailures.set(sessionId, (this.captureFailures.get(sessionId) ?? 0) + 1);
|
this.captureFailures.set(sessionId, (this.captureFailures.get(sessionId) ?? 0) + 1);
|
||||||
@@ -383,6 +348,33 @@ export class ObservationService {
|
|||||||
if (!this.disposed && this.observations.has(sessionId)) this.scheduleCapture(sessionId);
|
if (!this.disposed && this.observations.has(sessionId)) this.scheduleCapture(sessionId);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Insert a new frame, or no-op when the PNG bytes match the current one. */
|
||||||
|
private publishFrame(sessionId: string, entry: SessionObservation, data: Uint8Array): void {
|
||||||
|
const hash = createHash("sha256").update(data).digest("hex");
|
||||||
|
if (this.hashes.get(sessionId) === hash) return;
|
||||||
|
const seq = (this.seqs.get(sessionId) ?? 0) + 1;
|
||||||
|
this.seqs.set(sessionId, seq);
|
||||||
|
const id = `obs-${sessionId}-${seq}`;
|
||||||
|
this.frames.set(id, { data, mediaType: "image/png" });
|
||||||
|
this.hashes.set(sessionId, hash);
|
||||||
|
const ring = this.rings.get(sessionId) ?? [];
|
||||||
|
ring.push(id);
|
||||||
|
while (ring.length > FRAME_RING_SIZE) {
|
||||||
|
const evicted = ring.shift();
|
||||||
|
if (evicted !== undefined) this.frames.delete(evicted);
|
||||||
|
}
|
||||||
|
this.rings.set(sessionId, ring);
|
||||||
|
this.put({ ...entry, thumbnailAttachmentId: id });
|
||||||
|
}
|
||||||
|
|
||||||
|
private dropSessionFrames(sessionId: string): void {
|
||||||
|
for (const id of this.rings.get(sessionId) ?? []) this.frames.delete(id);
|
||||||
|
this.rings.delete(sessionId);
|
||||||
|
this.hashes.delete(sessionId);
|
||||||
|
this.seqs.delete(sessionId);
|
||||||
|
}
|
||||||
|
|
||||||
/** Mark a session dead: grey it out, stop asking for frames, keep the entry for the strip. */
|
/** Mark a session dead: grey it out, stop asking for frames, keep the entry for the strip. */
|
||||||
private markDead(sessionId: string): void {
|
private markDead(sessionId: string): void {
|
||||||
const entry = this.observations.get(sessionId);
|
const entry = this.observations.get(sessionId);
|
||||||
|
|||||||
@@ -9,15 +9,23 @@ const OBS_IDLE: SessionObservation = { sessionId: "s1", action: "idle", since: 1
|
|||||||
const OBS_BUSY: SessionObservation = { sessionId: "s1", action: "clicking", since: 2000 };
|
const OBS_BUSY: SessionObservation = { sessionId: "s1", action: "clicking", since: 2000 };
|
||||||
|
|
||||||
function harness(state: SessionObservation[] = []) {
|
function harness(state: SessionObservation[] = []) {
|
||||||
const fetches: { url: string; init?: { method?: string; body?: string } }[] = [];
|
const fetches: {
|
||||||
|
url: string;
|
||||||
|
init?: { method?: string; body?: string; headers?: Record<string, string> };
|
||||||
|
}[] = [];
|
||||||
const esInstances: EventSourceLike[] = [];
|
const esInstances: EventSourceLike[] = [];
|
||||||
const fetchFn = vi.fn(async (url: string, init?: { method?: string; body?: string }) => {
|
const fetchFn = vi.fn(
|
||||||
fetches.push({ url, init });
|
async (
|
||||||
if (url === "/bsk-observation/state") {
|
url: string,
|
||||||
return { ok: true, json: async () => ({ sessions: state }) };
|
init?: { method?: string; body?: string; headers?: Record<string, string> },
|
||||||
}
|
) => {
|
||||||
return { ok: true, json: async () => ({ interrupted: true }) };
|
fetches.push({ url, init });
|
||||||
});
|
if (url === "/bsk-observation/state") {
|
||||||
|
return { ok: true, json: async () => ({ sessions: state }) };
|
||||||
|
}
|
||||||
|
return { ok: true, json: async () => ({ interrupted: true }) };
|
||||||
|
},
|
||||||
|
);
|
||||||
const eventSourceFactory = (url: string) => {
|
const eventSourceFactory = (url: string) => {
|
||||||
const es: EventSourceLike = { onmessage: null, close: vi.fn() };
|
const es: EventSourceLike = { onmessage: null, close: vi.fn() };
|
||||||
esInstances.push(es);
|
esInstances.push(es);
|
||||||
@@ -89,7 +97,9 @@ describe("ObservationClientStore", () => {
|
|||||||
expect(await store.interrupt()).toBe(true);
|
expect(await store.interrupt()).toBe(true);
|
||||||
expect(await store.interrupt("s1")).toBe(true);
|
expect(await store.interrupt("s1")).toBe(true);
|
||||||
const calls = fetches.filter((f) => f.url === "/bsk-observation/interrupt");
|
const calls = fetches.filter((f) => f.url === "/bsk-observation/interrupt");
|
||||||
|
expect(calls[0].init?.headers?.["content-type"]).toBe("application/json");
|
||||||
expect(JSON.parse(calls[0].init?.body ?? "")).toEqual({});
|
expect(JSON.parse(calls[0].init?.body ?? "")).toEqual({});
|
||||||
|
expect(calls[1].init?.headers?.["content-type"]).toBe("application/json");
|
||||||
expect(JSON.parse(calls[1].init?.body ?? "")).toEqual({ sessionId: "s1" });
|
expect(JSON.parse(calls[1].init?.body ?? "")).toEqual({ sessionId: "s1" });
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
@@ -58,7 +58,12 @@ interface FakeRunner extends BskRunner {
|
|||||||
|
|
||||||
/** Runner that answers screenshot with a real temp PNG and records kills. */
|
/** Runner that answers screenshot with a real temp PNG and records kills. */
|
||||||
function fakeRunner(
|
function fakeRunner(
|
||||||
opts: { screenshotFails?: boolean; screenshotNotFound?: boolean; killCount?: number } = {},
|
opts: {
|
||||||
|
screenshotFails?: boolean;
|
||||||
|
screenshotNotFound?: boolean;
|
||||||
|
killCount?: number;
|
||||||
|
screenshotBytes?: () => Uint8Array;
|
||||||
|
} = {},
|
||||||
): FakeRunner {
|
): FakeRunner {
|
||||||
const calls: FakeRunner["calls"] = [];
|
const calls: FakeRunner["calls"] = [];
|
||||||
const killed: string[] = [];
|
const killed: string[] = [];
|
||||||
@@ -82,7 +87,7 @@ function fakeRunner(
|
|||||||
}
|
}
|
||||||
const outIndex = args.indexOf("--out") + 1;
|
const outIndex = args.indexOf("--out") + 1;
|
||||||
const out = args[outIndex];
|
const out = args[outIndex];
|
||||||
writeFileSync(out, Buffer.from([0x89, 0x50, 0x4e, 0x47]));
|
writeFileSync(out, opts.screenshotBytes?.() ?? Buffer.from([0x89, 0x50, 0x4e, 0x47]));
|
||||||
return {
|
return {
|
||||||
code: 0,
|
code: 0,
|
||||||
stdout: JSON.stringify({
|
stdout: JSON.stringify({
|
||||||
@@ -108,38 +113,20 @@ function fakeRunner(
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
function fakeCtx(attachments?: unknown) {
|
function fakeCtx() {
|
||||||
return { get: (key: string) => (key === "attachments" ? attachments : undefined) } as never;
|
return { get: () => undefined } as never;
|
||||||
}
|
}
|
||||||
|
|
||||||
function setup(opts: {
|
function setup(opts: {
|
||||||
runner?: FakeRunner;
|
runner?: FakeRunner;
|
||||||
attachments?: unknown;
|
|
||||||
registry?: SessionRegistry;
|
registry?: SessionRegistry;
|
||||||
scheduler?: ReturnType<typeof fakeScheduler>;
|
scheduler?: ReturnType<typeof fakeScheduler>;
|
||||||
}) {
|
}) {
|
||||||
const registry = opts.registry ?? new SessionRegistry(5);
|
const registry = opts.registry ?? new SessionRegistry(5);
|
||||||
const runner = opts.runner ?? fakeRunner();
|
const runner = opts.runner ?? fakeRunner();
|
||||||
const scheduler = opts.scheduler ?? fakeScheduler();
|
const scheduler = opts.scheduler ?? fakeScheduler();
|
||||||
const attachments =
|
|
||||||
"attachments" in opts
|
|
||||||
? opts.attachments
|
|
||||||
: {
|
|
||||||
saveImage: async (input: { data: Uint8Array; name?: string }) => ({
|
|
||||||
attachmentId: "att-1",
|
|
||||||
mediaType: "image/png",
|
|
||||||
bytes: input.data.byteLength,
|
|
||||||
width: 4,
|
|
||||||
height: 4,
|
|
||||||
name: input.name,
|
|
||||||
}),
|
|
||||||
readImage: async (ref: { attachmentId: unknown }) => ({
|
|
||||||
data: new Uint8Array([1, 2, 3]),
|
|
||||||
attachment: ref,
|
|
||||||
}),
|
|
||||||
};
|
|
||||||
const service = new ObservationService({
|
const service = new ObservationService({
|
||||||
ctx: fakeCtx(attachments),
|
ctx: fakeCtx(),
|
||||||
runner,
|
runner,
|
||||||
registry,
|
registry,
|
||||||
queue: new KeyedExecutor(),
|
queue: new KeyedExecutor(),
|
||||||
@@ -157,11 +144,6 @@ function own(registry: SessionRegistry, sessionId: string): void {
|
|||||||
registry.completeStart({ sessionId, startedAtMs: 1 });
|
registry.completeStart({ sessionId, startedAtMs: 1 });
|
||||||
}
|
}
|
||||||
|
|
||||||
async function flushMicrotasks(): Promise<void> {
|
|
||||||
// setImmediate yields to the poll phase so real fs reads inside capture() settle.
|
|
||||||
for (let i = 0; i < 20; i++) await new Promise((resolve) => setImmediate(resolve));
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Poll until the condition holds (capture completion is real async I/O). */
|
/** Poll until the condition holds (capture completion is real async I/O). */
|
||||||
async function waitFor(cond: () => boolean, timeoutMs = 2000): Promise<void> {
|
async function waitFor(cond: () => boolean, timeoutMs = 2000): Promise<void> {
|
||||||
const start = Date.now();
|
const start = Date.now();
|
||||||
@@ -218,7 +200,7 @@ describe("thumbnail cadence", () => {
|
|||||||
expect(scheduler.pending()).toEqual([0]);
|
expect(scheduler.pending()).toEqual([0]);
|
||||||
scheduler.runNext();
|
scheduler.runNext();
|
||||||
await waitFor(() => captures() === 1 && scheduler.pending().length === 1);
|
await waitFor(() => captures() === 1 && scheduler.pending().length === 1);
|
||||||
expect(service.getState()[0].thumbnailAttachmentId).toBe("att-1");
|
expect(service.getState()[0].thumbnailAttachmentId).toBe("obs-s1-1");
|
||||||
// Just captured with fresh activity: next frame on the fast cadence.
|
// Just captured with fresh activity: next frame on the fast cadence.
|
||||||
expect(scheduler.pending()).toEqual([1500]);
|
expect(scheduler.pending()).toEqual([1500]);
|
||||||
scheduler.runNext();
|
scheduler.runNext();
|
||||||
@@ -259,14 +241,14 @@ describe("thumbnail cadence", () => {
|
|||||||
expect(service.getState()[0].thumbnailAttachmentId).toBeUndefined();
|
expect(service.getState()[0].thumbnailAttachmentId).toBeUndefined();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("stays silent without an attachment store (headless composition)", async () => {
|
it("captures frames without an attachment store", async () => {
|
||||||
const scheduler = fakeScheduler();
|
const scheduler = fakeScheduler();
|
||||||
const runner = fakeRunner();
|
const runner = fakeRunner();
|
||||||
const { service } = setup({ runner, scheduler, attachments: undefined });
|
const { service } = setup({ runner, scheduler });
|
||||||
service.addSession("s1");
|
service.addSession("s1");
|
||||||
scheduler.runNext();
|
scheduler.runNext();
|
||||||
await flushMicrotasks();
|
await waitFor(() => service.getState()[0]?.thumbnailAttachmentId === "obs-s1-1");
|
||||||
expect(runner.calls.filter((c) => c.args[0] === "screenshot")).toHaveLength(0);
|
expect((await service.readThumbnail("obs-s1-1"))?.mediaType).toBe("image/png");
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -286,7 +268,7 @@ describe("observation traffic isolation", () => {
|
|||||||
const captureEvent = captureEvents[0];
|
const captureEvent = captureEvents[0];
|
||||||
if (captureEvent.type !== "upsert") throw new Error("unreachable");
|
if (captureEvent.type !== "upsert") throw new Error("unreachable");
|
||||||
expect(captureEvent.session?.action).toBe("idle");
|
expect(captureEvent.session?.action).toBe("idle");
|
||||||
expect(captureEvent.session?.thumbnailAttachmentId).toBe("att-1");
|
expect(captureEvent.session?.thumbnailAttachmentId).toBe("obs-s1-1");
|
||||||
// …and the registry's current pointer never moved through observation.
|
// …and the registry's current pointer never moved through observation.
|
||||||
expect(registry.current()).toBe("s1");
|
expect(registry.current()).toBe("s1");
|
||||||
expect(registry.list()[0].startedAtMs).toBe(1);
|
expect(registry.list()[0].startedAtMs).toBe(1);
|
||||||
@@ -551,9 +533,9 @@ describe("thumbnail read-back", () => {
|
|||||||
service.addSession("s1");
|
service.addSession("s1");
|
||||||
scheduler.runNext();
|
scheduler.runNext();
|
||||||
await waitFor(() => service.getState()[0]?.thumbnailAttachmentId !== undefined);
|
await waitFor(() => service.getState()[0]?.thumbnailAttachmentId !== undefined);
|
||||||
const frame = await service.readThumbnail("att-1");
|
const frame = await service.readThumbnail("obs-s1-1");
|
||||||
expect(frame?.mediaType).toBe("image/png");
|
expect(frame?.mediaType).toBe("image/png");
|
||||||
expect([...(frame?.data ?? [])]).toEqual([1, 2, 3]);
|
expect([...(frame?.data ?? [])]).toEqual([0x89, 0x50, 0x4e, 0x47]);
|
||||||
expect(await service.readThumbnail("nope")).toBeUndefined();
|
expect(await service.readThumbnail("nope")).toBeUndefined();
|
||||||
service.dispose();
|
service.dispose();
|
||||||
});
|
});
|
||||||
@@ -598,47 +580,65 @@ describe("scratch files and thumbnail references", () => {
|
|||||||
service.dispose();
|
service.dispose();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("drops the replaced frame reference and clears it on remove", async () => {
|
it("keeps the last two distinct frames and evicts the third", async () => {
|
||||||
let counter = 0;
|
let n = 0;
|
||||||
const attachments = {
|
const runner = fakeRunner({
|
||||||
saveImage: async (input: { data: Uint8Array; name?: string }) => {
|
screenshotBytes: () => Uint8Array.from([0x89, 0x50, 0x4e, 0x47, n++]),
|
||||||
counter += 1;
|
});
|
||||||
return {
|
|
||||||
attachmentId: `att-${counter}`,
|
|
||||||
mediaType: "image/png",
|
|
||||||
bytes: input.data.byteLength,
|
|
||||||
width: 4,
|
|
||||||
height: 4,
|
|
||||||
name: input.name,
|
|
||||||
};
|
|
||||||
},
|
|
||||||
readImage: async (ref: { attachmentId: unknown }) => ({
|
|
||||||
data: new Uint8Array([1, 2, 3]),
|
|
||||||
attachment: ref,
|
|
||||||
}),
|
|
||||||
};
|
|
||||||
const scheduler = fakeScheduler();
|
const scheduler = fakeScheduler();
|
||||||
const { service } = setup({ scheduler, attachments });
|
const { service } = setup({ scheduler, runner });
|
||||||
service.addSession("s1");
|
service.addSession("s1");
|
||||||
scheduler.runNext();
|
scheduler.runNext();
|
||||||
await waitFor(
|
await waitFor(
|
||||||
() =>
|
() =>
|
||||||
service.getState()[0]?.thumbnailAttachmentId === "att-1" &&
|
service.getState()[0]?.thumbnailAttachmentId === "obs-s1-1" &&
|
||||||
scheduler.pending().length === 1,
|
scheduler.pending().length === 1,
|
||||||
);
|
);
|
||||||
service.endAction("s1");
|
service.endAction("s1");
|
||||||
scheduler.runNext();
|
scheduler.runNext();
|
||||||
await waitFor(
|
await waitFor(
|
||||||
() =>
|
() =>
|
||||||
service.getState()[0]?.thumbnailAttachmentId === "att-2" &&
|
service.getState()[0]?.thumbnailAttachmentId === "obs-s1-2" &&
|
||||||
scheduler.pending().length === 1,
|
scheduler.pending().length === 1,
|
||||||
);
|
);
|
||||||
// the replaced frame's read path is gone
|
expect((await service.readThumbnail("obs-s1-1"))?.mediaType).toBe("image/png");
|
||||||
expect(await service.readThumbnail("att-1")).toBeUndefined();
|
expect((await service.readThumbnail("obs-s1-2"))?.mediaType).toBe("image/png");
|
||||||
expect((await service.readThumbnail("att-2"))?.mediaType).toBe("image/png");
|
service.endAction("s1");
|
||||||
// removing the session clears its live reference too
|
scheduler.runNext();
|
||||||
|
await waitFor(
|
||||||
|
() =>
|
||||||
|
service.getState()[0]?.thumbnailAttachmentId === "obs-s1-3" &&
|
||||||
|
scheduler.pending().length === 1,
|
||||||
|
);
|
||||||
|
expect(await service.readThumbnail("obs-s1-1")).toBeUndefined();
|
||||||
|
expect((await service.readThumbnail("obs-s1-2"))?.mediaType).toBe("image/png");
|
||||||
|
expect((await service.readThumbnail("obs-s1-3"))?.mediaType).toBe("image/png");
|
||||||
service.removeSession("s1");
|
service.removeSession("s1");
|
||||||
expect(await service.readThumbnail("att-2")).toBeUndefined();
|
expect(await service.readThumbnail("obs-s1-2")).toBeUndefined();
|
||||||
|
expect(await service.readThumbnail("obs-s1-3")).toBeUndefined();
|
||||||
|
service.dispose();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("reuses the current frame id when the PNG bytes are unchanged", async () => {
|
||||||
|
const scheduler = fakeScheduler();
|
||||||
|
const runner = fakeRunner();
|
||||||
|
const { service } = setup({ scheduler, runner });
|
||||||
|
service.addSession("s1");
|
||||||
|
scheduler.runNext();
|
||||||
|
await waitFor(
|
||||||
|
() =>
|
||||||
|
service.getState()[0]?.thumbnailAttachmentId === "obs-s1-1" &&
|
||||||
|
scheduler.pending().length === 1,
|
||||||
|
);
|
||||||
|
service.endAction("s1");
|
||||||
|
scheduler.runNext();
|
||||||
|
await waitFor(
|
||||||
|
() =>
|
||||||
|
runner.calls.filter((c) => c.args[0] === "screenshot").length >= 2 &&
|
||||||
|
scheduler.pending().length === 1,
|
||||||
|
);
|
||||||
|
expect(service.getState()[0]?.thumbnailAttachmentId).toBe("obs-s1-1");
|
||||||
|
expect((await service.readThumbnail("obs-s1-1"))?.mediaType).toBe("image/png");
|
||||||
service.dispose();
|
service.dispose();
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -733,6 +733,10 @@ describe("observation action instrumentation timing", () => {
|
|||||||
const navigateGate = new Promise<void>((resolve) => {
|
const navigateGate = new Promise<void>((resolve) => {
|
||||||
releaseNavigate = resolve;
|
releaseNavigate = resolve;
|
||||||
});
|
});
|
||||||
|
let releaseSnapshot!: () => void;
|
||||||
|
const snapshotGate = new Promise<void>((resolve) => {
|
||||||
|
releaseSnapshot = resolve;
|
||||||
|
});
|
||||||
const runner: BskRunner = {
|
const runner: BskRunner = {
|
||||||
async run(args: string[]): Promise<BskRunResult> {
|
async run(args: string[]): Promise<BskRunResult> {
|
||||||
if (args[0] === "session") {
|
if (args[0] === "session") {
|
||||||
@@ -754,6 +758,21 @@ describe("observation action instrumentation timing", () => {
|
|||||||
aborted: false,
|
aborted: false,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
if (args[0] === "snapshot") {
|
||||||
|
await snapshotGate;
|
||||||
|
return {
|
||||||
|
code: 0,
|
||||||
|
stdout: JSON.stringify({ text: "ok", ref_count: 0, tab_id: 7, truncated: false }),
|
||||||
|
stderr: "",
|
||||||
|
timedOut: false,
|
||||||
|
aborted: false,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
// Observation captures share the session queue; fail them fast so they
|
||||||
|
// cannot stall the action-label assertions.
|
||||||
|
if (args[0] === "screenshot") {
|
||||||
|
return { code: 1, stdout: "", stderr: "skip", timedOut: false, aborted: false };
|
||||||
|
}
|
||||||
return {
|
return {
|
||||||
code: 0,
|
code: 0,
|
||||||
stdout: JSON.stringify({ text: "ok", ref_count: 0, tab_id: 7, truncated: false }),
|
stdout: JSON.stringify({ text: "ok", ref_count: 0, tab_id: 7, truncated: false }),
|
||||||
@@ -792,6 +811,7 @@ describe("observation action instrumentation timing", () => {
|
|||||||
await vi.waitFor(() =>
|
await vi.waitFor(() =>
|
||||||
expect(observation.getState().find((s) => s.sessionId === "s1")?.action).toBe("snapshotting"),
|
expect(observation.getState().find((s) => s.sessionId === "s1")?.action).toBe("snapshotting"),
|
||||||
);
|
);
|
||||||
|
releaseSnapshot();
|
||||||
await b;
|
await b;
|
||||||
await vi.waitFor(() =>
|
await vi.waitFor(() =>
|
||||||
expect(observation.getState().find((s) => s.sessionId === "s1")?.action).toBe("idle"),
|
expect(observation.getState().find((s) => s.sessionId === "s1")?.action).toBe("idle"),
|
||||||
|
|||||||
@@ -25,6 +25,85 @@ function scene(nodes: VomNode[]): VomScene {
|
|||||||
}
|
}
|
||||||
|
|
||||||
describe("renderVom single-layer page", () => {
|
describe("renderVom single-layer page", () => {
|
||||||
|
it("does not derive handle context across frame scopes", () => {
|
||||||
|
const out = renderVom(
|
||||||
|
scene([
|
||||||
|
node({
|
||||||
|
id: 1,
|
||||||
|
role: "RootWebArea",
|
||||||
|
frameId: "main",
|
||||||
|
contextScopeId: "main",
|
||||||
|
}),
|
||||||
|
node({
|
||||||
|
id: 2,
|
||||||
|
parentId: 1,
|
||||||
|
role: "StaticText",
|
||||||
|
name: "创建于",
|
||||||
|
frameId: "main",
|
||||||
|
contextScopeId: "main",
|
||||||
|
}),
|
||||||
|
node({
|
||||||
|
id: 3,
|
||||||
|
parentId: 1,
|
||||||
|
role: "Iframe",
|
||||||
|
frameId: "main",
|
||||||
|
contextScopeId: "main",
|
||||||
|
}),
|
||||||
|
node({
|
||||||
|
id: 4,
|
||||||
|
parentId: 3,
|
||||||
|
role: "button",
|
||||||
|
name: "表格视图",
|
||||||
|
frameId: "child",
|
||||||
|
contextScopeId: "child",
|
||||||
|
}),
|
||||||
|
node({
|
||||||
|
id: 5,
|
||||||
|
parentId: 1,
|
||||||
|
role: "button",
|
||||||
|
name: "表格视图",
|
||||||
|
frameId: "main",
|
||||||
|
contextScopeId: "main",
|
||||||
|
}),
|
||||||
|
]),
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(out.text).toContain('@e1 button "表格视图"');
|
||||||
|
expect(out.text).not.toContain('@e1 button "表格视图" [ctx: 创建于]');
|
||||||
|
});
|
||||||
|
|
||||||
|
it("preserves handle context within a frame scope", () => {
|
||||||
|
const out = renderVom(
|
||||||
|
scene([
|
||||||
|
node({ id: 1, role: "RootWebArea", contextScopeId: "main" }),
|
||||||
|
node({ id: 2, parentId: 1, role: "Iframe", contextScopeId: "main" }),
|
||||||
|
node({
|
||||||
|
id: 3,
|
||||||
|
parentId: 2,
|
||||||
|
role: "StaticText",
|
||||||
|
name: "工具栏",
|
||||||
|
contextScopeId: "child",
|
||||||
|
}),
|
||||||
|
node({
|
||||||
|
id: 4,
|
||||||
|
parentId: 2,
|
||||||
|
role: "button",
|
||||||
|
name: "更多",
|
||||||
|
contextScopeId: "child",
|
||||||
|
}),
|
||||||
|
node({
|
||||||
|
id: 5,
|
||||||
|
parentId: 1,
|
||||||
|
role: "button",
|
||||||
|
name: "更多",
|
||||||
|
contextScopeId: "main",
|
||||||
|
}),
|
||||||
|
]),
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(out.text).toContain('@e1 button "更多" [ctx: 工具栏]');
|
||||||
|
});
|
||||||
|
|
||||||
it("always emits a complete @vom single-layer document when there is no blocker", () => {
|
it("always emits a complete @vom single-layer document when there is no blocker", () => {
|
||||||
const out = renderVom(
|
const out = renderVom(
|
||||||
scene([
|
scene([
|
||||||
@@ -157,7 +236,7 @@ describe("renderVom single-layer page", () => {
|
|||||||
expect(out.text).not.toContain("hunter2");
|
expect(out.text).not.toContain("hunter2");
|
||||||
});
|
});
|
||||||
|
|
||||||
it("does not imply a sensitive textbox has a value when the value is missing", () => {
|
it("renders a sensitive filled state without retaining its clear text value", () => {
|
||||||
const out = renderVom(
|
const out = renderVom(
|
||||||
scene([
|
scene([
|
||||||
node({ id: 1, role: "RootWebArea", name: "Login" }),
|
node({ id: 1, role: "RootWebArea", name: "Login" }),
|
||||||
@@ -168,6 +247,26 @@ describe("renderVom single-layer page", () => {
|
|||||||
name: "密码",
|
name: "密码",
|
||||||
tag: "input",
|
tag: "input",
|
||||||
sensitive: true,
|
sensitive: true,
|
||||||
|
inputState: "filled",
|
||||||
|
}),
|
||||||
|
]),
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(out.text).toContain('@e1 textbox "密码" [filled] ="•••"');
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not imply an empty sensitive textbox has a value", () => {
|
||||||
|
const out = renderVom(
|
||||||
|
scene([
|
||||||
|
node({ id: 1, role: "RootWebArea", name: "Login" }),
|
||||||
|
node({
|
||||||
|
id: 2,
|
||||||
|
parentId: 1,
|
||||||
|
role: "textbox",
|
||||||
|
name: "密码",
|
||||||
|
tag: "input",
|
||||||
|
sensitive: true,
|
||||||
|
inputState: "empty",
|
||||||
}),
|
}),
|
||||||
]),
|
]),
|
||||||
);
|
);
|
||||||
@@ -389,6 +488,102 @@ describe("renderVom single-layer page", () => {
|
|||||||
expect(out.refs).toEqual([{ ref: "e1", backendNodeId: 4 }]);
|
expect(out.refs).toEqual([{ ref: "e1", backendNodeId: 4 }]);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("keeps paint-order comparisons inside their frame document", () => {
|
||||||
|
const out = renderVom(
|
||||||
|
{
|
||||||
|
...scene([
|
||||||
|
node({ id: 1, role: "RootWebArea", frameId: "main" }),
|
||||||
|
node({
|
||||||
|
id: 2,
|
||||||
|
parentId: 1,
|
||||||
|
role: "button",
|
||||||
|
name: "Top-level action",
|
||||||
|
tag: "button",
|
||||||
|
rect: { x: 120, y: 120, w: 120, h: 40 },
|
||||||
|
paintOrder: 2,
|
||||||
|
frameId: "main",
|
||||||
|
}),
|
||||||
|
node({
|
||||||
|
id: 3,
|
||||||
|
parentId: 1,
|
||||||
|
role: "Iframe",
|
||||||
|
tag: "iframe",
|
||||||
|
rect: { x: 0, y: 0, w: 1000, h: 700 },
|
||||||
|
paintOrder: 1,
|
||||||
|
frameId: "main",
|
||||||
|
}),
|
||||||
|
node({
|
||||||
|
id: 4,
|
||||||
|
parentId: 3,
|
||||||
|
role: "generic",
|
||||||
|
rect: { x: 0, y: 0, w: 1000, h: 700 },
|
||||||
|
paintOrder: 100,
|
||||||
|
position: "fixed",
|
||||||
|
frameId: "child",
|
||||||
|
}),
|
||||||
|
node({
|
||||||
|
id: 5,
|
||||||
|
parentId: 4,
|
||||||
|
role: "button",
|
||||||
|
name: "Child action",
|
||||||
|
tag: "button",
|
||||||
|
rect: { x: 120, y: 120, w: 120, h: 40 },
|
||||||
|
paintOrder: 101,
|
||||||
|
frameId: "child",
|
||||||
|
}),
|
||||||
|
]),
|
||||||
|
rootFrameId: "main",
|
||||||
|
},
|
||||||
|
{ activeRegionPolicy: true },
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(out.text).toContain("@layers 1 focus=L1");
|
||||||
|
expect(out.text).toContain('button "Top-level action"');
|
||||||
|
expect(out.text).toContain('button "Child action"');
|
||||||
|
});
|
||||||
|
|
||||||
|
it("uses the iframe owner when a parent-frame region covers child content", () => {
|
||||||
|
const out = renderVom(
|
||||||
|
{
|
||||||
|
...scene([
|
||||||
|
node({ id: 1, role: "RootWebArea", frameId: "main" }),
|
||||||
|
node({
|
||||||
|
id: 2,
|
||||||
|
parentId: 1,
|
||||||
|
role: "Iframe",
|
||||||
|
tag: "iframe",
|
||||||
|
rect: { x: 100, y: 100, w: 400, h: 300 },
|
||||||
|
paintOrder: 1,
|
||||||
|
frameId: "main",
|
||||||
|
}),
|
||||||
|
node({
|
||||||
|
id: 3,
|
||||||
|
parentId: 2,
|
||||||
|
role: "button",
|
||||||
|
name: "Covered child action",
|
||||||
|
tag: "button",
|
||||||
|
rect: { x: 150, y: 150, w: 120, h: 40 },
|
||||||
|
paintOrder: 100,
|
||||||
|
frameId: "child",
|
||||||
|
}),
|
||||||
|
node({
|
||||||
|
id: 4,
|
||||||
|
parentId: 1,
|
||||||
|
role: "generic",
|
||||||
|
rect: { x: 120, y: 120, w: 200, h: 120 },
|
||||||
|
paintOrder: 2,
|
||||||
|
position: "fixed",
|
||||||
|
frameId: "main",
|
||||||
|
}),
|
||||||
|
]),
|
||||||
|
rootFrameId: "main",
|
||||||
|
},
|
||||||
|
{ activeRegionPolicy: true },
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(out.text).not.toContain("Covered child action");
|
||||||
|
});
|
||||||
|
|
||||||
it("renders conditional surface items inline on the trigger", () => {
|
it("renders conditional surface items inline on the trigger", () => {
|
||||||
const out = renderVom({
|
const out = renderVom({
|
||||||
...scene([
|
...scene([
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ export type {
|
|||||||
Viewport,
|
Viewport,
|
||||||
VomNode,
|
VomNode,
|
||||||
VomOptions,
|
VomOptions,
|
||||||
|
VomRef,
|
||||||
VomResult,
|
VomResult,
|
||||||
VomScene,
|
VomScene,
|
||||||
} from "./types";
|
} from "./types";
|
||||||
|
|||||||
@@ -50,10 +50,18 @@ function classifyLayer(
|
|||||||
return blockerCoverage >= MASK_COVERAGE_THRESHOLD ? "mask" : "modal";
|
return blockerCoverage >= MASK_COVERAGE_THRESHOLD ? "mask" : "modal";
|
||||||
}
|
}
|
||||||
|
|
||||||
export function detectBlockingLayer(nodes: VomNode[], vp: Viewport): BlockingLayer | null {
|
export function detectBlockingLayer(
|
||||||
|
nodes: VomNode[],
|
||||||
|
vp: Viewport,
|
||||||
|
rootFrameId?: string,
|
||||||
|
): BlockingLayer | null {
|
||||||
let blocker: { node: VomNode; coverage: number } | null = null;
|
let blocker: { node: VomNode; coverage: number } | null = null;
|
||||||
|
|
||||||
for (const node of nodes) {
|
for (const node of nodes) {
|
||||||
|
// A document's paint order is local to its iframe stacking context. Only
|
||||||
|
// the root document can establish a page-level blocking layer; child
|
||||||
|
// documents are constrained by their iframe owner in the parent document.
|
||||||
|
if (rootFrameId !== undefined && node.frameId !== rootFrameId) continue;
|
||||||
if (!isBlockingCandidate(node)) continue;
|
if (!isBlockingCandidate(node)) continue;
|
||||||
const cov = coverage(node.rect, vp);
|
const cov = coverage(node.rect, vp);
|
||||||
const qualifies = cov >= BLOCK_COVERAGE_THRESHOLD || (hasModalFeature(node) && cov >= 0.15);
|
const qualifies = cov >= BLOCK_COVERAGE_THRESHOLD || (hasModalFeature(node) && cov >= 0.15);
|
||||||
@@ -72,7 +80,9 @@ export function detectBlockingLayer(nodes: VomNode[], vp: Viewport): BlockingLay
|
|||||||
const threshold = blocker.node.paintOrder;
|
const threshold = blocker.node.paintOrder;
|
||||||
const members = new Set<number>();
|
const members = new Set<number>();
|
||||||
for (const node of nodes) {
|
for (const node of nodes) {
|
||||||
if (node.paintOrder >= threshold) members.add(node.id);
|
if (node.frameId === blocker.node.frameId && node.paintOrder >= threshold) {
|
||||||
|
members.add(node.id);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return {
|
return {
|
||||||
|
|||||||
+73
-11
@@ -6,6 +6,7 @@ import type {
|
|||||||
Rect,
|
Rect,
|
||||||
VomNode,
|
VomNode,
|
||||||
VomOptions,
|
VomOptions,
|
||||||
|
VomRef,
|
||||||
VomResult,
|
VomResult,
|
||||||
VomScene,
|
VomScene,
|
||||||
} from "./types";
|
} from "./types";
|
||||||
@@ -503,12 +504,18 @@ function isContextBoundary(node: VomNode): boolean {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function sharesContextScope(node: VomNode, candidate: VomNode): boolean {
|
||||||
|
return node.contextScopeId === candidate.contextScopeId;
|
||||||
|
}
|
||||||
|
|
||||||
function collectWeakLabelsFromSubtree(
|
function collectWeakLabelsFromSubtree(
|
||||||
node: VomNode,
|
node: VomNode,
|
||||||
|
target: VomNode,
|
||||||
nodeName: string,
|
nodeName: string,
|
||||||
state: RenderState,
|
state: RenderState,
|
||||||
labels: string[],
|
labels: string[],
|
||||||
): void {
|
): void {
|
||||||
|
if (!sharesContextScope(target, node)) return;
|
||||||
if (isVomReferenceNode(node)) return;
|
if (isVomReferenceNode(node)) return;
|
||||||
const label = weakLabelContext(node, nodeName);
|
const label = weakLabelContext(node, nodeName);
|
||||||
if (label) {
|
if (label) {
|
||||||
@@ -516,7 +523,7 @@ function collectWeakLabelsFromSubtree(
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
for (const child of state.children.get(node.id) ?? []) {
|
for (const child of state.children.get(node.id) ?? []) {
|
||||||
collectWeakLabelsFromSubtree(child, nodeName, state, labels);
|
collectWeakLabelsFromSubtree(child, target, nodeName, state, labels);
|
||||||
if (labels.length >= MAX_HANDLE_CONTEXT_ITEMS) return;
|
if (labels.length >= MAX_HANDLE_CONTEXT_ITEMS) return;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -532,6 +539,7 @@ function collectOwnedContext(
|
|||||||
while (current !== null && guard <= state.parentMap.size) {
|
while (current !== null && guard <= state.parentMap.size) {
|
||||||
const parent = state.nodesById.get(current);
|
const parent = state.nodesById.get(current);
|
||||||
if (!parent) break;
|
if (!parent) break;
|
||||||
|
if (!sharesContextScope(node, parent)) break;
|
||||||
const label = contextLabel(parent, nodeName);
|
const label = contextLabel(parent, nodeName);
|
||||||
if (label) contexts.push({ text: label, source: "owned", order: contexts.length });
|
if (label) contexts.push({ text: label, source: "owned", order: contexts.length });
|
||||||
current = state.parentMap.get(current) ?? null;
|
current = state.parentMap.get(current) ?? null;
|
||||||
@@ -553,12 +561,12 @@ function collectSameContainerContext(
|
|||||||
const siblings = state.children.get(parentId) ?? [];
|
const siblings = state.children.get(parentId) ?? [];
|
||||||
for (const sibling of siblings) {
|
for (const sibling of siblings) {
|
||||||
if (sibling.id === childId) break;
|
if (sibling.id === childId) break;
|
||||||
collectWeakLabelsFromSubtree(sibling, nodeName, state, labels);
|
collectWeakLabelsFromSubtree(sibling, node, nodeName, state, labels);
|
||||||
while (labels.length > MAX_HANDLE_CONTEXT_ITEMS) labels.shift();
|
while (labels.length > MAX_HANDLE_CONTEXT_ITEMS) labels.shift();
|
||||||
}
|
}
|
||||||
|
|
||||||
const parent = state.nodesById.get(parentId);
|
const parent = state.nodesById.get(parentId);
|
||||||
if (!parent || isContextBoundary(parent)) break;
|
if (!parent || !sharesContextScope(node, parent) || isContextBoundary(parent)) break;
|
||||||
childId = parentId;
|
childId = parentId;
|
||||||
parentId = state.parentMap.get(parentId) ?? null;
|
parentId = state.parentMap.get(parentId) ?? null;
|
||||||
guard += 1;
|
guard += 1;
|
||||||
@@ -578,6 +586,7 @@ function collectDomContext(
|
|||||||
|
|
||||||
for (const candidate of state.nodesById.values()) {
|
for (const candidate of state.nodesById.values()) {
|
||||||
if (candidate.id === node.id) continue;
|
if (candidate.id === node.id) continue;
|
||||||
|
if (!sharesContextScope(node, candidate)) continue;
|
||||||
if (isContextBoundary(candidate) && !ancestorIds.includes(candidate.id)) continue;
|
if (isContextBoundary(candidate) && !ancestorIds.includes(candidate.id)) continue;
|
||||||
const parentId = candidate.domParentId;
|
const parentId = candidate.domParentId;
|
||||||
if (parentId === undefined || parentId === null || !ancestors.has(parentId)) continue;
|
if (parentId === undefined || parentId === null || !ancestors.has(parentId)) continue;
|
||||||
@@ -627,8 +636,10 @@ function renderNodeLine(
|
|||||||
}
|
}
|
||||||
const placeholder = cleaned(node.placeholder);
|
const placeholder = cleaned(node.placeholder);
|
||||||
if (placeholder) line += ` placeholder=${JSON.stringify(placeholder)}`;
|
if (placeholder) line += ` placeholder=${JSON.stringify(placeholder)}`;
|
||||||
|
const sensitiveHasValue =
|
||||||
|
rawValue !== undefined || node.inputState === "filled" || node.inputState === "default";
|
||||||
const value = node.sensitive
|
const value = node.sensitive
|
||||||
? rawValue !== undefined
|
? sensitiveHasValue
|
||||||
? SENSITIVE_MASK
|
? SENSITIVE_MASK
|
||||||
: undefined
|
: undefined
|
||||||
: rawValue?.slice(0, MAX_VALUE_LENGTH);
|
: rawValue?.slice(0, MAX_VALUE_LENGTH);
|
||||||
@@ -684,7 +695,7 @@ function shouldSkipRedundantChildren(node: VomNode, state: RenderState): boolean
|
|||||||
|
|
||||||
interface RenderState {
|
interface RenderState {
|
||||||
lines: string[];
|
lines: string[];
|
||||||
refs: Array<{ ref: string; backendNodeId: number }>;
|
refs: VomRef[];
|
||||||
nextRef: number;
|
nextRef: number;
|
||||||
tokens: number;
|
tokens: number;
|
||||||
maxDepth: number;
|
maxDepth: number;
|
||||||
@@ -764,7 +775,11 @@ function renderTree(
|
|||||||
state.lines.push(line);
|
state.lines.push(line);
|
||||||
state.tokens = nextTokens;
|
state.tokens = nextTokens;
|
||||||
if (ref) {
|
if (ref) {
|
||||||
state.refs.push({ ref, backendNodeId: node.id });
|
state.refs.push({
|
||||||
|
ref,
|
||||||
|
backendNodeId: node.backendNodeId ?? node.id,
|
||||||
|
...(node.frameId ? { frameId: node.frameId } : {}),
|
||||||
|
});
|
||||||
state.nextRef += 1;
|
state.nextRef += 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -855,29 +870,75 @@ function activeRegionCandidatePriority(node: VomNode, viewportCoverage: number):
|
|||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function paintLineageByFrame(
|
||||||
|
node: VomNode,
|
||||||
|
parentMap: Map<number, number | null>,
|
||||||
|
nodesById: Map<number, VomNode>,
|
||||||
|
): Map<string | undefined, VomNode> {
|
||||||
|
const lineage = new Map<string | undefined, VomNode>();
|
||||||
|
let current: VomNode | undefined = node;
|
||||||
|
let guard = 0;
|
||||||
|
while (current && guard <= parentMap.size) {
|
||||||
|
if (!lineage.has(current.frameId)) lineage.set(current.frameId, current);
|
||||||
|
const parentId: number | null = parentMap.get(current.id) ?? null;
|
||||||
|
current = parentId === null ? undefined : nodesById.get(parentId);
|
||||||
|
guard += 1;
|
||||||
|
}
|
||||||
|
return lineage;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Resolve both nodes to the nearest document where their paint order is comparable. */
|
||||||
|
function comparablePaintNodes(
|
||||||
|
target: VomNode,
|
||||||
|
blocker: VomNode,
|
||||||
|
parentMap: Map<number, number | null>,
|
||||||
|
nodesById: Map<number, VomNode>,
|
||||||
|
): { target: VomNode; blocker: VomNode } | null {
|
||||||
|
if (target.frameId === blocker.frameId) return { target, blocker };
|
||||||
|
const targetLineage = paintLineageByFrame(target, parentMap, nodesById);
|
||||||
|
let blockerInFrame: VomNode | undefined = blocker;
|
||||||
|
let guard = 0;
|
||||||
|
while (blockerInFrame && guard <= parentMap.size) {
|
||||||
|
const targetInFrame = targetLineage.get(blockerInFrame.frameId);
|
||||||
|
if (targetInFrame) return { target: targetInFrame, blocker: blockerInFrame };
|
||||||
|
const parentId: number | null = parentMap.get(blockerInFrame.id) ?? null;
|
||||||
|
blockerInFrame = parentId === null ? undefined : nodesById.get(parentId);
|
||||||
|
guard += 1;
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
function isBlockedByRegion(
|
function isBlockedByRegion(
|
||||||
target: VomNode,
|
target: VomNode,
|
||||||
blocker: VomNode,
|
blocker: VomNode,
|
||||||
parentMap: Map<number, number | null>,
|
parentMap: Map<number, number | null>,
|
||||||
|
nodesById: Map<number, VomNode>,
|
||||||
viewportCoverage: number,
|
viewportCoverage: number,
|
||||||
): boolean {
|
): boolean {
|
||||||
if (target.id === blocker.id) return false;
|
if (target.id === blocker.id) return false;
|
||||||
if (isAncestorOf(parentMap, blocker.id, target.id)) return false;
|
if (isAncestorOf(parentMap, blocker.id, target.id)) return false;
|
||||||
if (isAncestorOf(parentMap, target.id, blocker.id)) return false;
|
if (isAncestorOf(parentMap, target.id, blocker.id)) return false;
|
||||||
if (!target.rect || !blocker.rect) return false;
|
if (!target.rect || !blocker.rect) return false;
|
||||||
|
const paintNodes = comparablePaintNodes(target, blocker, parentMap, nodesById);
|
||||||
|
if (!paintNodes) return false;
|
||||||
|
|
||||||
const points = interactionPoints(target.rect);
|
const points = interactionPoints(target.rect);
|
||||||
|
|
||||||
if (
|
if (
|
||||||
viewportCoverage < 0.9 &&
|
viewportCoverage < 0.9 &&
|
||||||
target.paintOrder >= blocker.paintOrder &&
|
paintNodes.target.paintOrder >= paintNodes.blocker.paintOrder &&
|
||||||
points.some(([x, y]) => rectContains(blocker.rect as Rect, x, y))
|
points.some(([x, y]) => rectContains(blocker.rect as Rect, x, y))
|
||||||
) {
|
) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (blocker.paintOrder < target.paintOrder) return false;
|
if (paintNodes.blocker.paintOrder < paintNodes.target.paintOrder) return false;
|
||||||
if (blocker.paintOrder === target.paintOrder && blocker.id <= target.id) return false;
|
if (
|
||||||
|
paintNodes.blocker.paintOrder === paintNodes.target.paintOrder &&
|
||||||
|
paintNodes.blocker.id <= paintNodes.target.id
|
||||||
|
) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
if (viewportCoverage >= 0.9) return true;
|
if (viewportCoverage >= 0.9) return true;
|
||||||
|
|
||||||
return points.some(([x, y]) => rectContains(blocker.rect as Rect, x, y));
|
return points.some(([x, y]) => rectContains(blocker.rect as Rect, x, y));
|
||||||
@@ -901,6 +962,7 @@ function collectDescendantsOfIds(nodes: VomNode[], roots: Set<number>): Set<numb
|
|||||||
|
|
||||||
function applyActiveRegionPolicy(nodes: VomNode[], scene: VomScene): VomNode[] {
|
function applyActiveRegionPolicy(nodes: VomNode[], scene: VomScene): VomNode[] {
|
||||||
const parentMap = buildParentMap(nodes);
|
const parentMap = buildParentMap(nodes);
|
||||||
|
const nodesById = new Map(nodes.map((node) => [node.id, node]));
|
||||||
const candidates = nodes
|
const candidates = nodes
|
||||||
.filter(isPositionedRegionCandidate)
|
.filter(isPositionedRegionCandidate)
|
||||||
.map((node) => {
|
.map((node) => {
|
||||||
@@ -917,7 +979,7 @@ function applyActiveRegionPolicy(nodes: VomNode[], scene: VomScene): VomNode[] {
|
|||||||
for (const target of nodes) {
|
for (const target of nodes) {
|
||||||
if (!isVomReferenceNode(target)) continue;
|
if (!isVomReferenceNode(target)) continue;
|
||||||
const blocked = candidates.some((candidate) =>
|
const blocked = candidates.some((candidate) =>
|
||||||
isBlockedByRegion(target, candidate.node, parentMap, candidate.viewportCoverage),
|
isBlockedByRegion(target, candidate.node, parentMap, nodesById, candidate.viewportCoverage),
|
||||||
);
|
);
|
||||||
if (blocked) blockedRoots.add(target.id);
|
if (blocked) blockedRoots.add(target.id);
|
||||||
}
|
}
|
||||||
@@ -972,7 +1034,7 @@ function renderDoubleLayer(scene: VomScene, layer: BlockingLayer, options: VomOp
|
|||||||
export function renderVom(scene: VomScene, options: VomOptions = {}): VomResult {
|
export function renderVom(scene: VomScene, options: VomOptions = {}): VomResult {
|
||||||
const nodes = applyVomInteractionRecovery(scene.nodes);
|
const nodes = applyVomInteractionRecovery(scene.nodes);
|
||||||
const renderScene = { ...scene, nodes };
|
const renderScene = { ...scene, nodes };
|
||||||
const layer = detectBlockingLayer(nodes, scene.viewport);
|
const layer = detectBlockingLayer(nodes, scene.viewport, scene.rootFrameId);
|
||||||
if (layer) return renderDoubleLayer(renderScene, layer, options);
|
if (layer) return renderDoubleLayer(renderScene, layer, options);
|
||||||
const visibleNodes = options.activeRegionPolicy ? applyActiveRegionPolicy(nodes, scene) : nodes;
|
const visibleNodes = options.activeRegionPolicy ? applyActiveRegionPolicy(nodes, scene) : nodes;
|
||||||
|
|
||||||
|
|||||||
@@ -21,6 +21,9 @@ export type LayerKind = "page" | "modal" | "mask";
|
|||||||
export interface VomNode {
|
export interface VomNode {
|
||||||
id: number;
|
id: number;
|
||||||
parentId: number | null;
|
parentId: number | null;
|
||||||
|
backendNodeId?: number;
|
||||||
|
frameId?: string;
|
||||||
|
contextScopeId?: string;
|
||||||
|
|
||||||
role?: string;
|
role?: string;
|
||||||
name?: string;
|
name?: string;
|
||||||
@@ -52,6 +55,8 @@ export interface VomNode {
|
|||||||
export interface VomScene {
|
export interface VomScene {
|
||||||
viewport: Viewport;
|
viewport: Viewport;
|
||||||
nodes: VomNode[];
|
nodes: VomNode[];
|
||||||
|
/** Root document whose paint order defines page-level blocking layers. */
|
||||||
|
rootFrameId?: string;
|
||||||
surfaces?: CondSurface[];
|
surfaces?: CondSurface[];
|
||||||
activeScopeBlocks?: ActiveScopeBlock[];
|
activeScopeBlocks?: ActiveScopeBlock[];
|
||||||
}
|
}
|
||||||
@@ -79,9 +84,15 @@ export interface ActiveScopeBlock {
|
|||||||
lines: string[];
|
lines: string[];
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface VomRef {
|
||||||
|
ref: string;
|
||||||
|
backendNodeId: number;
|
||||||
|
frameId?: string;
|
||||||
|
}
|
||||||
|
|
||||||
export interface VomResult {
|
export interface VomResult {
|
||||||
text: string;
|
text: string;
|
||||||
refs: Array<{ ref: string; backendNodeId: number }>;
|
refs: VomRef[];
|
||||||
truncated: boolean;
|
truncated: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user