Compare commits

..
Author SHA1 Message Date
Ljy-0827 6300eda0aa fix(record): rebase and dto policy 2026-08-23 21:26:40 +08:00
Ljy-0827 87cdcfe5ca fix(recording): using unified geometry 2026-08-23 21:26:40 +08:00
“haonan”andCursor 21476c3884 feat(record): capture Trace v3 states after pages settle
Wait for DOM quiet after actions, match targets to VOM refs, and reduce
drafts into deduped state-linked Trace v3 while keeping the v2 reducer.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-23 21:26:40 +08:00
“haonan” c2a225377b fix(protocol): align trace schemas with v2/v3 classify rules
RecordedTrace validation now follows classify_value — numeric version
selects v3, otherwise v2 — so mixed pages/states and v2 envelopes with
a version field fail Schema checks before Recall import. Keep standalone
trace_step as a v2|v3 union for tools that validate a single step.
2026-08-23 21:26:39 +08:00
“haonan” e64f4cac9f fix(protocol): keep generic trace schema compatible with v2 and v3
Split standalone v2/v3 schemas and drop TraceV2 deny_unknown_fields so legacy traces with extra keys still parse; mix and version checks stay in RecordedTrace.
2026-08-23 21:26:39 +08:00
Ljy-0827 368507b07d fix(protocol): compatible with dto 2026-08-23 21:26:39 +08:00
Ljy-0827 b750eed5d4 fix(protocol): integrate with vom 2026-08-23 21:26:39 +08:00
“haonan” 3d159c0254 test(protocol): align handshake fixtures with 1.0 floor and clarify v2/v3 naming 2026-08-23 21:26:39 +08:00
“haonan”andCursor 524516cb56 fix(protocol): align Trace v2/v3 classification and hover steps
Require Trace v3 version const 3, reject mixed pages/states payloads, and
align TypeScript selection optionality with the generated schema. Trace v2
on main already records hover steps, so the frozen v2 schema must accept
them instead of dropping them during reduction.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-23 21:26:39 +08:00
“haonan”andCursor f98198949a feat(protocol): negotiate Trace v3 with v2 compatibility
Define Trace v3 wire types and generated schemas while keeping protocol
1.1 peers compatible with 1.0. Default record start still produces Trace
v2, and only an explicit trace_version=3 request enables the new shape.
Refs #90

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-23 21:26:39 +08:00
Zhang GH d05279f991 Merge pull request #98 from NianJiuZst/codex/fix-daemon-idle-close-race
fix(daemon): make idle disconnect updates atomic
2026-08-23 21:25:24 +08:00
Zhang GH 2719e76438 Merge pull request #97 from NianJiuZst/codex/cancel-session-lifecycle-rpcs
fix(session): cancel lifecycle RPCs cooperatively
2026-08-22 14:51:38 +08:00
polarday dd9fc9edba Merge pull request #100 from Tencent/feat/record-vom-observation
feat(vom): expose record-safe semantic observations
2026-08-21 16:21:41 +08:00
Ljy-0827 005a6fd534 fix(vom): add record safe observations, unify definition and construction of safe DTO 2026-08-21 11:26:52 +08:00
“haonan” 20554508db fix(vom): omit raw AX/DOM trees from record-safe observations
Return only frame identity and node geometry for matching so a serialized capture result cannot leak form values.
2026-08-20 20:45:19 +08:00
NianJiuZst 8352c3c67a fix(daemon): refresh stale idle-task comment
After folding the IPC connection count and activity timestamp into a
single mutex-protected IpcActivityState, the idle task no longer
references the connection count directly; should_exit_for_idle already
returns false while a connection is open or within the idle window.
Update the comment to describe the post-refactor control flow so it
matches the code.
2026-08-20 13:53:23 +08:00
NianJiuZst 4eedd98c1d fix(daemon): make idle disconnect updates atomic 2026-08-20 13:53:23 +08:00
“haonan” 6aa931b60a chore: retrigger opensource scan 2026-08-19 19:56:07 +08:00
Ljy-0827 3df2738280 fix(vom): fix CI 2026-08-19 15:05:55 +08:00
Ljy-0827 f86e506d47 feat(vom): update contract 2026-08-19 15:05:55 +08:00
“haonan” 45a246c41b feat(vom): expose record-safe semantic observations
Share the VOM capture pipeline with recording, enrich rendered refs, and support value redaction without regressing cooperative cancellation.
2026-08-19 15:05:55 +08:00
BB-fat 6b78211e79 test(dsh-plugin): drop redundant thumbnail read-back case
Coverage already lives in the attachment-store and ring tests; this
case only re-hit readThumbnail and was flaky on CI.
2026-08-19 03:57:08 +00:00
BB-fat eebf334b40 fix(dsh-plugin): give observation waitFor more room on CI
The thumbnail read-back test timed out at the 2s waitFor budget under
CI load even though capture completed shortly after.
2026-08-19 03:50:14 +00:00
BB-fat a284ffc5ae Merge pull request #112 from Tencent/fix/dsh-plugin-id-and-edge-store
Fix/dsh plugin id and edge store
2026-08-19 11:43:13 +08:00
drakezhangandCursor e720f1ffe5 ci(dsh-plugin): publish from the npm-publish environment
The publish job named its GitHub Environment NPM_TOKEN, which is the
secret name rather than a deployment target, so the repository's
Deployments list showed a bogus NPM_TOKEN entry next to chrome-web-store.
The environment is now npm-publish; the secret it holds keeps its name.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-19 11:20:26 +08:00
drakezhangandCursor 2b7e10ca2b fix(dsh-plugin-browserskill): identify the plugin by its published name
Renaming the package to @wxg-prc-cpg/browser-skill-dsh-plugin left two
identifiers behind on the old directory name, and both are resolved by
name rather than by path.

cordis.patch.yml is the specifier the loader imports, so dsh failed with
ERR_MODULE_NOT_FOUND on startup and took the whole plugin tree down with
it — installing the published package made dsh unusable. The client
bundle registers itself under the same kind of key, which the web shell
looks up by package name, so the screenshot toolview and the observation
overlay were unreachable too.

The bundle id now reads package.json instead of repeating the name, so it
cannot drift again. Quoting is required in the YAML because a leading '@'
is reserved in plain scalars.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-19 11:20:07 +08:00
drakezhangandCursor 1ef93463dd docs: list the Edge Add-ons build alongside the Chrome Web Store
The extension is now published to Edge Add-ons, but every install path
still pointed Edge users at the Chrome Web Store. Both READMEs and
AGENT_INSTALL.md now name a store per browser, and `bsk doctor` offers
both listings when no browser is connected.

Two Edge-specific gaps went with it: Edge blocks content-script injection
on its own storefront the same way Chrome does, so the borrow-confirmation
overlay must treat microsoftedge.microsoft.com/addons as non-injectable;
and the Chinese README still described the pre-install.ps1 manual Windows
setup.

Also give the DeepSeek Harness section its npm package link and the
profile-start step, so plugin users have a complete path.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-19 11:19:47 +08:00
Hannah Liu c7252529f4 Merge pull request #109 from Tencent/fix/oopif-support
fix(extension): support OOPIF and cross-frame geometry
2026-08-18 21:16:39 +08:00
Ljy-0827 0b65b4b295 fix(vom): oopif fix 2026-08-18 19:28:16 +08:00
Ljy-0827 94ec7e73fc fix(vom): iframe oopif geometry 2026-08-18 19:28:16 +08:00
Ljy-0827 42403e55f2 feat(extension): support OOPIF refs with unified frame geometry 2026-08-18 19:28:16 +08:00
Hannah Liu 22019cdf07 Merge pull request #105 from Tencent/fix/vom-iframe
fix(vom): vom support observe iframe
2026-08-18 19:28:13 +08:00
NianJiuZst 7035e46024 fix(session): cancel lifecycle RPCs cooperatively 2026-08-17 15:13:53 +08:00
104 changed files with 11581 additions and 1749 deletions

No files matched your search

+2 -2
View File
@@ -66,8 +66,8 @@ jobs:
publish:
needs: [resolve, guard]
runs-on: ubuntu-latest
# Secret NPM_TOKEN is stored on the GitHub Environment also named NPM_TOKEN.
environment: NPM_TOKEN
# Secret NPM_TOKEN is stored on the "npm-publish" GitHub Environment.
environment: npm-publish
permissions:
contents: read
steps:
+9 -5
View File
@@ -34,13 +34,17 @@ where only `extension connected` fails is expected; go to Step 3.
## 3. Open the extension install page
If `extension connected` is `FAIL` (`0 browsers connected`), the user likely
has not installed or enabled the browser extension yet. Open the
[Chrome Web Store install page](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi)
in the user's browser, then tell the user:
has not installed or enabled the browser extension yet. Open the install page
matching the user's browser —
[Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi)
for Chrome and other Chromium browsers,
[Edge Add-ons](https://microsoftedge.microsoft.com/addons/detail/browserskill/emacgiaaaiojkkpkddmmdfhmokgmnikg)
for Microsoft Edge — then tell the user:
> Install **BrowserSkill** from the
> [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi),
> open the popup, and wait until it turns green. Reply when done.
> [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi)
> (or [Edge Add-ons](https://microsoftedge.microsoft.com/addons/detail/browserskill/emacgiaaaiojkkpkddmmdfhmokgmnikg)
> on Edge), open the popup, and wait until it turns green. Reply when done.
If opening the page fails, give the user the same link. Then run `bsk doctor`
once more. All `ok`/`na` → tell the user it's ready.
+22 -4
View File
@@ -67,7 +67,8 @@ Set up browser-skill on this machine by following https://raw.githubusercontent.
<br>
Install the CLI, then install the extension from the [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi).
Install the CLI, then install the extension from the [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi)
or [Edge Add-ons](https://microsoftedge.microsoft.com/addons/detail/browserskill/emacgiaaaiojkkpkddmmdfhmokgmnikg).
#### 1. Install the `bsk` CLI
@@ -91,7 +92,14 @@ bsk --version
#### 2. Install the browser extension
Install BrowserSkill from the [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi).
Install BrowserSkill from your browser's store:
| Browser | Store listing |
| --- | --- |
| Chrome | [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi) |
| Microsoft Edge | [Edge Add-ons](https://microsoftedge.microsoft.com/addons/detail/browserskill/emacgiaaaiojkkpkddmmdfhmokgmnikg) |
On other Chromium-based browsers, install the Chrome Web Store build.
#### 3. Install the skill
@@ -136,14 +144,24 @@ Start a new Agent session and write a prompt that needs the browser, for example
## DeepSeek Harness plugin
A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) plugin that
injects native `browser_*` tools (no shelling out to `bsk`) and a live Web UI
Using [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (`dsh`)?
BrowserSkill ships a first-class dsh plugin on npm as
[`@wxg-prc-cpg/browser-skill-dsh-plugin`](https://www.npmjs.com/package/@wxg-prc-cpg/browser-skill-dsh-plugin).
It injects native `browser_*` tools (no shelling out to `bsk`) and a live Web UI
overlay of each Agent Window.
Add it to a dsh profile, then start that profile:
```sh
dsh plugin --profile web add @wxg-prc-cpg/browser-skill-dsh-plugin
dsh --profile web
```
The plugin carries its own copy of the skill, so `bsk install-skill` is not needed
for dsh — but the `bsk` CLI and the browser extension are still prerequisites. See
the [plugin README](packages/dsh-plugin-browserskill/README.md) for the tool list,
configuration, and the observation overlay.
## How It Works
BrowserSkill is a local bridge between your agent harness and your browser.
+21 -5
View File
@@ -54,7 +54,8 @@ BrowserSkill 由两个本地运行组件组成:`bsk` CLI/daemon 和浏览器
<br>
先安装 CLI,再从 [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi) 安装浏览器扩展。
先安装 CLI,再从 [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi)
或 [Edge 加载项商店](https://microsoftedge.microsoft.com/addons/detail/browserskill/emacgiaaaiojkkpkddmmdfhmokgmnikg) 安装浏览器扩展。
#### 1. 安装 `bsk` CLI
@@ -64,8 +65,11 @@ BrowserSkill 由两个本地运行组件组成:`bsk` CLI/daemon 和浏览器
curl -fsSL https://raw.githubusercontent.com/Tencent/BrowserSkill/main/install.sh | sh
```
**Windows**:从 [最新 CLI release](https://github.com/Tencent/BrowserSkill/releases/latest)
下载 `bsk-v<version>-x86_64-pc-windows-msvc.zip`,解压后将 `bsk.exe` 加入 `PATH`。
**Windows**(PowerShell,安装到 `~/.local/bin`):
```powershell
irm https://raw.githubusercontent.com/Tencent/BrowserSkill/main/install.ps1 | iex
```
验证二进制:
@@ -75,7 +79,14 @@ bsk --version
#### 2. 安装浏览器扩展
从 [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi) 安装 BrowserSkill。
在对应浏览器的商店安装 BrowserSkill:
| 浏览器 | 商店页面 |
| --- | --- |
| Chrome | [Chrome Web Store](https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi) |
| Microsoft Edge | [Edge 加载项商店](https://microsoftedge.microsoft.com/addons/detail/browserskill/emacgiaaaiojkkpkddmmdfhmokgmnikg) |
其他基于 Chromium 的浏览器,安装 Chrome Web Store 版本即可。
#### 3. 安装 skill
@@ -114,12 +125,17 @@ bsk install-skill
## DeepSeek Harness 插件
[DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) 插件:注入原生 `browser_*` 工具(无需再通过 Shell 调用 `bsk`),并在 Web UI 中实时观察每个 Agent Window。
在用 [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness)(`dsh`)?BrowserSkill 提供了官方 dsh 插件,已发布到 npm:[`@wxg-prc-cpg/browser-skill-dsh-plugin`](https://www.npmjs.com/package/@wxg-prc-cpg/browser-skill-dsh-plugin)。它会注入原生 `browser_*` 工具(无需再通过 Shell 调用 `bsk`),并在 Web UI 中实时观察每个 Agent Window。
把它装进某个 dsh profile,然后启动该 profile:
```sh
dsh plugin --profile web add @wxg-prc-cpg/browser-skill-dsh-plugin
dsh --profile web
```
插件自带 skill,所以在 dsh 下无需执行 `bsk install-skill`;但 `bsk` CLI 和浏览器扩展仍是前置条件。工具清单、配置项与观察浮层见[插件 README](packages/dsh-plugin-browserskill/README.md)。
## 工作原理
BrowserSkill 是 Agent 运行时与浏览器之间的本地桥接层。
@@ -1,5 +1,5 @@
import { describe, expect, it, vi } from "vitest";
import { type CdpDebuggerApi, ChromiumCdp } from "../chromium-cdp";
import { type CdpDebuggee, type CdpDebuggerApi, ChromiumCdp } from "../chromium-cdp";
function fakeChromeEvent<TArgs extends unknown[]>() {
const listeners = new Set<(...args: TArgs) => void>();
@@ -14,7 +14,7 @@ function fakeChromeEvent<TArgs extends unknown[]>() {
}
function fakeApi() {
const onEvent = fakeChromeEvent<[chrome.debugger.Debuggee, string, unknown]>();
const onEvent = fakeChromeEvent<[CdpDebuggee, string, unknown]>();
const onDetach = fakeChromeEvent<[chrome.debugger.Debuggee, string]>();
const api: CdpDebuggerApi = {
attach: vi.fn(async () => {}),
@@ -29,6 +29,135 @@ function fakeApi() {
}
describe("ChromiumCdp", () => {
it("discovers multiple iframe targets and recursively routes nested OOPIF commands", async () => {
const { api, onEvent } = fakeApi();
(api.sendCommand as ReturnType<typeof vi.fn>).mockImplementation(
async (
target: chrome.debugger.Debuggee & { sessionId?: string },
method: string,
params?: { frameId?: string },
) => {
if (method === "DOM.getFrameOwner") {
return { backendNodeId: params?.frameId === "nested" ? 300 : 200 };
}
if (method !== "Page.getFrameTree") return {};
if (target.sessionId === "right-session") {
return {
frameTree: {
frame: { id: "right", url: "https://right.test" },
childFrames: [
{ frame: { id: "nested", parentId: "right", url: "https://nested.test" } },
],
},
};
}
if (target.sessionId === "nested-session") {
return { frameTree: { frame: { id: "nested", url: "https://nested.test" } } };
}
return {
frameTree: {
frame: { id: "main", url: "https://app.test" },
childFrames: [
{ frame: { id: "left", parentId: "main", url: "https://left.test" } },
{
frame: { id: "right", parentId: "main", url: "https://right.test" },
childFrames: [
{ frame: { id: "nested", parentId: "right", url: "https://nested.test" } },
],
},
],
},
};
},
);
const cdp = new ChromiumCdp(api);
await cdp.ensureAttached(4);
onEvent.fire({ tabId: 4 }, "Target.attachedToTarget", { sessionId: "right-session" });
onEvent.fire({ tabId: 4 } as chrome.debugger.Debuggee, "Target.attachedToTarget", {
sessionId: "nested-session",
});
const graph = await cdp.getFrameGraph(4);
expect(graph.frames).toHaveLength(4);
expect(graph.frames.find((frame) => frame.frameId === "left")?.target).toEqual({ tabId: 4 });
expect(graph.frames.find((frame) => frame.frameId === "right")?.target.sessionId).toBe(
"right-session",
);
expect(graph.frames.find((frame) => frame.frameId === "nested")?.target.sessionId).toBe(
"nested-session",
);
expect(graph.frames.find((frame) => frame.frameId === "right")?.ownerBackendNodeId).toBe(200);
expect(graph.frames.find((frame) => frame.frameId === "nested")?.ownerBackendNodeId).toBe(300);
expect(api.sendCommand).toHaveBeenCalledWith(
{ tabId: 4, sessionId: "right-session" },
"DOM.getFrameOwner",
{ frameId: "nested" },
);
expect(api.sendCommand).toHaveBeenCalledWith(
{ tabId: 4, sessionId: "nested-session" },
"Target.setAutoAttach",
expect.objectContaining({ flatten: true }),
);
});
it("waits for recursively attached iframe targets without a depth limit", async () => {
const { api, onEvent } = fakeApi();
const depth = 6;
const scheduled = new Set<string>();
const nestedTree = (index: number): Record<string, unknown> => ({
frame: {
id: index === 0 ? "main" : `frame-${index}`,
...(index > 0 ? { parentId: index === 1 ? "main" : `frame-${index - 1}` } : {}),
},
...(index < depth ? { childFrames: [nestedTree(index + 1)] } : {}),
});
(api.sendCommand as ReturnType<typeof vi.fn>).mockImplementation(
async (target: CdpDebuggee, method: string, params?: { frameId?: string }) => {
if (method === "Target.setAutoAttach") {
const parentIndex = target.sessionId
? Number(target.sessionId.replace("session-", ""))
: 0;
const nextIndex = parentIndex + 1;
const key = `${target.sessionId ?? "root"}:${nextIndex}`;
if (nextIndex <= depth && !scheduled.has(key)) {
scheduled.add(key);
setTimeout(() => {
onEvent.fire(
{ tabId: 4, ...(target.sessionId ? { sessionId: target.sessionId } : {}) },
"Target.attachedToTarget",
{
sessionId: `session-${nextIndex}`,
targetInfo: { type: "iframe" },
},
);
}, 5);
}
return {};
}
if (method === "Page.getFrameTree") {
if (!target.sessionId) return { frameTree: nestedTree(0) };
const index = Number(target.sessionId.replace("session-", ""));
return { frameTree: nestedTree(index) };
}
if (method === "DOM.getFrameOwner") {
return { backendNodeId: Number(params?.frameId?.replace("frame-", "")) + 100 };
}
return {};
},
);
const cdp = new ChromiumCdp(api);
const graph = await cdp.getFrameGraph(4);
expect(graph.frames).toHaveLength(depth + 1);
for (let index = 1; index <= depth; index += 1) {
expect(
graph.frames.find((frame) => frame.frameId === `frame-${index}`)?.target.sessionId,
).toBe(`session-${index}`);
}
});
it("coalesces concurrent attach calls for the same tab", async () => {
const { api } = fakeApi();
let releaseAttach!: () => void;
@@ -0,0 +1,136 @@
import { describe, expect, it } from "vitest";
import { buildFrameGraph, type CdpFrameTreeNode, type CdpFrameTreeSource } from "../frame-graph";
describe("buildFrameGraph", () => {
it("keeps sibling frames distinct and routes nested OOPIFs to their child sessions", () => {
const graph = buildFrameGraph([
{
target: { tabId: 4 },
tree: {
frame: { id: "main", url: "https://app.test" },
childFrames: [
{ frame: { id: "left", parentId: "main", url: "https://left.test" } },
{
frame: { id: "right", parentId: "main", url: "https://right.test" },
childFrames: [
{ frame: { id: "nested", parentId: "right", url: "https://nested.test" } },
],
},
],
},
},
{
target: { tabId: 4, sessionId: "right-session" },
tree: {
frame: { id: "right", url: "https://right.test" },
childFrames: [{ frame: { id: "nested", parentId: "right", url: "https://nested.test" } }],
},
},
{
target: { tabId: 4, sessionId: "nested-session" },
tree: { frame: { id: "nested", url: "https://nested.test" } },
},
]);
expect(graph?.frames).toHaveLength(4);
expect(graph?.frames.find((frame) => frame.frameId === "left")?.target).toEqual({ tabId: 4 });
expect(graph?.frames.find((frame) => frame.frameId === "right")).toMatchObject({
parentFrameId: "main",
target: { tabId: 4, sessionId: "right-session" },
});
expect(graph?.frames.find((frame) => frame.frameId === "nested")).toMatchObject({
parentFrameId: "right",
target: { tabId: 4, sessionId: "nested-session" },
});
});
it("propagates each target boundary to its same-process descendants", () => {
const sources: CdpFrameTreeSource[] = [
{
target: { tabId: 4 },
tree: {
frame: { id: "main" },
childFrames: [
{
frame: { id: "oopif-a", parentId: "main" },
childFrames: [
{
frame: { id: "same-process-b", parentId: "oopif-a" },
childFrames: [
{
frame: { id: "oopif-c", parentId: "same-process-b" },
childFrames: [{ frame: { id: "same-process-d", parentId: "oopif-c" } }],
},
],
},
],
},
],
},
},
{
target: { tabId: 4, sessionId: "session-a" },
tree: {
frame: { id: "oopif-a" },
childFrames: [
{
frame: { id: "same-process-b", parentId: "oopif-a" },
childFrames: [{ frame: { id: "oopif-c", parentId: "same-process-b" } }],
},
],
},
},
{
target: { tabId: 4, sessionId: "session-c" },
tree: {
frame: { id: "oopif-c" },
childFrames: [{ frame: { id: "same-process-d", parentId: "oopif-c" } }],
},
},
];
const graph = buildFrameGraph(sources);
expect(graph?.frames.map((frame) => [frame.frameId, frame.target.sessionId])).toEqual([
["main", undefined],
["oopif-a", "session-a"],
["same-process-b", "session-a"],
["oopif-c", "session-c"],
["same-process-d", "session-c"],
]);
expect(graph?.frames.find((frame) => frame.frameId === "oopif-a")?.parentFrameId).toBe("main");
expect(graph?.frames.find((frame) => frame.frameId === "oopif-c")?.parentFrameId).toBe(
"same-process-b",
);
const reordered = buildFrameGraph([sources[2], sources[1], sources[0]]);
expect(reordered).toEqual(graph);
});
it("walks deeply nested frame trees without consuming the JavaScript call stack", () => {
const depth = 5_000;
const root: CdpFrameTreeNode = { frame: { id: "frame-0" } };
let parent = root;
for (let index = 1; index <= depth; index += 1) {
const child: CdpFrameTreeNode = {
frame: { id: `frame-${index}`, parentId: `frame-${index - 1}` },
};
parent.childFrames = [child];
parent = child;
}
const graph = buildFrameGraph([{ target: { tabId: 4 }, tree: root }]);
expect(graph?.frames).toHaveLength(depth + 1);
expect(graph?.frames.at(-1)?.frameId).toBe(`frame-${depth}`);
});
it("does not loop when an in-memory frame tree contains an object cycle", () => {
const root: CdpFrameTreeNode = { frame: { id: "main" } };
root.childFrames = [root];
expect(buildFrameGraph([{ target: { tabId: 4 }, tree: root }])?.frames).toEqual([
{ frameId: "main", target: { tabId: 4 } },
]);
});
});
+211 -12
View File
@@ -31,6 +31,15 @@ import type {
NetworkEntryKind,
NetworkResult,
} from "@/transport/types";
import {
buildFrameGraph,
type CdpFrameGraph,
type CdpFrameTreeNode,
type CdpFrameTreeSource,
type CdpTarget,
} from "./frame-graph";
export type CdpDebuggee = chrome.debugger.Debuggee & { sessionId?: string };
/**
* Minimal slice of `chrome.debugger` the rest of the extension
@@ -38,21 +47,15 @@ import type {
* fake without monkey-patching the real `chrome` global.
*/
export interface CdpDebuggerApi {
attach(target: chrome.debugger.Debuggee, requiredVersion: string): Promise<void>;
detach(target: chrome.debugger.Debuggee): Promise<void>;
sendCommand(
target: chrome.debugger.Debuggee,
method: string,
commandParams?: object,
): Promise<unknown>;
attach(target: CdpDebuggee, requiredVersion: string): Promise<void>;
detach(target: CdpDebuggee): Promise<void>;
sendCommand(target: CdpDebuggee, method: string, commandParams?: object): Promise<unknown>;
/**
* Fires for every CDP event (`Page.lifecycleEvent`, `DOM.documentUpdated`,
* …). The first callback argument is the source debuggee; the second
* is the CDP method name; the third is the payload.
*/
onEvent: chrome.events.Event<
(source: chrome.debugger.Debuggee, method: string, params: unknown) => void
>;
onEvent: chrome.events.Event<(source: CdpDebuggee, method: string, params: unknown) => void>;
/**
* Fires when Chrome unilaterally detaches us — most commonly because
* the tab navigated to a chrome:// URL or the user clicked
@@ -110,6 +113,8 @@ const MAX_CONSOLE_STACK_FRAMES = 20;
const MAX_NETWORK_BUFFER = 200;
const MAX_NETWORK_FIELD_LENGTH = 4096;
const MAX_NETWORK_REQUEST_META = 1024;
const FRAME_DISCOVERY_TIMEOUT_MS = 1000;
const FRAME_DISCOVERY_QUIET_MS = 20;
interface ParsedDialogOpening {
type: JavaScriptDialogType;
@@ -131,6 +136,27 @@ interface NetworkRequestMeta {
truncated: boolean;
}
interface FrameDiscoveryState {
sessions: Set<string>;
pending: Set<Promise<void>>;
generation: number;
}
async function settleBeforeDeadline(promises: Promise<void>[], deadline: number): Promise<boolean> {
if (promises.length === 0) return true;
const remaining = deadline - Date.now();
if (remaining <= 0) return false;
let timer: ReturnType<typeof setTimeout> | undefined;
const timedOut = await Promise.race([
Promise.allSettled(promises).then(() => false),
new Promise<boolean>((resolve) => {
timer = setTimeout(() => resolve(true), remaining);
}),
]);
if (timer) clearTimeout(timer);
return !timedOut;
}
/**
* Wrapper around `chrome.debugger` that owns the "attach once per
* tabId" cache and exposes typed `send<T>()`.
@@ -149,10 +175,12 @@ export class ChromiumCdp {
private readonly networkSequences = new Map<number, number>();
private readonly networkDomainsEnabledTabs = new Set<number>();
private readonly networkRequestMeta = new Map<number, Map<string, NetworkRequestMeta>>();
private readonly frameDiscovery = new Map<number, FrameDiscoveryState>();
private detachSubscription: { dispose(): void } | null = null;
private dialogSubscription: { dispose(): void } | null = null;
private consoleSubscription: { dispose(): void } | null = null;
private networkSubscription: { dispose(): void } | null = null;
private frameTargetSubscription: { dispose(): void } | null = null;
constructor(api: CdpDebuggerApi = chromeDebuggerApi) {
this.api = api;
@@ -160,6 +188,7 @@ export class ChromiumCdp {
this.bindDialogHandler();
this.bindConsoleHandler();
this.bindNetworkHandler();
this.bindFrameTargetHandler();
}
/** Attach to `tabId` if we haven't already in this driver. */
@@ -177,6 +206,9 @@ export class ChromiumCdp {
await this.enableConsoleDomains(tabId);
await this.enableNetworkDomainBestEffort(tabId);
this.attachedTabs.add(tabId);
await this.enableFrameDiscovery({ tabId }).catch((err) => {
console.debug("[bsk cdp] frame discovery unavailable", { tabId, err });
});
} catch (err) {
// A CDP domain enable failed after the raw attach succeeded
// (e.g. `Page.enable` rejects because the tab just navigated to
@@ -222,6 +254,73 @@ export class ChromiumCdp {
}
}
async sendToTarget<T = unknown>(target: CdpTarget, method: string, params?: object): Promise<T> {
if (!this.attachedTabs.has(target.tabId)) {
await this.ensureAttached(target.tabId);
}
try {
return (await this.api.sendCommand(target, method, params ?? {})) as T;
} catch (err) {
throw normalizeError(err);
}
}
async getFrameGraph(tabId: number): Promise<CdpFrameGraph> {
await this.ensureAttached(tabId);
await this.enableFrameDiscovery({ tabId }).catch(() => {});
await this.drainFrameAttachTasks(tabId);
const sources: CdpFrameTreeSource[] = [];
const root = await this.sendToTarget<{ frameTree?: CdpFrameTreeNode }>(
{ tabId },
"Page.getFrameTree",
{},
);
if (root.frameTree) sources.push({ target: { tabId }, tree: root.frameTree });
const sessions = [...(this.frameDiscovery.get(tabId)?.sessions ?? [])];
const childTrees = await Promise.all(
sessions.map(async (sessionId): Promise<CdpFrameTreeSource | null> => {
const target = { tabId, sessionId };
try {
const reply = await this.sendToTarget<{ frameTree?: CdpFrameTreeNode }>(
target,
"Page.getFrameTree",
{},
);
return reply.frameTree ? { target, tree: reply.frameTree } : null;
} catch {
return null;
}
}),
);
for (const source of childTrees) {
if (source) sources.push(source);
}
const graph = buildFrameGraph(sources);
if (!graph) throw new Error("Page.getFrameTree returned no root frame");
const frameById = new Map(graph.frames.map((frame) => [frame.frameId, frame]));
await Promise.all(
graph.frames.map(async (frame) => {
if (!frame.parentFrameId) return;
const parent = frameById.get(frame.parentFrameId);
if (!parent) return;
try {
const owner = await this.sendToTarget<{ backendNodeId?: number }>(
parent.target,
"DOM.getFrameOwner",
{ frameId: frame.frameId },
);
if (owner.backendNodeId !== undefined) frame.ownerBackendNodeId = owner.backendNodeId;
} catch {
// The frame may have navigated between tree capture and owner lookup.
}
}),
);
return graph;
}
/** Return a cursor marking the current dialog sequence for `tabId`. */
dialogCursor(tabId: number): DialogCursor {
return this.dialogSequences.get(tabId) ?? 0;
@@ -332,6 +431,7 @@ export class ChromiumCdp {
this.clearDialogState(tabId);
this.clearConsoleState(tabId);
this.clearNetworkState(tabId);
this.clearFrameState(tabId);
try {
await this.api.detach({ tabId });
} catch (err) {
@@ -354,7 +454,7 @@ export class ChromiumCdp {
}
/** Subscribe to all CDP events. Returned disposable removes the listener. */
onEvent(handler: (source: chrome.debugger.Debuggee, method: string, params: unknown) => void): {
onEvent(handler: (source: CdpDebuggee, method: string, params: unknown) => void): {
dispose(): void;
} {
this.api.onEvent.addListener(handler);
@@ -378,6 +478,7 @@ export class ChromiumCdp {
this.networkSequences.clear();
this.networkDomainsEnabledTabs.clear();
this.networkRequestMeta.clear();
this.frameDiscovery.clear();
await Promise.all(
tabs.map(async (tabId) => {
try {
@@ -428,9 +529,104 @@ export class ChromiumCdp {
}
}
private async enableFrameDiscovery(target: CdpTarget): Promise<void> {
await this.api.sendCommand(target, "Target.setAutoAttach", {
autoAttach: true,
waitForDebuggerOnStart: false,
flatten: true,
filter: [{ type: "iframe", exclude: false }],
});
}
private bindFrameTargetHandler(): void {
if (this.frameTargetSubscription) return;
const listener = (source: CdpDebuggee, method: string, params: unknown) => {
const tabId = source.tabId;
if (typeof tabId !== "number") return;
const raw = (params ?? {}) as Record<string, unknown>;
const sessionId = typeof raw.sessionId === "string" ? raw.sessionId : undefined;
if (!sessionId) return;
if (method === "Target.detachedFromTarget") {
const state = this.frameDiscovery.get(tabId);
if (state?.sessions.delete(sessionId)) state.generation += 1;
return;
}
if (method !== "Target.attachedToTarget") return;
const targetInfo = raw.targetInfo as { type?: string } | undefined;
if (targetInfo?.type && targetInfo.type !== "iframe") return;
const state = this.frameDiscoveryState(tabId);
if (state.sessions.has(sessionId)) return;
state.sessions.add(sessionId);
state.generation += 1;
const task = this.initializeFrameTarget({ tabId, sessionId });
state.pending.add(task);
void task.finally(() => {
state.pending.delete(task);
});
};
this.api.onEvent.addListener(listener);
this.frameTargetSubscription = {
dispose: () => this.api.onEvent.removeListener(listener),
};
}
private async initializeFrameTarget(target: CdpTarget): Promise<void> {
try {
await this.enableFrameDiscovery(target);
} catch (err) {
const state = this.frameDiscovery.get(target.tabId);
if (state?.sessions.delete(target.sessionId as string)) state.generation += 1;
console.debug("[bsk cdp] child frame target initialization failed", { target, err });
}
}
private async drainFrameAttachTasks(tabId: number): Promise<void> {
const deadline = Date.now() + FRAME_DISCOVERY_TIMEOUT_MS;
while (Date.now() < deadline) {
const state = this.frameDiscovery.get(tabId);
const generation = state?.generation ?? 0;
if (state?.pending.size && !(await settleBeforeDeadline([...state.pending], deadline))) break;
// Target.setAutoAttach may enqueue the next attachedToTarget event after
// its command promise settles. Wait for a short quiet window, then finish
// only if neither the state object nor its generation changed.
const quietTime = Math.min(FRAME_DISCOVERY_QUIET_MS, deadline - Date.now());
if (quietTime <= 0) break;
await new Promise<void>((resolve) => setTimeout(resolve, quietTime));
const current = this.frameDiscovery.get(tabId);
if (
current === state &&
(current?.generation ?? 0) === generation &&
!current?.pending.size
) {
return;
}
}
console.debug("[bsk cdp] frame discovery did not reach quiescence before timeout", { tabId });
}
private frameDiscoveryState(tabId: number): FrameDiscoveryState {
const existing = this.frameDiscovery.get(tabId);
if (existing) return existing;
const created: FrameDiscoveryState = {
sessions: new Set(),
pending: new Set(),
generation: 0,
};
this.frameDiscovery.set(tabId, created);
return created;
}
private clearFrameState(tabId: number): void {
this.frameDiscovery.delete(tabId);
}
private bindDialogHandler(): void {
if (this.dialogSubscription) return;
const listener = (source: chrome.debugger.Debuggee, method: string, params: unknown) => {
const listener = (source: CdpDebuggee, method: string, params: unknown) => {
if (method !== "Page.javascriptDialogOpening") return;
const tabId = source.tabId;
if (typeof tabId !== "number") return;
@@ -614,6 +810,7 @@ export class ChromiumCdp {
this.clearDialogState(source.tabId);
this.clearConsoleState(source.tabId);
this.clearNetworkState(source.tabId);
this.clearFrameState(source.tabId);
}
};
this.api.onDetach.addListener(listener);
@@ -632,6 +829,8 @@ export class ChromiumCdp {
this.consoleSubscription = null;
this.networkSubscription?.dispose();
this.networkSubscription = null;
this.frameTargetSubscription?.dispose();
this.frameTargetSubscription = null;
}
/** Detach tabs only when no other live session has claimed them. */
@@ -0,0 +1,107 @@
export interface CdpTarget {
tabId: number;
sessionId?: string;
}
export interface CdpFrame {
frameId: string;
parentFrameId?: string;
ownerBackendNodeId?: number;
url?: string;
target: CdpTarget;
}
export interface CdpFrameGraph {
rootFrameId: string;
frames: CdpFrame[];
}
export interface CdpFrameTreeNode {
frame: {
id: string;
parentId?: string;
url?: string;
};
childFrames?: CdpFrameTreeNode[];
}
export interface CdpFrameTreeSource {
target: CdpTarget;
tree: CdpFrameTreeNode;
}
export function cdpTargetKey(target: CdpTarget): string {
return `${target.tabId}:${target.sessionId ?? "root"}`;
}
function mergeFrame(
frames: Map<string, CdpFrame>,
order: string[],
node: CdpFrameTreeNode,
target: CdpTarget,
): void {
const existing = frames.get(node.frame.id);
if (!existing) order.push(node.frame.id);
const parentFrameId = node.frame.parentId || existing?.parentFrameId;
const url = node.frame.url || existing?.url;
frames.set(node.frame.id, {
frameId: node.frame.id,
...(parentFrameId ? { parentFrameId } : {}),
...(url ? { url } : {}),
target,
});
}
function walkFrameTree(
source: CdpFrameTreeSource,
targetByRootFrameId: ReadonlyMap<string, CdpTarget>,
frames: Map<string, CdpFrame>,
order: string[],
): void {
const stack: Array<{ node: CdpFrameTreeNode; inheritedTarget: CdpTarget }> = [
{ node: source.tree, inheritedTarget: source.target },
];
const expanded = new WeakSet<CdpFrameTreeNode>();
while (stack.length > 0) {
const current = stack.pop();
if (!current || expanded.has(current.node)) continue;
expanded.add(current.node);
const target = targetByRootFrameId.get(current.node.frame.id) ?? current.inheritedTarget;
mergeFrame(frames, order, current.node, target);
const children = current.node.childFrames ?? [];
for (let index = children.length - 1; index >= 0; index -= 1) {
stack.push({ node: children[index], inheritedTarget: target });
}
}
}
export function buildFrameGraph(sources: CdpFrameTreeSource[]): CdpFrameGraph | null {
const rootSource = sources.find((source) => source.target.sessionId === undefined) ?? sources[0];
if (!rootSource) return null;
const frames = new Map<string, CdpFrame>();
const order: string[] = [];
const targetByRootFrameId = new Map(
sources.map((source) => [source.tree.frame.id, source.target] as const),
);
// The top-level source is authoritative if malformed input reports the same
// frame as both a root target and a child target.
targetByRootFrameId.set(rootSource.tree.frame.id, rootSource.target);
walkFrameTree(rootSource, targetByRootFrameId, frames, order);
for (const source of sources) {
if (source === rootSource) continue;
walkFrameTree(source, targetByRootFrameId, frames, order);
}
return {
rootFrameId: rootSource.tree.frame.id,
frames: order.flatMap((frameId) => {
const frame = frames.get(frameId);
return frame ? [frame] : [];
}),
};
}
@@ -8,6 +8,7 @@ import {
useRef,
useState,
} from "react";
import type { HelpHighlightRect } from "@/lib/help-bridge";
import logoUrl from "../../assets/logo.png";
export interface HelpRequestData {
@@ -19,6 +20,10 @@ export interface HelpRequestData {
displayMode?: "full" | "compact";
/** CSS selectors to scroll to + flash-highlight. */
selectors: string[];
/** Pre-resolved top-viewport rectangles, used for cross-frame targets. */
rects?: HelpHighlightRect[];
/** Re-resolve cross-frame rectangles after scroll, resize, or layout changes. */
refreshRects?: () => Promise<HelpHighlightRect[] | undefined>;
onContinue: (note: string) => void;
onCancel: () => void;
}
@@ -49,6 +54,7 @@ const PANEL_WIDTH = 420;
const FALLBACK_PANEL_H = 180;
const FALLBACK_PANEL_H_COLLAPSED = 44;
const EMPTY_SELECTORS: string[] = [];
const EMPTY_RECTS: HelpHighlightRect[] = [];
function clamp(value: number, min: number, max: number): number {
return Math.max(min, Math.min(max, value));
@@ -179,6 +185,8 @@ export function HelpRequestOverlay({ request }: Props) {
const { t } = useTranslation("extension");
const isCompact = request?.displayMode === "compact";
const effectiveSelectors = isCompact ? EMPTY_SELECTORS : (request?.selectors ?? EMPTY_SELECTORS);
const [liveRects, setLiveRects] = useState<HelpHighlightRect[] | null>(null);
const effectiveRects = isCompact ? EMPTY_RECTS : (liveRects ?? request?.rects ?? EMPTY_RECTS);
const [note, setNote] = useState("");
const [collapsed, setCollapsed] = useState(false);
const [boxes, setBoxes] = useState<Box[]>([]);
@@ -201,6 +209,7 @@ export function HelpRequestOverlay({ request }: Props) {
// Reset the note and collapse state whenever a new request appears.
useEffect(() => {
setNote("");
setLiveRects(null);
setCollapsed(false);
placementRef.current = null;
setDragPos(null);
@@ -209,6 +218,32 @@ export function HelpRequestOverlay({ request }: Props) {
dragStartRef.current = null;
}, [request?.id]);
useEffect(() => {
if (!request || isCompact || !request.refreshRects) return;
let disposed = false;
let resolving = false;
const refresh = async () => {
if (resolving) return;
resolving = true;
try {
const rects = await request.refreshRects?.();
if (!disposed && rects) setLiveRects(rects);
} finally {
resolving = false;
}
};
const onViewportChange = () => void refresh();
window.addEventListener("scroll", onViewportChange, true);
window.addEventListener("resize", onViewportChange);
const interval = window.setInterval(onViewportChange, 500);
return () => {
disposed = true;
window.removeEventListener("scroll", onViewportChange, true);
window.removeEventListener("resize", onViewportChange);
window.clearInterval(interval);
};
}, [request, isCompact]);
// Scroll the first matched target into view once per distinct request id.
useLayoutEffect(() => {
if (!request) return;
@@ -231,7 +266,7 @@ export function HelpRequestOverlay({ request }: Props) {
return;
}
const update = () => {
const nextBoxes = measure(effectiveSelectors);
const nextBoxes = [...measure(effectiveSelectors), ...effectiveRects];
setBoxes((prev) => (boxesEqual(prev, nextBoxes) ? prev : nextBoxes));
};
update();
@@ -243,7 +278,7 @@ export function HelpRequestOverlay({ request }: Props) {
window.removeEventListener("resize", update);
window.clearInterval(interval);
};
}, [request, effectiveSelectors]);
}, [request, effectiveSelectors, effectiveRects]);
const onHeaderPointerDown = useCallback(
(e: ReactPointerEvent<HTMLDivElement>) => {
@@ -305,7 +340,7 @@ export function HelpRequestOverlay({ request }: Props) {
return;
}
if (userMovedRef.current) return;
const measured = boxes.length > 0 ? boxes : measure(effectiveSelectors);
const measured = boxes.length > 0 ? boxes : [...measure(effectiveSelectors), ...effectiveRects];
const anchor = unionBox(measured);
const banner = bannerRef.current;
if (!anchor || !banner) {
@@ -329,7 +364,7 @@ export function HelpRequestOverlay({ request }: Props) {
}
const nextPos = placePanel(anchor, panelW, panelH, placementRef.current.placement);
setPanelPos((prev) => (panelPosEqual(prev, nextPos) ? prev : nextPos));
}, [request, boxes, collapsed, isCompact, effectiveSelectors]);
}, [request, boxes, collapsed, isCompact, effectiveSelectors, effectiveRects]);
if (!request) return null;
@@ -1,6 +1,6 @@
import { i18n } from "@browser-skill/i18n";
import { I18nextProvider } from "@browser-skill/i18n/react";
import { cleanup, fireEvent, render, screen } from "@testing-library/react";
import { cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react";
import { createElement } from "react";
import { afterEach, describe, expect, it, vi } from "vitest";
import { type HelpRequestData, HelpRequestOverlay } from "../HelpRequestOverlay";
@@ -38,6 +38,36 @@ describe("HelpRequestOverlay", () => {
expect(screen.getByText("Please complete the captcha")).toBeTruthy();
});
it("renders explicit viewport rectangles for cross-frame targets", () => {
const { container } = renderOverlay(
baseRequest({ rects: [{ top: 30, left: 40, width: 120, height: 50 }] }),
);
const highlight = container.querySelector<HTMLElement>("[data-slot='help-highlight']");
expect(highlight?.style.top).toBe("30px");
expect(highlight?.style.left).toBe("40px");
expect(highlight?.style.width).toBe("120px");
expect(highlight?.style.height).toBe("50px");
});
it("re-resolves cross-frame rectangles after the viewport changes", async () => {
const refreshRects = vi.fn(async () => [{ top: 80, left: 90, width: 140, height: 60 }]);
const { container } = renderOverlay(
baseRequest({
rects: [{ top: 30, left: 40, width: 120, height: 50 }],
refreshRects,
}),
);
window.dispatchEvent(new Event("scroll"));
await waitFor(() => {
const highlight = container.querySelector<HTMLElement>("[data-slot='help-highlight']");
expect(highlight?.style.top).toBe("80px");
expect(highlight?.style.left).toBe("90px");
});
expect(refreshRects).toHaveBeenCalledTimes(1);
});
it("keeps the inactive render stable", () => {
const { container, rerender } = renderOverlay(null);
@@ -0,0 +1,64 @@
import { describe, expect, it, vi } from "vitest";
import { createHelpRequestData } from "../help-request";
describe("createHelpRequestData", () => {
it("gives live delivery and recovery the same rectangle refresh mechanism", async () => {
const finish = vi.fn();
const query = vi.fn(async () => ({
active: true,
request: {
requestId: "help-1",
prompt: "complete the challenge",
selectors: [],
rects: [{ top: 80, left: 90, width: 140, height: 60 }],
timeoutMs: 1_000,
},
}));
const request = createHelpRequestData(
{
requestId: "help-1",
prompt: "complete the challenge",
selectors: [],
rects: [{ top: 10, left: 20, width: 30, height: 40 }],
timeoutMs: 1_000,
},
{ finish, query },
);
await expect(request.refreshRects?.()).resolves.toEqual([
{ top: 80, left: 90, width: 140, height: 60 },
]);
request.onContinue(" done ");
request.onCancel();
expect(query).toHaveBeenCalledOnce();
expect(finish).toHaveBeenNthCalledWith(1, "help-1", "continued", " done ");
expect(finish).toHaveBeenNthCalledWith(2, "help-1", "cancelled");
});
it("ignores rectangle replies for a replaced request", async () => {
const request = createHelpRequestData(
{
requestId: "help-1",
prompt: "complete the challenge",
selectors: [],
timeoutMs: 1_000,
},
{
finish: vi.fn(),
query: vi.fn(async () => ({
active: true,
request: {
requestId: "help-2",
prompt: "new challenge",
selectors: [],
rects: [{ top: 1, left: 2, width: 3, height: 4 }],
timeoutMs: 1_000,
},
})),
},
);
await expect(request.refreshRects?.()).resolves.toBeUndefined();
});
});
@@ -0,0 +1,31 @@
import type { HelpQueryResponse, HelpRequestMessage } from "@/lib/help-bridge";
import type { HelpRequestData } from "./HelpRequestOverlay";
export interface HelpRequestDataDeps {
finish(requestId: string, outcome: "continued" | "cancelled", note?: string): void;
query(): Promise<HelpQueryResponse | undefined>;
}
/** Build the content-side representation shared by live delivery and recovery. */
export function createHelpRequestData(
message: Omit<HelpRequestMessage, "type">,
deps: HelpRequestDataDeps,
): HelpRequestData {
return {
id: message.requestId,
prompt: message.prompt,
...(message.title ? { title: message.title } : {}),
...(message.displayMode ? { displayMode: message.displayMode } : {}),
selectors: message.selectors,
rects: message.rects,
refreshRects: async () => {
const response = await deps.query();
return response?.active && response.request?.requestId === message.requestId
? response.request.rects
: undefined;
},
onContinue: (note: string) =>
deps.finish(message.requestId, "continued", note.trim() ? note : undefined),
onCancel: () => deps.finish(message.requestId, "cancelled"),
};
}
+12 -9
View File
@@ -1,9 +1,9 @@
import {
type CaptureTargetDescriptor,
describeEventTarget,
describeTarget,
resolveClickableElement,
resolveHoverElement,
type TargetDescriptor,
} from "@/lib/describe-target";
import {
evaluateHoverTrigger,
@@ -27,7 +27,7 @@ import {
type RecordStopAck,
type RecordStopMessage,
} from "@/lib/record-bridge";
import { shouldRecordPress } from "@/lib/trace-reducer";
import { shouldRecordPress } from "@/lib/recording/draft-policy";
import {
closestHoverSurfaceCandidate,
collectHoverSurfaceStates,
@@ -65,14 +65,14 @@ export interface RecordCaptureController {
interface FillSession {
element: FillableElement;
target: TargetDescriptor;
target: CaptureTargetDescriptor;
baselineValue: string;
lastValue: string;
}
interface HoverCandidate {
element: Element;
target: TargetDescriptor;
target: CaptureTargetDescriptor;
recordedAt: number;
score: number;
eligible: boolean;
@@ -232,7 +232,10 @@ function collectHoverTriggerLabelText(root: Element): string {
return normalizeLabelText(text);
}
function compactHoverTargetName(el: Element, desc: TargetDescriptor): TargetDescriptor {
function compactHoverTargetName(
el: Element,
desc: CaptureTargetDescriptor,
): CaptureTargetDescriptor {
if (!desc.name) return desc;
const fullText = normalizeLabelText(el.textContent ?? "");
const compactName = collectHoverTriggerLabelText(el);
@@ -250,7 +253,7 @@ function compactHoverTargetName(el: Element, desc: TargetDescriptor): TargetDesc
return desc;
}
function isWeakHoverTarget(target: TargetDescriptor): boolean {
function isWeakHoverTarget(target: CaptureTargetDescriptor): boolean {
return !target.role && !target.name && target.tag === "div";
}
@@ -261,9 +264,9 @@ function looksLikeAvatarElement(el: Element): boolean {
function normalizeHoverTarget(
el: Element,
desc: TargetDescriptor,
desc: CaptureTargetDescriptor,
decision: HoverTriggerDecision,
): TargetDescriptor {
): CaptureTargetDescriptor {
if (desc.role === "img" && !desc.name && looksLikeAvatarElement(el)) {
return { ...desc, name: "image" };
}
@@ -277,7 +280,7 @@ function normalizeHoverTarget(
return desc;
}
function hoverTriggerSignals(el: Element, desc: TargetDescriptor) {
function hoverTriggerSignals(el: Element, desc: CaptureTargetDescriptor) {
if (!(el instanceof HTMLElement)) return null;
const style = hoverTriggerStyle(el);
return {
+14 -21
View File
@@ -7,6 +7,7 @@ import { BorrowConfirmationOverlay } from "@/content/BorrowConfirmationOverlay";
import { ControlOverlay } from "@/content/ControlOverlay";
import { createCaptureSuppressController } from "@/content/capture-suppress";
import { HelpRequestOverlay } from "@/content/HelpRequestOverlay";
import { createHelpRequestData } from "@/content/help-request";
import overlayCss from "@/content/overlay.css?inline";
import { OverlayController, shouldShowAgentControlOverlay } from "@/content/overlay-controller";
import { RecordOverlay } from "@/content/RecordOverlay";
@@ -344,16 +345,7 @@ export default defineContentScript({
if (isHelpRequestMessage(message)) {
const helpMsg = message as HelpRequestMessage;
const previousHelp = overlays.setAgentHelpRequest({
id: helpMsg.requestId,
prompt: helpMsg.prompt,
...(helpMsg.title ? { title: helpMsg.title } : {}),
...(helpMsg.displayMode ? { displayMode: helpMsg.displayMode } : {}),
selectors: helpMsg.selectors,
onContinue: (note: string) =>
void sendHelpFinish(helpMsg.requestId, "continued", note.trim() ? note : undefined),
onCancel: () => void sendHelpFinish(helpMsg.requestId, "cancelled"),
});
const previousHelp = mountHelpRequest(helpMsg);
if (previousHelp && previousHelp.id !== helpMsg.requestId) {
void sendHelpFinish(previousHelp.id, "cancelled");
}
@@ -406,17 +398,18 @@ export default defineContentScript({
});
}
function mountHelpRequest(helpMsg: Omit<HelpRequestMessage, "type">): void {
overlays.setAgentHelpRequest({
id: helpMsg.requestId,
prompt: helpMsg.prompt,
...(helpMsg.title ? { title: helpMsg.title } : {}),
...(helpMsg.displayMode ? { displayMode: helpMsg.displayMode } : {}),
selectors: helpMsg.selectors,
onContinue: (note: string) =>
void sendHelpFinish(helpMsg.requestId, "continued", note.trim() ? note : undefined),
onCancel: () => void sendHelpFinish(helpMsg.requestId, "cancelled"),
});
function mountHelpRequest(helpMsg: Omit<HelpRequestMessage, "type">) {
return overlays.setAgentHelpRequest(
createHelpRequestData(helpMsg, {
finish: (requestId, outcome, note) => {
void sendHelpFinish(requestId, outcome, note);
},
query: () =>
chrome.runtime.sendMessage({ type: HELP_QUERY }) as Promise<
HelpQueryResponse | undefined
>,
}),
);
}
async function queryActiveHelpWithRetry(): Promise<void> {
@@ -0,0 +1,101 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import type { CdpRunner } from "@/tools/shared";
import { waitForDocumentSettled } from "../recording/document-settle";
function quietCdp(): { cdp: CdpRunner; send: ReturnType<typeof vi.fn> } {
const send = vi.fn(async () => ({
result: { value: { idleMs: 1_000, readyState: "complete" } },
}));
return {
cdp: { send: send as unknown as CdpRunner["send"] },
send,
};
}
describe("waitForDocumentSettled", () => {
afterEach(() => {
vi.useRealTimers();
});
it("waits through the minimum observation floor before accepting a quiet page", async () => {
vi.useFakeTimers();
const { cdp, send } = quietCdp();
const settled = waitForDocumentSettled(cdp, { target: { tabId: 7 } });
await vi.advanceTimersByTimeAsync(180);
await expect(settled).resolves.toBe("quiet");
expect(send).toHaveBeenCalledTimes(3);
});
it("cancels before probing a superseded observation", async () => {
vi.useFakeTimers();
const { cdp, send } = quietCdp();
const controller = new AbortController();
const settled = waitForDocumentSettled(
cdp,
{ target: { tabId: 7 } },
{
signal: controller.signal,
},
);
controller.abort();
await vi.advanceTimersByTimeAsync(60);
await expect(settled).resolves.toBe("cancelled");
expect(send).not.toHaveBeenCalled();
});
it("probes a same-process iframe through its isolated execution context", async () => {
vi.useFakeTimers();
const send = vi.fn(async (_tabId: number, method: string) => {
if (method === "Page.createIsolatedWorld") return { executionContextId: 91 };
return { result: { value: { idleMs: 1_000, readyState: "complete" } } };
});
const cdp: CdpRunner = { send: send as unknown as CdpRunner["send"] };
const settled = waitForDocumentSettled(cdp, {
target: { tabId: 7 },
frameId: "child-frame",
});
await vi.advanceTimersByTimeAsync(180);
await expect(settled).resolves.toBe("quiet");
expect(send).toHaveBeenCalledWith(
7,
"Page.createIsolatedWorld",
expect.objectContaining({ frameId: "child-frame" }),
);
expect(send).toHaveBeenCalledWith(
7,
"Runtime.evaluate",
expect.objectContaining({ contextId: 91 }),
);
});
it("routes an OOPIF probe through its CDP target session", async () => {
vi.useFakeTimers();
const send = vi.fn();
const sendToTarget = vi.fn(async (_target, method: string) => {
if (method === "Page.createIsolatedWorld") return { executionContextId: 27 };
return { result: { value: { idleMs: 1_000, readyState: "complete" } } };
});
const cdp: CdpRunner = {
send: send as unknown as CdpRunner["send"],
sendToTarget: sendToTarget as unknown as NonNullable<CdpRunner["sendToTarget"]>,
};
const target = { tabId: 7, sessionId: "oopif-session" };
const settled = waitForDocumentSettled(cdp, { target, frameId: "oopif-frame" });
await vi.advanceTimersByTimeAsync(180);
await expect(settled).resolves.toBe("quiet");
expect(send).not.toHaveBeenCalled();
expect(sendToTarget).toHaveBeenCalledWith(
target,
"Runtime.evaluate",
expect.objectContaining({ contextId: 27 }),
);
});
});
@@ -32,6 +32,7 @@ describe("help-bridge", () => {
prompt: "log in",
displayMode: "compact",
selectors: ["#login"],
rects: [{ top: 10, left: 20, width: 100, height: 40 }],
timeoutMs: 1000,
}),
).toBe(true);
@@ -0,0 +1,141 @@
import { describe, expect, it } from "vitest";
import { ObservationNodeIndex } from "../recording/observation-capture";
import { RecordingObservationSession } from "../recording/observation-session";
import { RecordingStateRegistry } from "../recording/state-registry";
import { buildTraceV3 } from "../recording/trace-builder-v3";
import type { RecordingDraftStep } from "../recording/types";
const URL = "https://example.com/login";
function sessionWithInput(redactValues = false): RecordingObservationSession {
const session = new RecordingObservationSession({ redactValues });
const state = session.registry.register({
url: URL,
rawVomText: '@vom 1\ntextbox "Password" value="••••••" [ref=e1]',
});
session.cursor.lastSettled = {
stateId: state.id,
rootFrameId: "root",
index: new ObservationNodeIndex({
rootFrameId: "root",
matchNodes: [
{
frameId: "root",
backendNodeId: 42,
tag: "input",
rect: { x: 20, y: 40, w: 200, h: 30 },
localRect: { x: 20, y: 40, w: 200, h: 30 },
},
],
refs: [{ ref: "e1", backendNodeId: 42, role: "textbox", name: "Password", line: 1 }],
}),
url: URL,
};
return session;
}
function finalizedFillBody(value: string, redactValues: boolean): string {
const session = sessionWithInput(redactValues);
const draft: RecordingDraftStep = {
op: "fill",
captureTarget: { tag: "input", role: "textbox", name: "Password" },
value,
targetHint: {
geometry: { rect: { x: 20, y: 40, w: 200, h: 30 }, tag: "input" },
},
};
session.bindDraft(draft, 1);
draft.postStateId = draft.preStateId;
return buildTraceV3({
registry: session.registry,
drafts: [draft],
annotations: session.annotations,
startedAt: "2026-08-12T00:00:00.000Z",
stoppedBy: "user_finish",
bskVersion: "test",
}).states[0]!.body;
}
describe("record observation annotations", () => {
it("omits a fill literal when values are redacted", () => {
const secret = "hunter2-private";
const body = finalizedFillBody(secret, true);
expect(body).toContain("step 1: fill");
expect(body).not.toContain(secret);
});
it("keeps ordinary fill details", () => {
expect(finalizedFillBody("ordinary text", false)).toContain('step 1: fill: "ordinary text"');
});
it("encodes title and URL so line breaks cannot corrupt state metadata", () => {
const registry = new RecordingStateRegistry();
const state = registry.register({
url: "https://example.com/a\nb",
title: "hello\nworld",
rawVomText: "@vom 1",
});
const trace = buildTraceV3({
registry,
drafts: [{ op: "scroll", preStateId: state.id, postStateId: state.id }],
startedAt: "2026-08-12T00:00:00.000Z",
stoppedBy: "user_finish",
bskVersion: "test",
});
expect(trace.states[0]?.body).toContain('url: "https://example.com/a\\nb"');
expect(trace.states[0]?.body).toContain('title: "hello\\nworld"');
});
});
describe("recording state ownership", () => {
it("deduplicates within one recording and isolates ids between recordings", () => {
const first = new RecordingStateRegistry();
const second = new RecordingStateRegistry();
expect(first.register({ url: URL, rawVomText: "same" }).id).toBe("s1");
expect(first.register({ url: URL, rawVomText: "same" }).id).toBe("s1");
expect(second.register({ url: URL, rawVomText: "other" }).id).toBe("s1");
});
it("enriches metadata when a deduplicated observation becomes more complete", () => {
const registry = new RecordingStateRegistry();
registry.register({ url: URL, rawVomText: "same" });
const state = registry.register({
url: URL,
title: "Login",
rawVomText: "same",
truncated: true,
});
expect(state).toMatchObject({ id: "s1", title: "Login", truncated: true });
});
});
describe("draft binding", () => {
it("keeps capture semantics when no observation exists", () => {
const session = new RecordingObservationSession();
const draft: RecordingDraftStep = {
op: "hover",
captureTarget: { tag: "button", role: "button", name: "新建" },
};
session.bindDraft(draft, 1);
expect(draft.matchedTarget).toEqual({ role: "button", name: "新建", unmatched: true });
expect(draft.preStateId).toBeUndefined();
});
it("does not bind an unmatched new action to a stale observation", () => {
const session = sessionWithInput();
const draft: RecordingDraftStep = {
op: "click",
captureTarget: { tag: "button", role: "button", name: "Confirm" },
targetHint: {
geometry: { rect: { x: 400, y: 300, w: 80, h: 30 }, tag: "button" },
},
};
session.bindDraft(draft, 2, true);
expect(draft.preStateId).toBeUndefined();
expect("matchedTarget" in draft ? draft.matchedTarget : undefined).toEqual({
role: "button",
name: "Confirm",
unmatched: true,
});
});
});
@@ -0,0 +1,136 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import type { CdpRunner, ChromeTabsApi } from "@/tools/shared";
import { ObservationNodeIndex, type RegisteredObservation } from "../recording/observation-capture";
import { RecordingObservationSession } from "../recording/observation-session";
import { SettleController } from "../recording/settle-controller";
import type { RecordingDraftStep } from "../recording/types";
const OBSERVATION: RegisteredObservation = {
stateId: "s-next",
rootFrameId: "root",
index: new ObservationNodeIndex({ rootFrameId: "root", matchNodes: [], refs: [] }),
url: "https://example.com/next",
};
describe("SettleController", () => {
afterEach(() => vi.useRealTimers());
it("aborts superseded capture work and lands the prior action on the next origin", async () => {
vi.useFakeTimers();
const cdp: CdpRunner = {
send: vi.fn(async () => ({
result: { value: { idleMs: 1_000, readyState: "complete" } },
})) as unknown as CdpRunner["send"],
};
const tabsApi: ChromeTabsApi = {
get: vi.fn(async () => ({ id: 7, url: OBSERVATION.url }) as chrome.tabs.Tab),
query: vi.fn(async () => []),
};
const session = new RecordingObservationSession();
let firstSignal: AbortSignal | undefined;
vi.spyOn(session, "capture")
.mockImplementationOnce(async (_cdp, _tabs, _tabId, signal) => {
firstSignal = signal;
return new Promise<RegisteredObservation>((_resolve, reject) => {
signal?.addEventListener(
"abort",
() => reject(new DOMException("observation aborted", "AbortError")),
{ once: true },
);
});
})
.mockResolvedValueOnce(OBSERVATION);
const drafts: RecordingDraftStep[] = [
{ op: "click", captureTarget: { tag: "button", name: "First" }, preStateId: "s1" },
];
const controller = new SettleController({ session, cdp, tabsApi, tabId: 7 });
controller.schedule(drafts, 0);
await vi.advanceTimersByTimeAsync(180);
drafts.push({
op: "click",
captureTarget: { tag: "button", name: "Second" },
preStateId: "s-next",
});
controller.schedule(drafts, 1);
expect(firstSignal?.aborted).toBe(true);
expect(drafts[0]?.postStateId).toBe("s-next");
await vi.advanceTimersByTimeAsync(500);
await controller.flush();
expect(drafts[1]?.postStateId).toBe("s-next");
});
it("does not consume a newer redirect while reading the prior landing URL", async () => {
vi.useFakeTimers();
const cdp: CdpRunner = {
send: vi.fn(async () => ({
result: { value: { idleMs: 1_000, readyState: "complete" } },
})) as unknown as CdpRunner["send"],
};
let resolveFirstTab!: (tab: chrome.tabs.Tab) => void;
const firstTab = new Promise<chrome.tabs.Tab>((resolve) => {
resolveFirstTab = resolve;
});
const tabsApi: ChromeTabsApi = {
get: vi
.fn()
.mockImplementationOnce(() => firstTab)
.mockResolvedValue({ id: 7, url: OBSERVATION.url }),
query: vi.fn(async () => []),
};
const session = new RecordingObservationSession();
session.cursor.lastSettled = OBSERVATION;
const drafts: RecordingDraftStep[] = [];
const controller = new SettleController({ session, cdp, tabsApi, tabId: 7 });
controller.scheduleRedirect(drafts, "https://example.com/intermediate");
await vi.advanceTimersByTimeAsync(180);
expect(tabsApi.get).toHaveBeenCalledTimes(1);
controller.scheduleRedirect(drafts, OBSERVATION.url);
resolveFirstTab({ id: 7, url: "https://example.com/intermediate" } as chrome.tabs.Tab);
await vi.advanceTimersByTimeAsync(500);
await controller.flushRedirects();
expect(drafts).toEqual([]);
expect(controller.hasPending).toBe(false);
});
it("settles an action in its own OOPIF document scope", async () => {
vi.useFakeTimers();
const send = vi.fn();
const sendToTarget = vi.fn(async (_target, method: string) => {
if (method === "Page.createIsolatedWorld") return { executionContextId: 39 };
return { result: { value: { idleMs: 1_000, readyState: "complete" } } };
});
const cdp: CdpRunner = {
send: send as unknown as CdpRunner["send"],
sendToTarget: sendToTarget as unknown as NonNullable<CdpRunner["sendToTarget"]>,
};
const tabsApi: ChromeTabsApi = {
get: vi.fn(async () => ({ id: 7, url: OBSERVATION.url }) as chrome.tabs.Tab),
query: vi.fn(async () => []),
};
const session = new RecordingObservationSession();
vi.spyOn(session, "capture").mockResolvedValue(OBSERVATION);
const drafts: RecordingDraftStep[] = [
{ op: "click", captureTarget: { tag: "button", name: "Inside frame" } },
];
const controller = new SettleController({ session, cdp, tabsApi, tabId: 7 });
const target = { tabId: 7, sessionId: "oopif-session" };
controller.schedule(drafts, 0, { target, frameId: "oopif-frame" });
await vi.advanceTimersByTimeAsync(180);
await controller.flush();
expect(send).not.toHaveBeenCalled();
expect(sendToTarget).toHaveBeenCalledWith(
target,
"Runtime.evaluate",
expect.objectContaining({ contextId: 39 }),
);
expect(drafts[0]?.postStateId).toBe("s-next");
});
});
@@ -1,5 +1,5 @@
import { describe, expect, it } from "vitest";
import { appendRecordedPayload, observeRecordedNavigation } from "../recording-step-buffer";
import { appendRecordedPayload, observeRecordedNavigation } from "../recording/step-buffer";
describe("recording-step-buffer", () => {
it("stores semantic click without summary", () => {
@@ -12,7 +12,7 @@ describe("recording-step-buffer", () => {
expect(buffer.steps).toEqual([
{
op: "click",
target: { tag: "button", role: "button", name: "发布" },
captureTarget: { tag: "button", role: "button", name: "发布" },
},
]);
expect(buffer.pendingNavigation).toBe(true);
@@ -23,7 +23,7 @@ describe("recording-step-buffer", () => {
steps: [
{
op: "click" as const,
target: { tag: "button", role: "button", name: "发布" },
captureTarget: { tag: "button", role: "button", name: "发布" },
},
],
currentUrl: "https://example.com/a",
@@ -34,8 +34,8 @@ describe("recording-step-buffer", () => {
expect(buffer.steps).toEqual([
{
op: "click",
target: { tag: "button", role: "button", name: "发布" },
navigated_to: "https://example.com/b",
captureTarget: { tag: "button", role: "button", name: "发布" },
navigatedTo: "https://example.com/b",
},
]);
expect(JSON.stringify(buffer.steps)).not.toContain("wait_for_navigation");
@@ -46,7 +46,7 @@ describe("recording-step-buffer", () => {
steps: [
{
op: "select" as const,
target: { tag: "select", role: "combobox", name: "分类" },
captureTarget: { tag: "select", role: "combobox", name: "分类" },
values: ["tech"],
},
],
@@ -58,9 +58,9 @@ describe("recording-step-buffer", () => {
expect(buffer.steps).toEqual([
{
op: "select",
target: { tag: "select", role: "combobox", name: "分类" },
captureTarget: { tag: "select", role: "combobox", name: "分类" },
values: ["tech"],
navigated_to: "https://example.com/list?cat=tech",
navigatedTo: "https://example.com/list?cat=tech",
},
]);
});
@@ -0,0 +1,127 @@
import type { RenderedRef } from "@browser-skill/vom";
import { describe, expect, it } from "vitest";
import type { CaptureVomMatchNode } from "@/tools/capture-vom-observation";
import { ObservationNodeIndex, type RegisteredObservation } from "../recording/observation-capture";
import { matchObservationTarget } from "../recording/target-matcher";
function node(backendNodeId: number, frameId: string, x = 10): CaptureVomMatchNode {
return {
backendNodeId,
frameId,
tag: "button",
rect: { x, y: 20, w: 100, h: 30 },
localRect: { x, y: 20, w: 100, h: 30 },
};
}
function observation(
matchNodes: CaptureVomMatchNode[],
refs: RenderedRef[],
): RegisteredObservation {
return {
stateId: "s1",
rootFrameId: "root",
index: new ObservationNodeIndex({ rootFrameId: "root", matchNodes, refs }),
url: "https://example.com",
};
}
describe("matchObservationTarget", () => {
it("indexes the safe geometry DTO without discarding frame-local geometry", () => {
const geometry = node(42, "child");
geometry.localRect = { x: 5, y: 6, w: 100, h: 30 };
const ref: RenderedRef = {
ref: "e1",
backendNodeId: 42,
frameId: "child",
line: 1,
};
const candidate = observation([geometry], [ref]).index.candidates("child", "button")[0];
expect(candidate).toEqual({ frameId: "child", geometry, ref });
});
it("matches a unique node using canonical top-level viewport geometry", () => {
const target = matchObservationTarget({
observation: observation(
[node(42, "root")],
[{ ref: "e1", backendNodeId: 42, role: "button", name: "发布", line: 1 }],
),
hint: { geometry: { rect: { x: 10, y: 20, w: 100, h: 30 }, tag: "button" } },
});
expect(target).toEqual({ ref: "e1", role: "button", name: "发布" });
});
it("uses frame id with backend node id so sibling frames cannot collide", () => {
const target = matchObservationTarget({
observation: observation(
[node(42, "left"), node(42, "right")],
[
{ ref: "e1", backendNodeId: 42, frameId: "left", line: 1 },
{ ref: "e2", backendNodeId: 42, frameId: "right", line: 2 },
],
),
hint: {
frameId: "right",
geometry: { rect: { x: 10, y: 20, w: 100, h: 30 }, tag: "button" },
},
});
expect(target.ref).toBe("e2");
});
it("restricts a missing frame hint to the root frame", () => {
const target = matchObservationTarget({
observation: observation(
[node(42, "child")],
[{ ref: "e1", backendNodeId: 42, frameId: "child", line: 1 }],
),
hint: { geometry: { rect: { x: 10, y: 20, w: 100, h: 30 }, tag: "button" } },
fallback: { tag: "button", name: "发布" },
});
expect(target).toEqual({ name: "发布", unmatched: true });
});
it("returns unmatched for ambiguous geometry", () => {
const target = matchObservationTarget({
observation: observation(
[node(42, "root"), node(43, "root")],
[
{ ref: "e1", backendNodeId: 42, line: 1 },
{ ref: "e2", backendNodeId: 43, line: 2 },
],
),
hint: { geometry: { rect: { x: 10, y: 20, w: 100, h: 30 }, tag: "button" } },
});
expect(target.unmatched).toBe(true);
});
it("uses semantics when geometry is unavailable without crossing frame boundaries", () => {
const target = matchObservationTarget({
observation: observation(
[node(41, "root"), node(42, "child")],
[
{ ref: "e1", backendNodeId: 41, role: "button", name: "保存", line: 1 },
{ ref: "e2", backendNodeId: 42, frameId: "child", role: "button", name: "保存", line: 2 },
],
),
hint: { frameId: "child" },
fallback: { tag: "button", role: "button", name: "保存" },
});
expect(target.ref).toBe("e2");
});
it("uses semantics to disambiguate equal geometry in one frame", () => {
const target = matchObservationTarget({
observation: observation(
[node(42, "root"), node(43, "root")],
[
{ ref: "e1", backendNodeId: 42, role: "button", name: "保存", line: 1 },
{ ref: "e2", backendNodeId: 43, role: "button", name: "取消", line: 2 },
],
),
hint: { geometry: { rect: { x: 10, y: 20, w: 100, h: 30 }, tag: "button" } },
fallback: { tag: "button", role: "button", name: "取消" },
});
expect(target.ref).toBe("e2");
});
});
@@ -1,6 +1,15 @@
import { describe, expect, it } from "vitest";
import type { DraftTraceStep } from "@/transport/types";
import { reduceTraceSteps, resolveTraceStartUrl, shouldRecordPress } from "../trace-reducer";
import type { RecordingDraftStep } from "@/lib/recording/types";
import { shouldRecordPress } from "../recording/draft-policy";
import { buildTraceV2 } from "../recording/trace-reducer-v2";
function reduceTraceSteps(steps: RecordingDraftStep[], startUrl?: string) {
return buildTraceV2({
steps,
startedAt: "2026-01-01T00:00:00.000Z",
...(startUrl ? { startUrl } : {}),
});
}
describe("shouldRecordPress", () => {
it("keeps Enter and Escape", () => {
@@ -19,35 +28,35 @@ describe("shouldRecordPress", () => {
describe("reduceTraceSteps", () => {
it("builds steps with pages dictionary and page id references", () => {
const drafts: DraftTraceStep[] = [
const drafts: RecordingDraftStep[] = [
{
op: "navigate",
url: "https://example.com/search?q=hello&utm_source=x",
page_url: "https://example.com/search?q=hello&utm_source=x",
pageUrl: "https://example.com/search?q=hello&utm_source=x",
},
{
op: "fill",
target: { tag: "input", role: "textbox", name: "搜索", name_attr: "q" },
captureTarget: { tag: "input", role: "textbox", name: "搜索", name_attr: "q" },
value: "browser skill",
page_url: "https://example.com/search?q=hello&utm_source=x",
pageUrl: "https://example.com/search?q=hello&utm_source=x",
},
{
op: "press",
key: "Enter",
target: { tag: "input", role: "textbox", name: "搜索", name_attr: "q" },
navigated_to: "https://example.com/results/42",
page_url: "https://example.com/search?q=hello&utm_source=x",
captureTarget: { tag: "input", role: "textbox", name: "搜索", name_attr: "q" },
navigatedTo: "https://example.com/results/42",
pageUrl: "https://example.com/search?q=hello&utm_source=x",
},
{
op: "click",
target: { tag: "button", role: "button", name: "发布" },
navigated_to: "https://example.com/p/99",
page_url: "https://example.com/results/42",
captureTarget: { tag: "button", role: "button", name: "发布" },
navigatedTo: "https://example.com/p/99",
pageUrl: "https://example.com/results/42",
},
{
op: "press",
key: "a",
page_url: "https://example.com/p/99",
pageUrl: "https://example.com/p/99",
},
];
@@ -109,9 +118,9 @@ describe("reduceTraceSteps", () => {
[
{
op: "fill",
target: { tag: "input", role: "textbox", name: "Search query" },
captureTarget: { tag: "input", role: "textbox", name: "Search query" },
value: "",
page_url: "https://example.com/search",
pageUrl: "https://example.com/search",
},
],
"https://example.com/search",
@@ -125,16 +134,16 @@ describe("reduceTraceSteps", () => {
]);
});
it("maps select navigated_to onto effect.navigated_to (page id)", () => {
it("maps select navigatedTo onto effect.navigatedTo (page id)", () => {
const { pages, steps } = reduceTraceSteps(
[
{
op: "select",
target: { tag: "select", role: "combobox", name: "分类" },
captureTarget: { tag: "select", role: "combobox", name: "分类" },
values: ["tech"],
labels: ["技术"],
navigated_to: "https://example.com/list?cat=tech",
page_url: "https://example.com/list",
navigatedTo: "https://example.com/list?cat=tech",
pageUrl: "https://example.com/list",
},
],
"https://example.com/list",
@@ -156,13 +165,13 @@ describe("reduceTraceSteps", () => {
[
{
op: "hover",
target: { tag: "span", role: "button", name: "Account" },
page_url: "https://example.com/app",
captureTarget: { tag: "span", role: "button", name: "Account" },
pageUrl: "https://example.com/app",
},
{
op: "click",
target: { tag: "a", role: "link", name: "Profile" },
page_url: "https://example.com/app",
captureTarget: { tag: "a", role: "link", name: "Profile" },
pageUrl: "https://example.com/app",
},
],
"https://example.com/app",
@@ -177,10 +186,11 @@ describe("reduceTraceSteps", () => {
it("resolveTraceStartUrl prefers explicit start URL", () => {
expect(
resolveTraceStartUrl(
[{ op: "navigate", url: "https://example.com/other" }],
"https://example.com/start",
),
buildTraceV2({
steps: [{ op: "navigate", url: "https://example.com/other" }],
startedAt: "2026-01-01T00:00:00.000Z",
startUrl: "https://example.com/start",
}).entry.start_url,
).toBe("https://example.com/start");
});
});
@@ -0,0 +1,62 @@
import { describe, expect, it } from "vitest";
import { TRACE_VERSION_V3, VOM_FORMAT_VERSION } from "@/transport/types";
import { RecordingStateRegistry } from "../recording/state-registry";
import { buildTraceV3 } from "../recording/trace-builder-v3";
import { reduceTraceStepsV3 } from "../recording/trace-reducer-v3";
import type { RecordingDraftStep } from "../recording/types";
describe("trace reducer v3", () => {
it("collapses redirect hops while retaining draft-to-step identity", () => {
const drafts: RecordingDraftStep[] = [
{ op: "navigate", url: "https://example.com/start", preStateId: "s1", postStateId: "s2" },
{
op: "navigate",
url: "https://example.com/final",
transitionQualifiers: ["server_redirect"],
preStateId: "s2",
postStateId: "s3",
},
];
const before = structuredClone(drafts);
const reduced = reduceTraceStepsV3(drafts);
expect(drafts).toEqual(before);
expect(reduced.steps).toEqual([
expect.objectContaining({
id: 1,
op: "navigate",
state: "s1",
to: "https://example.com/final",
result: { state: "s3" },
}),
]);
expect(reduced.stepIdByDraftId.get(1)).toBe(1);
expect(reduced.stepIdByDraftId.get(2)).toBe(1);
});
it("builds the wire model from protocol constants", () => {
const registry = new RecordingStateRegistry();
const state = registry.register({ url: "https://example.com", rawVomText: "@vom 1" });
const trace = buildTraceV3({
registry,
drafts: [
{
op: "click",
captureTarget: { tag: "button", role: "button", name: "Save" },
preStateId: state.id,
postStateId: state.id,
},
],
startedAt: "2026-08-12T00:00:00.000Z",
stoppedBy: "user_finish",
bskVersion: "test",
});
expect(trace.version).toBe(TRACE_VERSION_V3);
expect(trace.recorder.vom).toBe(VOM_FORMAT_VERSION);
expect(trace.steps[0]).toMatchObject({
op: "click",
target: { role: "button", name: "Save", unmatched: true },
});
});
});
+6 -5
View File
@@ -1,12 +1,13 @@
/**
* Build a semantic TargetDescriptor for an interacted element.
* Build a semantic capture descriptor for an interacted element.
*
* Trace steps are an LLM *textbook*: each click must say what to look for
* on screen (usually a short visible name). Tag-only noise like
* `{ "tag": "div" }` / “点击div” is useless and must not be recorded.
*/
export interface TargetDescriptor {
/** Content-script capture descriptor before VOM geometric matching. */
export interface CaptureTargetDescriptor {
role?: string;
name?: string;
tag: string;
@@ -247,7 +248,7 @@ export function resolveHoverElement(target: Element): Element | null {
* (visible name), or at least a form `name_attr` for checkbox/radio.
* Recording “点击div” with no name fails this bar.
*/
export function isMeaningfulClickTarget(target: TargetDescriptor): boolean {
export function isMeaningfulClickTarget(target: CaptureTargetDescriptor): boolean {
const name = target.name?.trim();
if (name && isActionableLabel(name)) return true;
if (
@@ -323,7 +324,7 @@ function nearbyLabelText(el: Element): string | undefined {
return undefined;
}
export function describeTarget(el: Element): TargetDescriptor {
export function describeTarget(el: Element): CaptureTargetDescriptor {
const tag = el.tagName.toLowerCase();
const role = inferRole(el);
const name = accessibleName(el);
@@ -350,7 +351,7 @@ export function describeTarget(el: Element): TargetDescriptor {
};
}
export function describeEventTarget(target: EventTarget | null): TargetDescriptor | null {
export function describeEventTarget(target: EventTarget | null): CaptureTargetDescriptor | null {
if (!(target instanceof Element)) return null;
const clickable = resolveClickableElement(target);
if (!clickable) return null;
+19
View File
@@ -16,6 +16,13 @@ export const HELP_ACK = "bsk-help-ack";
export const HELP_QUERY = "bsk-help-query";
export const HELP_FINISH = "bsk-help-finish";
export interface HelpHighlightRect {
top: number;
left: number;
width: number;
height: number;
}
export interface HelpRequestMessage {
type: typeof HELP_REQUEST;
requestId: string;
@@ -26,6 +33,8 @@ export interface HelpRequestMessage {
displayMode?: "full" | "compact";
/** CSS selectors to scroll to + flash-highlight (may be empty). */
selectors: string[];
/** Top-viewport rectangles for targets that cannot be represented by a root selector. */
rects?: HelpHighlightRect[];
timeoutMs: number;
}
@@ -74,6 +83,16 @@ export function isHelpRequestMessage(msg: unknown): msg is HelpRequestMessage {
(m.displayMode === undefined || m.displayMode === "full" || m.displayMode === "compact") &&
Array.isArray(m.selectors) &&
m.selectors.every((selector) => typeof selector === "string") &&
(m.rects === undefined ||
(Array.isArray(m.rects) &&
m.rects.every(
(rect) =>
typeof rect === "object" &&
rect !== null &&
["top", "left", "width", "height"].every((key) =>
Number.isFinite((rect as Record<string, unknown>)[key]),
),
))) &&
typeof m.timeoutMs === "number"
);
}
+2 -2
View File
@@ -3,7 +3,7 @@
* service worker and a tab's content script.
*/
import type { TargetDescriptor } from "./describe-target";
import type { CaptureTargetDescriptor } from "./describe-target";
export const RECORD_START = "bsk-record-start";
export const RECORD_STEP = "bsk-record-step";
@@ -42,7 +42,7 @@ export interface RecordStartMessage {
export interface RecordStepPayload {
op: "click" | "hover" | "fill" | "press" | "select" | "navigate";
target?: TargetDescriptor;
target?: CaptureTargetDescriptor;
value?: string;
key?: string;
modifiers?: Array<"alt" | "ctrl" | "meta" | "shift">;
@@ -0,0 +1,134 @@
import type { CdpTarget } from "@/browser-driver/frame-graph";
import { type CdpRunner, sendToCdpTarget } from "@/tools/shared";
const SETTLE_MIN_MS = 150;
const SETTLE_QUIET_MS = 250;
const SETTLE_MAX_MS = 2_000;
const SETTLE_POLL_MS = 60;
const SETTLE_WORLD_NAME = "__bsk_record_settle__";
const QUIET_PROBE = `(() => {
const scope = window;
let probe = scope.__bskRecordQuiet;
if (!probe) {
probe = { changedAt: Date.now() };
const observer = new MutationObserver(() => {
probe.changedAt = Date.now();
});
observer.observe(document.documentElement, {
subtree: true,
childList: true,
attributes: true,
characterData: true,
});
scope.__bskRecordQuiet = probe;
}
return { idleMs: Date.now() - probe.changedAt, readyState: document.readyState };
})()`;
export interface DocumentSettleScope {
target: CdpTarget;
/** Omit only for the target's root document. */
frameId?: string;
}
type SettleOutcome = "quiet" | "timeout" | "cancelled";
interface QuietProbe {
idleMs: number;
readyState: string;
}
interface ProbeContext {
executionContextId?: number;
}
function sleep(ms: number, signal?: AbortSignal): Promise<void> {
return new Promise((resolve) => {
if (signal?.aborted) return resolve();
const timer = setTimeout(resolve, ms);
signal?.addEventListener(
"abort",
() => {
clearTimeout(timer);
resolve();
},
{ once: true },
);
});
}
async function executionContextForScope(
cdp: CdpRunner,
scope: DocumentSettleScope,
): Promise<number | undefined> {
if (!scope.frameId) return undefined;
const result = await sendToCdpTarget<{ executionContextId?: number }>(
cdp,
scope.target,
"Page.createIsolatedWorld",
{
frameId: scope.frameId,
worldName: SETTLE_WORLD_NAME,
grantUniveralAccess: false,
},
);
return result.executionContextId;
}
async function readQuietProbe(
cdp: CdpRunner,
scope: DocumentSettleScope,
context: ProbeContext,
): Promise<QuietProbe | null> {
try {
if (scope.frameId && context.executionContextId === undefined) {
context.executionContextId = await executionContextForScope(cdp, scope);
if (context.executionContextId === undefined) return null;
}
const reply = await sendToCdpTarget<{ result?: { value?: unknown } }>(
cdp,
scope.target,
"Runtime.evaluate",
{
expression: QUIET_PROBE,
returnByValue: true,
...(context.executionContextId !== undefined
? { contextId: context.executionContextId }
: {}),
},
);
const value = reply.result?.value;
if (!value || typeof value !== "object") return null;
const { idleMs, readyState } = value as { idleMs?: unknown; readyState?: unknown };
if (typeof idleMs !== "number" || typeof readyState !== "string") return null;
return { idleMs, readyState };
} catch {
context.executionContextId = undefined;
return null;
}
}
export async function waitForDocumentSettled(
cdp: CdpRunner,
scope: DocumentSettleScope,
options: { signal?: AbortSignal } = {},
): Promise<SettleOutcome> {
const startedAt = Date.now();
const floor = startedAt + SETTLE_MIN_MS;
const deadline = startedAt + SETTLE_MAX_MS;
const context: ProbeContext = {};
for (;;) {
if (options.signal?.aborted) return "cancelled";
await sleep(SETTLE_POLL_MS, options.signal);
if (options.signal?.aborted) return "cancelled";
const probe = await readQuietProbe(cdp, scope, context);
const now = Date.now();
if (now < floor) continue;
if (now >= deadline) return "timeout";
if (!probe || probe.readyState === "loading") continue;
if (probe.idleMs >= SETTLE_QUIET_MS) return "quiet";
}
}
@@ -0,0 +1,34 @@
import type { RecordingDraftStep } from "./types";
const CLIPBOARD_KEYS = new Set(["a", "c", "v", "x", "A", "C", "V", "X"]);
const MODIFIER_ONLY_KEYS = new Set(["Meta", "Control", "Alt", "Shift", "OS", "Hyper", "Super"]);
export function shouldRecordPress(
key: string,
modifiers?: Array<"alt" | "ctrl" | "meta" | "shift">,
): boolean {
if (MODIFIER_ONLY_KEYS.has(key)) return false;
const mods = modifiers ?? [];
const hasCtrlOrMeta = mods.includes("ctrl") || mods.includes("meta");
if (hasCtrlOrMeta && CLIPBOARD_KEYS.has(key)) return false;
if (key === "Enter" || key === "Escape") return true;
if (key.length === 1 && !hasCtrlOrMeta && !mods.includes("alt")) return false;
return false;
}
export function shouldIncludeDraft(step: RecordingDraftStep): boolean {
return step.op !== "press" || shouldRecordPress(step.key, step.modifiers);
}
export function resolveDraftStartUrl(
drafts: RecordingDraftStep[],
explicitStartUrl?: string,
fallbackUrl?: string,
): string {
if (explicitStartUrl) return explicitStartUrl;
const navigation = drafts.find(
(step): step is Extract<RecordingDraftStep, { op: "navigate" }> => step.op === "navigate",
);
if (navigation) return navigation.url;
return drafts.find((step) => step.pageUrl)?.pageUrl ?? fallbackUrl ?? "about:blank";
}
@@ -0,0 +1,116 @@
import type { RenderedRef } from "@browser-skill/vom";
import {
type CaptureVomMatchNode,
type CaptureVomObservationResult,
captureVomObservation,
} from "@/tools/capture-vom-observation";
import type { CdpRunner, ChromeTabsApi } from "@/tools/shared";
export interface IndexedObservationNode {
frameId: string;
geometry: CaptureVomMatchNode;
ref?: RenderedRef;
}
export interface CapturedRecordingObservation {
rootFrameId: string;
index: ObservationNodeIndex;
url: string;
title?: string;
vomText: string;
truncated: boolean;
}
export interface RegisteredObservation {
stateId: string;
rootFrameId: string;
index: ObservationNodeIndex;
url: string;
}
function nodeKey(frameId: string, backendNodeId: number): string {
return `${frameId}:${backendNodeId}`;
}
function frameTagKey(frameId: string, tag: string): string {
return `${frameId}:${tag.toLowerCase()}`;
}
export class ObservationNodeIndex {
readonly #nodesByFrameTag = new Map<string, IndexedObservationNode[]>();
readonly #refById = new Map<string, RenderedRef>();
readonly #refsByFrame = new Map<string, RenderedRef[]>();
constructor(input: Pick<CaptureVomObservationResult, "rootFrameId" | "matchNodes" | "refs">) {
const refByNode = new Map<string, RenderedRef>();
for (const ref of input.refs) {
const frameId = ref.frameId ?? input.rootFrameId;
refByNode.set(nodeKey(frameId, ref.backendNodeId), ref);
this.#refById.set(ref.ref, ref);
const frameRefs = this.#refsByFrame.get(frameId) ?? [];
frameRefs.push(ref);
this.#refsByFrame.set(frameId, frameRefs);
}
for (const geometry of input.matchNodes) {
const { frameId } = geometry;
const entry = {
frameId,
geometry,
ref: refByNode.get(nodeKey(frameId, geometry.backendNodeId)),
};
const key = frameTagKey(frameId, geometry.tag);
const bucket = this.#nodesByFrameTag.get(key) ?? [];
bucket.push(entry);
this.#nodesByFrameTag.set(key, bucket);
}
}
candidates(frameId: string, tag: string): readonly IndexedObservationNode[] {
return this.#nodesByFrameTag.get(frameTagKey(frameId, tag)) ?? [];
}
ref(refId: string): RenderedRef | undefined {
return this.#refById.get(refId);
}
refs(frameId: string): readonly RenderedRef[] {
return this.#refsByFrame.get(frameId) ?? [];
}
}
async function readTabMeta(
tabsApi: ChromeTabsApi,
tabId: number,
): Promise<{ url: string; title?: string }> {
try {
const tab = await tabsApi.get(tabId);
return { url: tab.url ?? "about:blank", title: tab.title };
} catch {
return { url: "about:blank" };
}
}
export async function captureRecordingObservation(input: {
cdp: CdpRunner;
tabsApi: ChromeTabsApi;
tabId: number;
maxTokens: number;
redactValues: boolean;
signal?: AbortSignal;
}): Promise<CapturedRecordingObservation> {
const { url, title } = await readTabMeta(input.tabsApi, input.tabId);
const captured = await captureVomObservation(input.cdp, input.tabId, url, {
maxTokens: input.maxTokens,
redactValues: input.redactValues,
conditionalSurfaceProbe: false,
signal: input.signal,
});
return {
rootFrameId: captured.rootFrameId,
index: new ObservationNodeIndex(captured),
url,
title,
vomText: captured.text,
truncated: captured.truncated,
};
}
@@ -0,0 +1,122 @@
import type { CdpRunner, ChromeTabsApi } from "@/tools/shared";
import { captureRecordingObservation, type RegisteredObservation } from "./observation-capture";
import { RecordingStateRegistry } from "./state-registry";
import { matchObservationTarget, unmatchedTarget } from "./target-matcher";
import type { RecordingDraftStep, StepAnnotation, TargetedRecordingDraft } from "./types";
const DEFAULT_MAX_PAGE_TOKENS = 3_000;
const MIN_CAPTURE_INTERVAL_MS = 200;
export interface TabObservationCursor {
lastSettled: RegisteredObservation | null;
lastCaptureAt: number;
}
function abortableDelay(ms: number, signal?: AbortSignal): Promise<void> {
return new Promise((resolve) => {
if (signal?.aborted) return resolve();
const timer = setTimeout(resolve, ms);
signal?.addEventListener(
"abort",
() => {
clearTimeout(timer);
resolve();
},
{ once: true },
);
});
}
function isTargeted(draft: RecordingDraftStep): draft is TargetedRecordingDraft {
return draft.op !== "navigate" && draft.op !== "scroll";
}
export class RecordingObservationSession {
readonly registry: RecordingStateRegistry;
readonly cursor: TabObservationCursor;
readonly annotations: StepAnnotation[];
readonly #maxTokens: number;
readonly #redactValues: boolean;
constructor(
options: {
registry?: RecordingStateRegistry;
cursor?: TabObservationCursor;
annotations?: StepAnnotation[];
maxTokens?: number;
redactValues?: boolean;
} = {},
) {
this.registry = options.registry ?? new RecordingStateRegistry();
this.cursor = options.cursor ?? { lastSettled: null, lastCaptureAt: 0 };
this.annotations = options.annotations ?? [];
this.#maxTokens = options.maxTokens ?? DEFAULT_MAX_PAGE_TOKENS;
this.#redactValues = options.redactValues ?? false;
}
async capture(
cdp: CdpRunner,
tabsApi: ChromeTabsApi,
tabId: number,
signal?: AbortSignal,
): Promise<RegisteredObservation> {
const waitMs = Math.max(0, MIN_CAPTURE_INTERVAL_MS - (Date.now() - this.cursor.lastCaptureAt));
if (waitMs > 0) await abortableDelay(waitMs, signal);
if (signal?.aborted) throw new DOMException("observation aborted", "AbortError");
const captured = await captureRecordingObservation({
cdp,
tabsApi,
tabId,
maxTokens: this.#maxTokens,
redactValues: this.#redactValues,
signal,
});
const state = this.registry.register({
url: captured.url,
title: captured.title,
rawVomText: captured.vomText,
truncated: captured.truncated,
});
const observation: RegisteredObservation = {
stateId: state.id,
rootFrameId: captured.rootFrameId,
index: captured.index,
url: captured.url,
};
this.cursor.lastSettled = observation;
this.cursor.lastCaptureAt = Date.now();
return observation;
}
bindDraft(draft: RecordingDraftStep, draftId: number, previousActionPending = false): void {
const observation = this.cursor.lastSettled;
if (isTargeted(draft)) {
draft.matchedTarget = observation
? matchObservationTarget({
observation,
hint: draft.targetHint,
fallback: draft.captureTarget,
})
: unmatchedTarget(draft.captureTarget);
}
if (!observation) return;
const unmatched = isTargeted(draft) && draft.matchedTarget?.unmatched === true;
if (previousActionPending && unmatched) return;
draft.preStateId = observation.stateId;
this.registry.markStep(observation.stateId, draftId);
if (!isTargeted(draft) || !draft.matchedTarget?.ref) return;
const ref = observation.index.ref(draft.matchedTarget.ref);
if (!ref) return;
this.annotations.push({
draftId,
op: draft.op,
line: ref.line,
stateId: observation.stateId,
...(draft.op === "fill" && !this.#redactValues
? { detail: JSON.stringify(draft.value) }
: {}),
});
}
}
@@ -0,0 +1,232 @@
import type { CdpRunner, ChromeTabsApi } from "@/tools/shared";
import { type DocumentSettleScope, waitForDocumentSettled } from "./document-settle";
import { RecordingObservationSession } from "./observation-session";
import type { RecordingDraftStep } from "./types";
interface PendingSettle {
abort: AbortController;
scope: DocumentSettleScope;
}
interface PendingRedirect {
url: string;
abort: AbortController;
}
const CAPTURE_RETRY_DELAY_MS = 250;
function delay(ms: number, signal?: AbortSignal): Promise<void> {
return new Promise((resolve) => {
if (signal?.aborted) return resolve();
const timer = setTimeout(resolve, ms);
signal?.addEventListener(
"abort",
() => {
clearTimeout(timer);
resolve();
},
{ once: true },
);
});
}
function isAbortError(error: unknown): boolean {
return (
typeof error === "object" &&
error !== null &&
(error as { name?: string }).name === "AbortError"
);
}
export function inferMissingPostStates(drafts: RecordingDraftStep[]): void {
for (let index = 0; index < drafts.length - 1; index += 1) {
const draft = drafts[index];
const next = drafts[index + 1];
if (draft && next && !draft.postStateId && next.preStateId) {
draft.postStateId = next.preStateId;
}
}
}
export class SettleController {
readonly #session: RecordingObservationSession;
readonly #cdp: CdpRunner;
readonly #tabsApi: ChromeTabsApi;
readonly #tabId: number;
readonly #rootScope: DocumentSettleScope;
readonly #pending = new Map<number, PendingSettle>();
#queue = Promise.resolve();
#pendingRedirect: PendingRedirect | null = null;
#redirectQueue = Promise.resolve();
constructor(input: {
session: RecordingObservationSession;
cdp: CdpRunner;
tabsApi: ChromeTabsApi;
tabId: number;
}) {
this.#session = input.session;
this.#cdp = input.cdp;
this.#tabsApi = input.tabsApi;
this.#tabId = input.tabId;
this.#rootScope = { target: { tabId: input.tabId } };
}
get hasPending(): boolean {
return this.#pending.size > 0 || this.#pendingRedirect !== null;
}
async #captureWithRetry(signal?: AbortSignal) {
try {
return await this.#session.capture(this.#cdp, this.#tabsApi, this.#tabId, signal);
} catch (error) {
if (signal?.aborted) throw error;
await delay(CAPTURE_RETRY_DELAY_MS, signal);
return this.#session.capture(this.#cdp, this.#tabsApi, this.#tabId, signal);
}
}
schedule(
drafts: RecordingDraftStep[],
draftIndex: number,
scope: DocumentSettleScope = this.#rootScope,
): void {
if (scope.target.tabId !== this.#tabId) {
throw new Error(
`settle scope tab ${scope.target.tabId} does not belong to tab ${this.#tabId}`,
);
}
const landing = drafts[draftIndex]?.preStateId;
for (const [index, pending] of this.#pending) {
if (index >= draftIndex) continue;
pending.abort.abort();
this.#pending.delete(index);
if (landing && drafts[index] && !drafts[index].postStateId) {
drafts[index].postStateId = landing;
}
}
this.#pending.get(draftIndex)?.abort.abort();
const pending = { abort: new AbortController(), scope };
this.#pending.set(draftIndex, pending);
const task = async () => {
if (pending.abort.signal.aborted) return;
try {
const outcome = await waitForDocumentSettled(this.#cdp, pending.scope, {
signal: pending.abort.signal,
});
if (outcome === "cancelled") return;
const observation = await this.#captureWithRetry(pending.abort.signal);
if (!pending.abort.signal.aborted && drafts[draftIndex]) {
drafts[draftIndex].postStateId = observation.stateId;
}
} catch (error) {
if (!isAbortError(error)) {
console.warn(
`[bsk record] post-action observation failed for step ${draftIndex + 1}`,
error,
);
}
} finally {
if (this.#pending.get(draftIndex) === pending) this.#pending.delete(draftIndex);
}
};
this.#queue = this.#queue.then(task, task).catch(() => {});
}
cancel(): void {
for (const pending of this.#pending.values()) pending.abort.abort();
this.#pending.clear();
this.clearRedirect();
}
async flush(): Promise<void> {
for (let round = 0; round < 10; round += 1) {
const current = this.#queue;
await current;
if (current === this.#queue) return;
}
console.warn("[bsk record] settle queue kept growing; using completed observations");
}
async settleTrailing(drafts: RecordingDraftStep[]): Promise<void> {
const trailing: RecordingDraftStep[] = [];
for (let index = drafts.length - 1; index >= 0; index -= 1) {
const draft = drafts[index];
if (!draft || draft.postStateId) break;
trailing.push(draft);
}
if (trailing.length === 0) return;
try {
const observation = await this.#captureWithRetry();
for (const draft of trailing) draft.postStateId = observation.stateId;
} catch (error) {
console.warn("[bsk record] final observation at stop failed", error);
}
}
clearRedirect(): void {
this.#pendingRedirect?.abort.abort();
this.#pendingRedirect = null;
}
scheduleRedirect(drafts: RecordingDraftStep[], url: string): void {
this.#pendingRedirect?.abort.abort();
const pending: PendingRedirect = { url, abort: new AbortController() };
this.#pendingRedirect = pending;
const task = () => this.#settleRedirect(drafts, pending);
this.#redirectQueue = this.#redirectQueue.then(task, task).catch(() => {});
}
async #settleRedirect(drafts: RecordingDraftStep[], pending: PendingRedirect): Promise<void> {
const outcome = await waitForDocumentSettled(this.#cdp, this.#rootScope, {
signal: pending.abort.signal,
});
if (outcome === "cancelled" || this.#pendingRedirect !== pending) return;
let finalUrl = pending.url;
try {
finalUrl = (await this.#tabsApi.get(this.#tabId)).url || finalUrl;
} catch {
// The navigation event URL remains the best available destination.
}
if (this.#pendingRedirect !== pending) return;
this.#pendingRedirect = null;
if (
!finalUrl ||
finalUrl === "about:blank" ||
this.#session.cursor.lastSettled?.url === finalUrl
) {
return;
}
const last = drafts[drafts.length - 1];
if (last?.op === "navigate" && last.url === finalUrl) {
if (!last.postStateId) this.schedule(drafts, drafts.length - 1);
await this.flush();
return;
}
const draft: RecordingDraftStep = {
op: "navigate",
url: finalUrl,
pageUrl: finalUrl,
cause: "browser",
preStateId: this.#session.cursor.lastSettled?.stateId,
};
drafts.push(draft);
const draftIndex = drafts.length - 1;
if (draft.preStateId) this.#session.registry.markStep(draft.preStateId, draftIndex + 1);
this.schedule(drafts, draftIndex);
await this.flush();
}
async flushRedirects(): Promise<void> {
for (let round = 0; round < 10; round += 1) {
const current = this.#redirectQueue;
await current;
if (current === this.#redirectQueue) return;
}
console.warn("[bsk record] redirect queue kept growing; using the final completed landing");
}
}
@@ -0,0 +1,56 @@
export interface RecordedStateEntry {
id: string;
url: string;
title?: string;
rawVomText: string;
truncated: boolean;
stepsHere: number[];
}
function stateIdentity(url: string, body: string): string {
return `${url}\0${body}`;
}
export class RecordingStateRegistry {
readonly #entriesById = new Map<string, RecordedStateEntry>();
readonly #idByIdentity = new Map<string, string>();
#nextId = 1;
register(input: {
url: string;
title?: string;
rawVomText: string;
truncated?: boolean;
}): RecordedStateEntry {
const identity = stateIdentity(input.url, input.rawVomText);
const existingId = this.#idByIdentity.get(identity);
if (existingId) {
const existing = this.#entriesById.get(existingId)!;
if (!existing.title && input.title) existing.title = input.title;
if (input.truncated) existing.truncated = true;
return existing;
}
const entry: RecordedStateEntry = {
id: `s${this.#nextId}`,
url: input.url,
...(input.title ? { title: input.title } : {}),
rawVomText: input.rawVomText,
truncated: input.truncated ?? false,
stepsHere: [],
};
this.#nextId += 1;
this.#entriesById.set(entry.id, entry);
this.#idByIdentity.set(identity, entry.id);
return entry;
}
values(): RecordedStateEntry[] {
return [...this.#entriesById.values()];
}
markStep(stateId: string, draftId: number): void {
const entry = this.#entriesById.get(stateId);
if (entry && !entry.stepsHere.includes(draftId)) entry.stepsHere.push(draftId);
}
}
@@ -1,8 +1,8 @@
import type { DraftTraceStep } from "@/transport/types";
import type { RecordStepPayload } from "./record-bridge";
import type { RecordStepPayload } from "../record-bridge";
import type { RecordingDraftStep } from "./types";
export interface RecordingStepBuffer {
steps: DraftTraceStep[];
steps: RecordingDraftStep[];
currentUrl?: string;
pendingNavigation: boolean;
pendingNavigationDeadline?: number;
@@ -10,33 +10,33 @@ export interface RecordingStepBuffer {
const NAVIGATION_TRIGGER_WINDOW_MS = 3_000;
function toDraftStep(payload: RecordStepPayload): DraftTraceStep | null {
function toDraftStep(payload: RecordStepPayload): RecordingDraftStep | null {
const pageUrl = payload.page_url;
switch (payload.op) {
case "click":
return payload.target
? {
op: "click",
target: payload.target,
...(pageUrl ? { page_url: pageUrl } : {}),
captureTarget: payload.target,
...(pageUrl ? { pageUrl } : {}),
}
: null;
case "hover":
return payload.target
? {
op: "hover",
target: payload.target,
...(pageUrl ? { page_url: pageUrl } : {}),
captureTarget: payload.target,
...(pageUrl ? { pageUrl } : {}),
}
: null;
case "fill":
return payload.target
? {
op: "fill",
target: payload.target,
captureTarget: payload.target,
value: payload.value ?? "",
...(payload.redacted ? { redacted: true } : {}),
...(pageUrl ? { page_url: pageUrl } : {}),
...(pageUrl ? { pageUrl } : {}),
}
: null;
case "press":
@@ -44,19 +44,19 @@ function toDraftStep(payload: RecordStepPayload): DraftTraceStep | null {
? {
op: "press",
key: payload.key,
...(payload.target ? { target: payload.target } : {}),
...(payload.target ? { captureTarget: payload.target } : {}),
...(payload.modifiers?.length ? { modifiers: payload.modifiers } : {}),
...(pageUrl ? { page_url: pageUrl } : {}),
...(pageUrl ? { pageUrl } : {}),
}
: null;
case "select":
return payload.target && payload.values
? {
op: "select",
target: payload.target,
captureTarget: payload.target,
values: payload.values,
...(payload.labels?.length ? { labels: payload.labels } : {}),
...(pageUrl ? { page_url: pageUrl } : {}),
...(pageUrl ? { pageUrl } : {}),
}
: null;
case "navigate":
@@ -69,7 +69,7 @@ function annotateLastStepNavigation(buffer: RecordingStepBuffer, url: string): b
const step = buffer.steps[i];
if (!step) continue;
if (step.op === "click" || step.op === "press" || step.op === "select") {
buffer.steps[i] = { ...step, navigated_to: url };
buffer.steps[i] = { ...step, navigatedTo: url };
return true;
}
break;
@@ -96,7 +96,7 @@ export function observeRecordedNavigation(
buffer.steps.push({
op: "navigate",
url,
page_url: url,
pageUrl: url,
});
}
return;
@@ -107,7 +107,7 @@ export function observeRecordedNavigation(
buffer.steps.push({
op: "navigate",
url,
page_url: url,
pageUrl: url,
});
}
@@ -0,0 +1,76 @@
import type { RenderedRef } from "@browser-skill/vom";
import type { TargetDescriptorV3 } from "@/transport/types";
import type { CaptureTargetDescriptor } from "../describe-target";
import type { IndexedObservationNode, RegisteredObservation } from "./observation-capture";
import type { TargetGeometry, TargetMatchHint } from "./types";
const MATCH_TOLERANCE_PX = 2;
function close(a: number, b: number): boolean {
return Math.abs(a - b) <= MATCH_TOLERANCE_PX;
}
function rectMatches(a: TargetGeometry["rect"], b: TargetGeometry["rect"]): boolean {
return close(a.x, b.x) && close(a.y, b.y) && close(a.w, b.w) && close(a.h, b.h);
}
function candidateMatches(candidate: IndexedObservationNode, geometry: TargetGeometry): boolean {
return candidate.geometry.rect !== null && rectMatches(geometry.rect, candidate.geometry.rect);
}
export function unmatchedTarget(fallback?: CaptureTargetDescriptor): TargetDescriptorV3 {
return {
...(fallback?.role ? { role: fallback.role } : {}),
...(fallback?.name ? { name: fallback.name } : {}),
unmatched: true,
};
}
function normalized(value: string | undefined): string {
return value?.replace(/\s+/g, " ").trim().toLowerCase() ?? "";
}
function matchesSemantics(ref: RenderedRef, fallback?: CaptureTargetDescriptor): boolean {
const role = normalized(fallback?.role);
const name = normalized(fallback?.name);
if (!role && !name) return false;
if (role && normalized(ref.role) !== role) return false;
if (name && normalized(ref.name) !== name) return false;
return true;
}
function descriptor(ref: RenderedRef): TargetDescriptorV3 {
return {
ref: ref.ref,
...(ref.role ? { role: ref.role } : {}),
...(ref.name ? { name: ref.name } : {}),
...(ref.ctx ? { ctx: ref.ctx } : {}),
};
}
export function matchObservationTarget(input: {
observation: RegisteredObservation;
hint?: TargetMatchHint;
fallback?: CaptureTargetDescriptor;
}): TargetDescriptorV3 {
const frameId = input.hint?.frameId ?? input.observation.rootFrameId;
const geometry = input.hint?.geometry;
if (geometry) {
const matches = input.observation.index
.candidates(frameId, geometry.tag)
.filter((candidate) => candidate.ref && candidateMatches(candidate, geometry));
if (matches.length === 1) return descriptor(matches[0]!.ref!);
const semanticMatches = matches.filter(
(candidate) => candidate.ref && matchesSemantics(candidate.ref, input.fallback),
);
if (semanticMatches.length === 1) return descriptor(semanticMatches[0]!.ref!);
return unmatchedTarget(input.fallback);
}
const semanticMatches = input.observation.index
.refs(frameId)
.filter((ref) => matchesSemantics(ref, input.fallback));
return semanticMatches.length === 1
? descriptor(semanticMatches[0]!)
: unmatchedTarget(input.fallback);
}
@@ -0,0 +1,87 @@
import {
type StepV3,
type StopReason,
TRACE_VERSION_V3,
type TraceStateV3,
type TraceV3,
VOM_FORMAT_VERSION,
} from "@/transport/types";
import { resolveDraftStartUrl } from "./draft-policy";
import type { RecordedStateEntry, RecordingStateRegistry } from "./state-registry";
import { reduceTraceStepsV3 } from "./trace-reducer-v3";
import { formatTraceStateBody } from "./trace-state-body";
import type { RecordingDraftStep, StepAnnotation } from "./types";
function publishedEntries(registry: RecordingStateRegistry, steps: StepV3[]): RecordedStateEntry[] {
const entries = registry.values();
if (steps.length === 0) return entries.slice(0, 1);
const referenced = new Set(steps.flatMap((step) => [step.state, step.result.state]));
return entries.filter((entry) => referenced.has(entry.id));
}
function remapDraftIds(draftIds: number[], stepIdByDraftId: Map<number, number>): number[] {
return [
...new Set(
draftIds.flatMap((id) => {
const stepId = stepIdByDraftId.get(id);
return stepId === undefined ? [] : [stepId];
}),
),
].sort((a, b) => a - b);
}
export function buildTraceV3(input: {
registry: RecordingStateRegistry;
drafts: RecordingDraftStep[];
annotations?: StepAnnotation[];
startedAt: string;
purpose?: string;
startUrl?: string;
stoppedBy: StopReason;
bskVersion: string;
}): TraceV3 {
const reduced = reduceTraceStepsV3(input.drafts);
const entries = publishedEntries(input.registry, reduced.steps);
const publishedId = new Map(entries.map((entry, index) => [entry.id, `s${index + 1}`]));
const annotationsByState = new Map<string, StepAnnotation[]>();
for (const annotation of input.annotations ?? []) {
const bucket = annotationsByState.get(annotation.stateId) ?? [];
bucket.push(annotation);
annotationsByState.set(annotation.stateId, bucket);
}
const steps = reduced.steps.map((step) => ({
...step,
state: publishedId.get(step.state) ?? step.state,
result: { state: publishedId.get(step.result.state) ?? step.result.state },
}));
const states: TraceStateV3[] = entries.map((entry) => {
const id = publishedId.get(entry.id) ?? entry.id;
return {
id,
url: entry.url,
...(entry.title ? { title: entry.title } : {}),
body: formatTraceStateBody({
stateId: id,
url: entry.url,
title: entry.title,
stepIds: remapDraftIds(entry.stepsHere, reduced.stepIdByDraftId),
vomText: entry.rawVomText,
annotations: annotationsByState.get(entry.id) ?? [],
stepIdByDraftId: reduced.stepIdByDraftId,
}),
...(entry.truncated ? { truncated: true } : {}),
};
});
return {
version: TRACE_VERSION_V3,
...(input.purpose ? { purpose: input.purpose } : {}),
recorded_at: new Date().toISOString(),
started_at: input.startedAt,
stopped_by: input.stoppedBy,
entry: { start_url: resolveDraftStartUrl(input.drafts, input.startUrl, states[0]?.url) },
recorder: { bsk: input.bskVersion, vom: VOM_FORMAT_VERSION },
states,
steps,
};
}
@@ -1,30 +1,10 @@
import type { DraftTraceStep, PageRef, SelectedOption, Step } from "@/transport/types";
const CLIPBOARD_KEYS = new Set(["a", "c", "v", "x", "A", "C", "V", "X"]);
const MODIFIER_ONLY_KEYS = new Set(["Meta", "Control", "Alt", "Shift", "OS", "Hyper", "Super"]);
export function shouldRecordPress(
key: string,
modifiers?: Array<"alt" | "ctrl" | "meta" | "shift">,
): boolean {
if (MODIFIER_ONLY_KEYS.has(key)) return false;
const mods = modifiers ?? [];
const hasCtrlOrMeta = mods.includes("ctrl") || mods.includes("meta");
if (hasCtrlOrMeta && CLIPBOARD_KEYS.has(key)) return false;
if (key === "Enter" || key === "Escape") return true;
// Drop bare character typing — FillSession already records the value.
if (key.length === 1 && !hasCtrlOrMeta && !mods.includes("alt")) return false;
return false;
}
function shouldIncludeDraft(step: DraftTraceStep): boolean {
if (step.op === "press" && !shouldRecordPress(step.key, step.modifiers)) return false;
return true;
}
import type { PageRefV2, SelectedOptionV2, StepV2, TraceV2 } from "@/transport/types";
import { resolveDraftStartUrl, shouldIncludeDraft } from "./draft-policy";
import type { RecordingDraftStep } from "./types";
/** Collapse consecutive navigations to the last hop. */
function collapseNavigations(steps: DraftTraceStep[]): DraftTraceStep[] {
const out: DraftTraceStep[] = [];
function collapseNavigations(steps: RecordingDraftStep[]): RecordingDraftStep[] {
const out: RecordingDraftStep[] = [];
for (const step of steps) {
const prev = out[out.length - 1];
if (step.op === "navigate" && prev?.op === "navigate") {
@@ -36,7 +16,7 @@ function collapseNavigations(steps: DraftTraceStep[]): DraftTraceStep[] {
return out;
}
function collectUrls(steps: DraftTraceStep[], startUrl?: string): string[] {
function collectUrls(steps: RecordingDraftStep[], startUrl?: string): string[] {
const urls: string[] = [];
if (startUrl) urls.push(startUrl);
for (const step of steps) {
@@ -44,8 +24,8 @@ function collectUrls(steps: DraftTraceStep[], startUrl?: string): string[] {
urls.push(step.url);
continue;
}
if ("page_url" in step && step.page_url) urls.push(step.page_url);
if ("navigated_to" in step && step.navigated_to) urls.push(step.navigated_to);
if (step.pageUrl) urls.push(step.pageUrl);
if ("navigatedTo" in step && step.navigatedTo) urls.push(step.navigatedTo);
}
const seen = new Set<string>();
const unique: string[] = [];
@@ -58,9 +38,9 @@ function collectUrls(steps: DraftTraceStep[], startUrl?: string): string[] {
}
function buildPageRegistry(
steps: DraftTraceStep[],
steps: RecordingDraftStep[],
startUrl?: string,
): { pages: PageRef[]; urlToId: Map<string, string> } {
): { pages: PageRefV2[]; urlToId: Map<string, string> } {
const urls = collectUrls(steps, startUrl);
const urlToId = new Map<string, string>();
const pages = urls.map((url, index) => {
@@ -81,28 +61,28 @@ function pageIdFor(
return urlToId.values().next().value ?? "p1";
}
function pageUrlForDraft(step: DraftTraceStep, fallbackUrl?: string): string | undefined {
if (step.op === "navigate") return step.page_url ?? step.url;
if ("page_url" in step && step.page_url) return step.page_url;
function pageUrlForDraft(step: RecordingDraftStep, fallbackUrl?: string): string | undefined {
if (step.op === "navigate") return step.pageUrl ?? step.url;
if (step.pageUrl) return step.pageUrl;
return fallbackUrl;
}
function effectForNavigation(
navigatedTo: string | undefined,
urlToId: Map<string, string>,
): Step["effect"] {
): StepV2["effect"] {
if (!navigatedTo) return undefined;
const pageId = urlToId.get(navigatedTo);
if (!pageId) return undefined;
return { navigated_to: pageId };
}
function withEffect(step: Step, effect: Step["effect"]): Step {
function withEffect(step: StepV2, effect: StepV2["effect"]): StepV2 {
if (!effect) return step;
return { ...step, effect };
}
function toSelection(values: string[], labels?: string[]): SelectedOption[] {
function toSelection(values: string[], labels?: string[]): SelectedOptionV2[] {
return values.map((value, index) => ({
value,
...(labels?.[index] ? { label: labels[index] } : {}),
@@ -110,11 +90,11 @@ function toSelection(values: string[], labels?: string[]): SelectedOption[] {
}
function toV2Step(
step: DraftTraceStep,
step: RecordingDraftStep,
id: number,
urlToId: Map<string, string>,
fallbackUrl?: string,
): Step | null {
): StepV2 | null {
if (!shouldIncludeDraft(step)) return null;
const pageUrl = pageUrlForDraft(step, fallbackUrl);
@@ -129,28 +109,31 @@ function toV2Step(
to: step.url,
};
case "click":
if (!step.captureTarget) return null;
return withEffect(
{
op: "click",
id,
page,
target: step.target,
target: step.captureTarget,
},
effectForNavigation(step.navigated_to, urlToId),
effectForNavigation(step.navigatedTo, urlToId),
);
case "hover":
if (!step.captureTarget) return null;
return {
op: "hover",
id,
page,
target: step.target,
target: step.captureTarget,
};
case "fill":
if (!step.captureTarget) return null;
return {
op: "fill",
id,
page,
target: step.target,
target: step.captureTarget,
value: step.value,
...(step.redacted ? { redacted: true } : {}),
};
@@ -161,44 +144,47 @@ function toV2Step(
id,
page,
key: step.key,
...(step.target ? { target: step.target } : {}),
...(step.captureTarget ? { target: step.captureTarget } : {}),
...(step.modifiers?.length ? { modifiers: step.modifiers } : {}),
},
effectForNavigation(step.navigated_to, urlToId),
effectForNavigation(step.navigatedTo, urlToId),
);
case "select":
if (!step.captureTarget) return null;
return withEffect(
{
op: "select",
id,
page,
target: step.target,
target: step.captureTarget,
selection: toSelection(step.values, step.labels),
},
effectForNavigation(step.navigated_to, urlToId),
effectForNavigation(step.navigatedTo, urlToId),
);
case "scroll":
return null;
}
}
export interface ReducedTrace {
pages: PageRef[];
steps: Step[];
interface ReducedTrace {
pages: PageRefV2[];
steps: StepV2[];
}
/**
* Compile capture drafts into record-only trace v2 steps.
* Variable inputs are NOT classified here — executing agents infer that at run time.
*/
export function reduceTraceSteps(steps: DraftTraceStep[], startUrl?: string): ReducedTrace {
function reduceTraceSteps(steps: RecordingDraftStep[], startUrl?: string): ReducedTrace {
const collapsed = collapseNavigations(steps);
const { pages, urlToId } = buildPageRegistry(collapsed, startUrl);
const out: Step[] = [];
const out: StepV2[] = [];
let id = 1;
let lastUrl = startUrl;
for (const draft of collapsed) {
if (draft.op === "navigate") lastUrl = draft.url;
else if ("navigated_to" in draft && draft.navigated_to) lastUrl = draft.navigated_to;
else if ("page_url" in draft && draft.page_url) lastUrl = draft.page_url;
else if ("navigatedTo" in draft && draft.navigatedTo) lastUrl = draft.navigatedTo;
else if (draft.pageUrl) lastUrl = draft.pageUrl;
const step = toV2Step(draft, id, urlToId, lastUrl);
if (!step) continue;
out.push(step);
@@ -207,18 +193,27 @@ export function reduceTraceSteps(steps: DraftTraceStep[], startUrl?: string): Re
return { pages, steps: out };
}
export function resolveTraceStartUrl(
drafts: DraftTraceStep[],
function resolveTraceStartUrl(
drafts: RecordingDraftStep[],
startUrl?: string,
pages?: PageRef[],
pages?: PageRefV2[],
): string {
if (startUrl) return startUrl;
const navigate = drafts.find((step): step is Extract<DraftTraceStep, { op: "navigate" }> => {
return step.op === "navigate";
});
if (navigate) return navigate.url;
for (const draft of drafts) {
if ("page_url" in draft && draft.page_url) return draft.page_url;
}
return pages?.[0]?.url ?? "about:blank";
return resolveDraftStartUrl(drafts, startUrl, pages?.[0]?.url);
}
export function buildTraceV2(input: {
steps: RecordingDraftStep[];
startedAt: string;
startUrl?: string;
purpose?: string;
}): TraceV2 {
const { pages, steps } = reduceTraceSteps(input.steps, input.startUrl);
return {
recorded_at: new Date().toISOString(),
started_at: input.startedAt,
...(input.purpose ? { purpose: input.purpose } : {}),
entry: { start_url: resolveTraceStartUrl(input.steps, input.startUrl, pages) },
pages,
steps,
};
}
@@ -0,0 +1,129 @@
import type { NavigationCause, StepV3 } from "@/transport/types";
import { shouldIncludeDraft } from "./draft-policy";
import { unmatchedTarget } from "./target-matcher";
import type { RecordingDraftStep } from "./types";
interface CollapsedDraft {
draft: RecordingDraftStep;
draftIds: number[];
}
const REDIRECT_QUALIFIERS = new Set(["client_redirect", "server_redirect"]);
const TRANSITION_CAUSES: Record<string, NavigationCause> = {
typed: "user_typed",
generated: "user_typed",
keyword: "user_typed",
keyword_generated: "user_typed",
link: "link",
form_submit: "form_submit",
reload: "reload",
auto_bookmark: "browser",
start_page: "browser",
};
function isRedirect(step: Extract<RecordingDraftStep, { op: "navigate" }>): boolean {
return (step.transitionQualifiers ?? []).some((qualifier) => REDIRECT_QUALIFIERS.has(qualifier));
}
function collapseRedirects(steps: RecordingDraftStep[]): CollapsedDraft[] {
const output: CollapsedDraft[] = [];
steps.forEach((step, index) => {
const previous = output[output.length - 1];
if (step.op === "navigate" && previous?.draft.op === "navigate" && isRedirect(step)) {
previous.draft = {
...previous.draft,
url: step.url,
postStateId: step.postStateId ?? previous.draft.postStateId,
};
previous.draftIds.push(index + 1);
return;
}
output.push({ draft: { ...step }, draftIds: [index + 1] });
});
return output;
}
function navigationCause(step: Extract<RecordingDraftStep, { op: "navigate" }>): NavigationCause {
if (step.cause) return step.cause;
const qualifiers = step.transitionQualifiers ?? [];
if (qualifiers.includes("forward_back")) return "history";
if (qualifiers.includes("from_address_bar")) return "user_typed";
return TRANSITION_CAUSES[step.transitionType ?? ""] ?? "browser";
}
function selection(values: string[], labels?: string[]): Array<{ value: string; label?: string }> {
return values.map((value, index) => ({
value,
...(labels?.[index] ? { label: labels[index] } : {}),
}));
}
function reduceDraft(draft: RecordingDraftStep, id: number): StepV3 | null {
if (!shouldIncludeDraft(draft)) return null;
const state = draft.preStateId ?? draft.postStateId;
const resultState = draft.postStateId ?? draft.preStateId;
if (!state || !resultState) return null;
const common = { id, state, result: { state: resultState } };
switch (draft.op) {
case "navigate":
return { op: "navigate", ...common, to: draft.url, cause: navigationCause(draft) };
case "click":
return {
op: "click",
...common,
target: draft.matchedTarget ?? unmatchedTarget(draft.captureTarget),
};
case "hover":
return {
op: "hover",
...common,
target: draft.matchedTarget ?? unmatchedTarget(draft.captureTarget),
};
case "fill":
return {
op: "fill",
...common,
target: draft.matchedTarget ?? unmatchedTarget(draft.captureTarget),
value: draft.value,
commit: draft.commit ?? "blur",
...(draft.redacted ? { redacted: true } : {}),
};
case "press":
return {
op: "press",
...common,
key: draft.key,
...(draft.captureTarget || draft.matchedTarget
? { target: draft.matchedTarget ?? unmatchedTarget(draft.captureTarget) }
: {}),
...(draft.modifiers?.length ? { modifiers: draft.modifiers } : {}),
};
case "select":
return {
op: "select",
...common,
target: draft.matchedTarget ?? unmatchedTarget(draft.captureTarget),
selection: selection(draft.values, draft.labels),
};
case "scroll":
return { op: "scroll", ...common };
}
}
export interface ReducedTraceV3 {
steps: StepV3[];
stepIdByDraftId: Map<number, number>;
}
export function reduceTraceStepsV3(steps: RecordingDraftStep[]): ReducedTraceV3 {
const output: StepV3[] = [];
const stepIdByDraftId = new Map<number, number>();
for (const { draft, draftIds } of collapseRedirects(steps)) {
const step = reduceDraft(draft, output.length + 1);
if (!step) continue;
output.push(step);
for (const draftId of draftIds) stepIdByDraftId.set(draftId, step.id);
}
return { steps: output, stepIdByDraftId };
}
@@ -0,0 +1,39 @@
import type { StepAnnotation } from "./types";
function annotationText(annotation: StepAnnotation, stepId: number): string {
return ` ⟵ step ${stepId}: ${annotation.op}${annotation.detail ? `: ${annotation.detail}` : ""}`;
}
export function formatTraceStateBody(input: {
stateId: string;
url: string;
title?: string;
stepIds: number[];
vomText: string;
annotations: StepAnnotation[];
stepIdByDraftId: Map<number, number>;
}): string {
const lines = ["# bsk-observation 1", `state: ${JSON.stringify(input.stateId)}`];
lines.push(`url: ${JSON.stringify(input.url)}`);
if (input.title) lines.push(`title: ${JSON.stringify(input.title)}`);
if (input.stepIds.length > 0) lines.push(`steps_here: [${input.stepIds.join(", ")}]`);
lines.push("---");
const byLine = new Map<number, Array<{ annotation: StepAnnotation; stepId: number }>>();
for (const annotation of input.annotations) {
const stepId = input.stepIdByDraftId.get(annotation.draftId);
if (stepId === undefined) continue;
const bucket = byLine.get(annotation.line) ?? [];
bucket.push({ annotation, stepId });
byLine.set(annotation.line, bucket);
}
input.vomText.split("\n").forEach((bodyLine, lineIndex) => {
let line = bodyLine;
for (const item of byLine.get(lineIndex) ?? []) {
line += annotationText(item.annotation, item.stepId);
}
lines.push(line);
});
return `${lines.join("\n")}\n`;
}
+83
View File
@@ -0,0 +1,83 @@
import type { CaptureTargetDescriptor } from "@/lib/describe-target";
import type {
FillCommit,
KeyModifier,
NavigationCause,
StepV3,
TargetDescriptorV3,
} from "@/transport/types";
export interface TargetGeometry {
/** Top-level viewport-relative CSS pixels, as defined by the geometry module. */
rect: { x: number; y: number; w: number; h: number };
tag: string;
}
export interface TargetMatchHint {
geometry?: TargetGeometry;
/** Missing means the current top frame, never an unrestricted frame search. */
frameId?: string;
}
export interface StepAnnotation {
draftId: number;
op: StepV3["op"];
line: number;
stateId: string;
detail?: string;
}
interface DraftStateLink {
pageUrl?: string;
preStateId?: string;
postStateId?: string;
}
interface DraftTarget {
captureTarget?: CaptureTargetDescriptor;
targetHint?: TargetMatchHint;
matchedTarget?: TargetDescriptorV3;
}
interface DraftNavigationEffect {
navigatedTo?: string;
}
export type RecordingDraftStep =
| ({ op: "click" } & DraftStateLink & DraftTarget & DraftNavigationEffect)
| ({ op: "hover" } & DraftStateLink & DraftTarget)
| ({
op: "fill";
value: string;
commit?: FillCommit;
redacted?: boolean;
} & DraftStateLink &
DraftTarget &
DraftNavigationEffect)
| ({
op: "press";
key: string;
modifiers?: KeyModifier[];
} & DraftStateLink &
DraftTarget &
DraftNavigationEffect)
| ({
op: "select";
values: string[];
labels?: string[];
} & DraftStateLink &
DraftTarget &
DraftNavigationEffect)
| ({ op: "scroll" } & DraftStateLink)
| ({
op: "navigate";
url: string;
cause?: NavigationCause;
transitionType?: string;
transitionQualifiers?: string[];
} & DraftStateLink);
export type TargetedRecordingDraft = Extract<
RecordingDraftStep,
{ op: "click" | "hover" | "fill" | "press" | "select" }
>;
@@ -33,4 +33,19 @@ describe("RefStore", () => {
expect(s.resolve("@e11")).toBe(11);
expect(s.size()).toBe(2);
});
it("preserves the child CDP session as part of ref identity", () => {
const s = new RefStore();
s.set("e1", 42, {
tabId: 7,
frameId: "child-frame",
cdpSessionId: "child-session",
});
expect(s.resolveEntry("e1")).toMatchObject({
backendNodeId: 42,
tabId: 7,
frameId: "child-frame",
cdpSessionId: "child-session",
});
});
});
@@ -1,9 +1,10 @@
/**
* Per-session map from `@e<N>` snapshot refs to CDP `backendNodeId`.
* Per-session map from `@e<N>` snapshot refs to a CDP node address.
*
* Each fresh `tool.snapshot` resets the store: M6 will call
* `replace(...)` with the new ref → backendNodeId pairs. Tools like
* `tool.click` consume the store via `resolve("@e1")`.
* `replace(...)` with the new ref → node address pairs. A node address
* includes the owning flat CDP session/frame when the element lives in
* an OOPIF; tools resolve live geometry from that identity at call time.
*
* Refs are session-scoped (§7): looking up a ref in the wrong session
* returns `null`, never silently leaks. Storing values in different
@@ -15,10 +16,19 @@ export type BackendNodeId = number;
export interface RefEntry {
backendNodeId: BackendNodeId;
tabId: number | null;
frameId?: string;
cdpSessionId?: string;
generation: number;
}
type RefInput = BackendNodeId | { backendNodeId: BackendNodeId; tabId: number };
export type RefInput =
| BackendNodeId
| {
backendNodeId: BackendNodeId;
tabId: number;
frameId?: string;
cdpSessionId?: string;
};
export class RefStore {
private readonly map = new Map<string, RefEntry>();
@@ -44,7 +54,7 @@ export class RefStore {
}
/**
* Replace the entire store with a new ref → backendNodeId mapping.
* Replace the entire store with a new ref → CDP node identity mapping.
* Used after every fresh `tool.snapshot`.
*/
replace(entries: Iterable<readonly [string, RefInput]>): void {
@@ -53,10 +63,20 @@ export class RefStore {
for (const [ref, input] of entries) this.map.set(normaliseRef(ref), this.entry(input));
}
set(ref: string, id: BackendNodeId, opts: { tabId?: number } = {}): void {
set(
ref: string,
id: BackendNodeId,
opts: {
tabId?: number;
frameId?: string;
cdpSessionId?: string;
} = {},
): void {
this.map.set(normaliseRef(ref), {
backendNodeId: id,
tabId: opts.tabId ?? null,
...(opts.frameId ? { frameId: opts.frameId } : {}),
...(opts.cdpSessionId ? { cdpSessionId: opts.cdpSessionId } : {}),
generation: this.generation,
});
}
@@ -80,6 +100,8 @@ export class RefStore {
return {
backendNodeId: input.backendNodeId,
tabId: input.tabId,
...(input.frameId ? { frameId: input.frameId } : {}),
...(input.cdpSessionId ? { cdpSessionId: input.cdpSessionId } : {}),
generation: this.generation,
};
}
@@ -612,4 +612,14 @@ describe("isInjectableContentScriptUrl", () => {
).toBe(false);
expect(isInjectableContentScriptUrl("https://chromewebstore.google.com/")).toBe(false);
});
it("rejects the Edge Add-ons store", () => {
expect(
isInjectableContentScriptUrl(
"https://microsoftedge.microsoft.com/addons/detail/browserskill/emacgiaaaiojkkpkddmmdfhmokgmnikg",
),
).toBe(false);
// Non-store paths on the same host stay injectable.
expect(isInjectableContentScriptUrl("https://microsoftedge.microsoft.com/")).toBe(true);
});
});
@@ -203,6 +203,55 @@ describe("ToolDispatcher", () => {
expect(sent[0]).toEqual({ id: "r-1", result: {} });
});
it("forwards daemon cancellation into an in-flight session stop", async () => {
const { transport, sent, deliver } = fakeTransport();
const remove = vi.fn(async () => {});
const sessions = new SessionManager({
agentWindow: {
create: vi.fn(async () => 4242),
remove,
ensureActiveTab: vi.fn(async () => {}),
},
});
await sessions.start("aa11");
let finishDetach: () => void = () => {};
const detach = new Promise<void>((resolve) => {
finishDetach = resolve;
});
const cdp = {
send: vi.fn(),
detachSession: vi.fn(() => detach),
ensureNetworkCapture: vi.fn(async () => {}),
networkEntriesSince: vi.fn(() => ({
tab_id: 7,
entries: [],
next_since: 0,
truncated: false,
})),
setDeviceMetricsOverride: vi.fn(async () => {}),
clearDeviceMetricsOverride: vi.fn(async () => {}),
setUserAgentOverride: vi.fn(async () => {}),
setTouchEmulationEnabled: vi.fn(async () => {}),
};
const dispatcher = new ToolDispatcher({ transport, sessions, cdp: cdp as TestDispatcherCdp });
dispatcher.start();
deliver(makeRequest("tool.session_stop", { session_id: "aa11" }));
await vi.waitFor(() => expect(cdp.detachSession).toHaveBeenCalledWith("aa11"));
deliver({ id: "cancel-stop", method: "cancel", params: { rpc_id: "r-1" } });
await flushMicrotasks();
finishDetach();
await flushMicrotasks();
expect(sent).toContainEqual({ id: "cancel-stop", result: { cancelled: true } });
expect(sent).toContainEqual({
id: "r-1",
error: expect.objectContaining({ code: "cancelled" }),
});
expect(sessions.has("aa11")).toBe(true);
expect(remove).not.toHaveBeenCalled();
});
it("returns not_found when stopping an unknown session", async () => {
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
@@ -1,24 +1,54 @@
import { describe, expect, it } from "vitest";
import { nodeBoundingRect, quadBoundingRect } from "../element-geometry";
import { nodeContentRegion } from "../element-geometry";
import type { CdpRunner } from "../shared";
describe("quadBoundingRect", () => {
it("computes axis-aligned bounds from an 8-double quad", () => {
expect(quadBoundingRect([10, 20, 110, 20, 110, 60, 10, 60])).toEqual({
x: 10,
y: 20,
width: 100,
height: 40,
describe("nodeContentRegion", () => {
it("rejects invalid content quads before using the box model fallback", async () => {
const send = async (_tabId: number, method: string) => {
if (method === "DOM.getContentQuads") return { quads: [[0, 0, 0, 0, 0, 0, 0, 0]] };
if (method === "DOM.getBoxModel") {
return { model: { content: [5, 10, 25, 10, 25, 30, 5, 30] } };
}
throw new Error(`unexpected CDP call ${method}`);
};
await expect(nodeContentRegion({ send: send as CdpRunner["send"] }, 7, 555)).resolves.toEqual([
[
{ x: 5, y: 10 },
{ x: 25, y: 10 },
{ x: 25, y: 30 },
{ x: 5, y: 30 },
],
]);
});
it("preserves every visible content fragment", async () => {
const send = async () => ({
quads: [
[10, 20, 60, 20, 60, 40, 10, 40],
[15, 50, 90, 50, 90, 80, 15, 80],
],
});
const cdp: CdpRunner = {
send: send as CdpRunner["send"],
};
await expect(nodeContentRegion(cdp, 7, 555)).resolves.toEqual([
[
{ x: 10, y: 20 },
{ x: 60, y: 20 },
{ x: 60, y: 40 },
{ x: 10, y: 40 },
],
[
{ x: 15, y: 50 },
{ x: 90, y: 50 },
{ x: 90, y: 80 },
{ x: 15, y: 80 },
],
]);
});
it("returns null for degenerate quads", () => {
expect(quadBoundingRect([0, 0, 0, 0, 0, 0, 0, 0])).toBeNull();
expect(quadBoundingRect([1, 2, 3])).toBeNull();
});
});
describe("nodeBoundingRect", () => {
it("falls back to visible descendant bounds for zero-size containers", async () => {
const calls: Array<{ method: string; params?: object }> = [];
const send = async (_tabId: number, method: string, params?: object) => {
@@ -38,12 +68,14 @@ describe("nodeBoundingRect", () => {
};
const cdp = { send: send as CdpRunner["send"] };
await expect(nodeBoundingRect(cdp, 7, 555)).resolves.toEqual({
x: 242,
y: 468,
width: 769,
height: 180,
});
await expect(nodeContentRegion(cdp, 7, 555)).resolves.toEqual([
[
{ x: 242, y: 468 },
{ x: 1011, y: 468 },
{ x: 1011, y: 648 },
{ x: 242, y: 648 },
],
]);
expect(calls.map((c) => c.method)).toEqual([
"DOM.getContentQuads",
"DOM.getBoxModel",
@@ -0,0 +1,286 @@
import { describe, expect, it, vi } from "vitest";
import { resolveFrameProjection, resolveNodeGeometry } from "../frame-geometry";
import {
clipPolygon,
polygonArea,
polygonBounds,
projectAndClipRegion,
projectUnitPoint,
type Quad,
rectPolygon,
regionBounds,
} from "../geometry";
import type { CdpRunner } from "../shared";
describe("frame geometry projection", () => {
it("keeps region bounds separate from polygon area", () => {
const region = [
rectPolygon({ x: 0, y: 0, w: 10, h: 10 }),
rectPolygon({ x: 20, y: 5, w: 5, h: 5 }),
];
expect(regionBounds(region)).toEqual({ x: 0, y: 0, width: 25, height: 10 });
expect(polygonArea(region[0])).toBe(100);
});
it("maps through the iframe content box instead of its border box", () => {
const contentQuad: Quad = [
{ x: 204, y: 306 },
{ x: 604, y: 306 },
{ x: 604, y: 506 },
{ x: 204, y: 506 },
];
const projected = projectAndClipRegion(
[rectPolygon({ x: 10, y: 20, w: 100, h: 40 })],
[{ sourceViewport: { width: 200, height: 100 }, destinationQuad: contentQuad }],
{ width: 1000, height: 800 },
);
expect(polygonBounds(projected.flat())).toEqual({ x: 224, y: 346, width: 200, height: 80 });
});
it("uses a projective mapping for perspective-transformed iframe quads", () => {
const quad: Quad = [
{ x: 0, y: 0 },
{ x: 200, y: 20 },
{ x: 180, y: 140 },
{ x: 20, y: 100 },
];
expect(projectUnitPoint({ x: 0, y: 0 }, quad)).toEqual(quad[0]);
expect(projectUnitPoint({ x: 1, y: 0 }, quad)).toEqual(quad[1]);
expect(projectUnitPoint({ x: 1, y: 1 }, quad)).toEqual(quad[2]);
expect(projectUnitPoint({ x: 0, y: 1 }, quad)).toEqual(quad[3]);
});
it("clips against the iframe polygon rather than its bounding box", () => {
const frameQuad: Quad = [
{ x: 50, y: 0 },
{ x: 100, y: 50 },
{ x: 50, y: 100 },
{ x: 0, y: 50 },
];
expect(clipPolygon(rectPolygon({ x: 0, y: 0, w: 10, h: 10 }), frameQuad)).toEqual([]);
});
it("does not reapply same-target frame transforms around an OOPIF", async () => {
const send = vi.fn(async (_tabId, method, params) => {
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 800, clientHeight: 600 } };
}
if (method === "DOM.getBoxModel") {
const backendNodeId = (params as { backendNodeId?: number }).backendNodeId;
return backendNodeId === 20
? { model: { content: [110, 70, 310, 70, 310, 170, 110, 170] } }
: { model: { content: [100, 50, 400, 50, 400, 250, 100, 250] } };
}
throw new Error(`unexpected root command ${method}`);
});
const cdp: CdpRunner = {
send: send as CdpRunner["send"],
sendToTarget: vi.fn(async (_target, method) => {
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } };
}
throw new Error(`unexpected child command ${method}`);
}) as CdpRunner["sendToTarget"],
};
const projection = await resolveFrameProjection(
cdp,
{
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "same-process-parent",
parentFrameId: "main",
ownerBackendNodeId: 10,
target: { tabId: 4 },
},
{
frameId: "oopif",
parentFrameId: "same-process-parent",
ownerBackendNodeId: 20,
target: { tabId: 4, sessionId: "oopif-session" },
},
],
},
"oopif",
);
expect(projection?.edges).toEqual([
{
sourceViewport: { width: 200, height: 100 },
destinationQuad: [
{ x: 110, y: 70 },
{ x: 310, y: 70 },
{ x: 310, y: 170 },
{ x: 110, y: 170 },
],
destinationClips: [
[
{ x: 100, y: 50 },
{ x: 400, y: 50 },
{ x: 400, y: 250 },
{ x: 100, y: 250 },
],
],
},
]);
});
it("clips live OOPIF geometry at every frame boundary", async () => {
const cdp: CdpRunner = {
send: vi.fn(async (_tabId, method) => {
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 800, clientHeight: 600 } };
}
if (method === "DOM.getBoxModel") {
return { model: { content: [100, 100, 300, 100, 300, 200, 100, 200] } };
}
throw new Error(`unexpected root command ${method}`);
}) as CdpRunner["send"],
sendToTarget: vi.fn(async (_target, method) => {
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } };
}
if (method === "DOM.getContentQuads") {
return { quads: [[-50, -20, 250, -20, 250, 120, -50, 120]] };
}
throw new Error(`unexpected child command ${method}`);
}) as CdpRunner["sendToTarget"],
getFrameGraph: vi.fn(async () => ({
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "child",
parentFrameId: "main",
ownerBackendNodeId: 10,
target: { tabId: 4, sessionId: "child-session" },
},
],
})),
};
const geometry = await resolveNodeGeometry(cdp, 4, {
target: { tabId: 4, sessionId: "child-session" },
frameId: "child",
backendNodeId: 101,
});
expect(geometry).toMatchObject({
topBounds: { x: 100, y: 100, width: 200, height: 100 },
actionPoint: { x: 200, y: 150 },
});
});
it("reuses one frame graph while scrolling and projecting live geometry", async () => {
const getFrameGraph = vi.fn(async () => ({
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "child",
parentFrameId: "main",
ownerBackendNodeId: 10,
target: { tabId: 4, sessionId: "child-session" },
},
],
}));
const cdp: CdpRunner = {
send: vi.fn(async (_tabId, method) => {
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 800, clientHeight: 600 } };
}
if (method === "DOM.getBoxModel") {
return { model: { content: [100, 100, 300, 100, 300, 200, 100, 200] } };
}
throw new Error(`unexpected root command ${method}`);
}) as CdpRunner["send"],
sendToTarget: vi.fn(async (_target, method) => {
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } };
}
if (method === "DOM.getContentQuads") {
return { quads: [[0, 0, 20, 0, 20, 20, 0, 20]] };
}
throw new Error(`unexpected child command ${method}`);
}) as CdpRunner["sendToTarget"],
getFrameGraph,
};
await expect(
resolveNodeGeometry(
cdp,
4,
{
target: { tabId: 4, sessionId: "child-session" },
frameId: "child",
backendNodeId: 101,
},
{ scrollIntoView: true },
),
).resolves.not.toMatchObject({ code: expect.any(String) });
expect(getFrameGraph).toHaveBeenCalledOnce();
});
it("clips live geometry to same-target iframe ancestors", async () => {
const cdp: CdpRunner = {
send: vi.fn(async (_tabId, method) => {
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 800, clientHeight: 600 } };
}
if (method === "DOM.getBoxModel") {
return { model: { content: [100, 100, 300, 100, 300, 200, 100, 200] } };
}
if (method === "DOM.getContentQuads") {
return { quads: [[50, 50, 350, 50, 350, 250, 50, 250]] };
}
throw new Error(`unexpected ${method}`);
}) as CdpRunner["send"],
getFrameGraph: vi.fn(async () => ({
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "child",
parentFrameId: "main",
ownerBackendNodeId: 10,
target: { tabId: 4 },
},
],
})),
};
const geometry = await resolveNodeGeometry(cdp, 4, {
target: { tabId: 4 },
frameId: "child",
backendNodeId: 101,
});
expect(geometry).toMatchObject({
topBounds: { x: 100, y: 100, width: 200, height: 100 },
actionPoint: { x: 200, y: 150 },
});
});
it("fails closed when an OOPIF frame graph is unavailable", async () => {
const cdp: CdpRunner = {
send: vi.fn(),
sendToTarget: vi.fn(async (_target, method) => {
if (method === "DOM.getContentQuads") {
return { quads: [[0, 0, 20, 0, 20, 20, 0, 20]] };
}
throw new Error(`unexpected ${method}`);
}) as CdpRunner["sendToTarget"],
};
await expect(
resolveNodeGeometry(cdp, 4, {
target: { tabId: 4, sessionId: "child-session" },
frameId: "child",
backendNodeId: 101,
}),
).resolves.toMatchObject({ code: "cdp_failed" });
});
});
@@ -1,5 +1,6 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { SessionManager } from "@/session-manager/manager";
import { RefStore } from "@/session-manager/ref-store";
import type { RequestHelpParams } from "@/transport/types";
import { handleRequestHelp, type RequestHelpDeps, resetHelpLifecycleForTests } from "../human-loop";
@@ -50,11 +51,10 @@ function installHelpLifecycleChrome() {
}
function fakeManager(sessionId: string, agentWindowId: number, tabId: number) {
const refStore = new RefStore();
const mgr = {
get: (id: string) =>
id === sessionId
? { sessionId, agentWindowId, refStore: { resolve: () => null }, borrowedTabs: new Map() }
: null,
id === sessionId ? { sessionId, agentWindowId, refStore, borrowedTabs: new Map() } : null,
findByWindowId: (wid: number) => (wid === agentWindowId ? { sessionId } : null),
} as unknown as SessionManager;
return mgr;
@@ -339,26 +339,23 @@ describe("handleRequestHelp", () => {
ac.abort();
await expect(pending).resolves.toMatchObject({ code: "cancelled" });
expect(cdpSend).toHaveBeenCalledWith(
5,
"DOM.querySelectorAll",
expect.objectContaining({ selector: "[data-bsk-help]" }),
);
expect(cdpSend).toHaveBeenCalledWith(
6,
"DOM.querySelectorAll",
expect.objectContaining({ selector: "[data-bsk-help]" }),
expect(cdpSend).not.toHaveBeenCalledWith(
expect.anything(),
"DOM.removeAttribute",
expect.anything(),
);
});
it("tags ref targets via CDP and reports them matched", async () => {
it("resolves ref targets to explicit viewport rectangles without mutating the page", async () => {
const refStore = new RefStore();
refStore.set("e1", 42, { tabId: 5 });
const mgr = {
get: (id: string) =>
id === "abcd"
? {
sessionId: "abcd",
agentWindowId: 99,
refStore: { resolve: () => 42 },
refStore,
borrowedTabs: new Map(),
}
: null,
@@ -366,7 +363,17 @@ describe("handleRequestHelp", () => {
} as unknown as SessionManager;
const deps = baseDeps({
cdp: {
send: vi.fn(async () => ({ object: { objectId: "obj-1" } })),
send: vi.fn(async (_tabId, method) => {
if (method === "DOM.getDocument") return { root: { nodeId: 1 } };
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "DOM.getContentQuads") {
return { quads: [[10, 20, 110, 20, 110, 60, 10, 60]] };
}
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
}
throw new Error(`unexpected ${method}`);
}),
} as unknown as RequestHelpDeps["cdp"],
});
const res = await handleRequestHelp(
@@ -375,14 +382,97 @@ describe("handleRequestHelp", () => {
deps,
);
const sentMsg = (deps.sendToTab as ReturnType<typeof vi.fn>).mock.calls[0][1];
expect(sentMsg.selectors).toContain('[data-bsk-help="0"]');
expect(sentMsg.selectors).toEqual([]);
expect(sentMsg.rects).toEqual([{ top: 20, left: 10, width: 100, height: 40 }]);
expect(res).toMatchObject({
outcome: "continued",
tab_id: 5,
resolved_targets: [{ matched: true, ref: "@e1" }],
resolved_targets: [
{
matched: true,
ref: "@e1",
},
],
});
});
it("projects an OOPIF ref through its live content quad for the help overlay", async () => {
const mgr = {
get: (id: string) =>
id === "abcd"
? {
sessionId: "abcd",
agentWindowId: 99,
refStore: {
resolveEntry: () => ({
backendNodeId: 42,
tabId: 5,
frameId: "child",
cdpSessionId: "child-session",
generation: 1,
}),
},
borrowedTabs: new Map(),
}
: null,
findByWindowId: (wid: number) => (wid === 99 ? { sessionId: "abcd" } : null),
} as unknown as SessionManager;
const send = vi.fn(async (_tabId, method) => {
if (method === "DOM.getDocument") return { root: { nodeId: 1 } };
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "DOM.getBoxModel") {
return { model: { content: [204, 306, 604, 306, 604, 506, 204, 506] } };
}
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
}
throw new Error(`unexpected root ${method}`);
});
const sendToTarget = vi.fn(async (_target, method) => {
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "DOM.getContentQuads") {
return { quads: [[10, 20, 110, 20, 110, 60, 10, 60]] };
}
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } };
}
throw new Error(`unexpected child ${method}`);
});
const deps = baseDeps({
cdp: {
send,
sendToTarget,
getFrameGraph: vi.fn(async () => ({
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 5 } },
{
frameId: "child",
parentFrameId: "main",
ownerBackendNodeId: 99,
target: { tabId: 5, sessionId: "child-session" },
},
],
})),
} as unknown as NonNullable<RequestHelpDeps["cdp"]>,
});
const res = await handleRequestHelp(
mgr,
baseParams({ tab_id: 5, targets: [{ ref: "@e1" }] }),
deps,
);
const sentMsg = (deps.sendToTab as ReturnType<typeof vi.fn>).mock.calls[0][1];
expect(sentMsg.rects).toEqual([{ top: 346, left: 224, width: 200, height: 80 }]);
expect(res).toMatchObject({ resolved_targets: [{ matched: true, ref: "@e1" }] });
expect(sendToTarget).toHaveBeenCalledWith(
{ tabId: 5, sessionId: "child-session" },
"DOM.getContentQuads",
{ backendNodeId: 42 },
);
});
it("reports ref target unmatched when ref does not resolve", async () => {
const deps = baseDeps();
const res = await handleRequestHelp(
@@ -398,16 +488,15 @@ describe("handleRequestHelp", () => {
});
it("reports ref target unmatched when ref is for another tab", async () => {
const refStore = new RefStore();
refStore.set("e1", 42, { tabId: 4 });
const mgr = {
get: (id: string) =>
id === "abcd"
? {
sessionId: "abcd",
agentWindowId: 99,
refStore: {
resolve: (ref: string, opts: { tabId?: number }) =>
ref === "e1" && opts.tabId === 4 ? 42 : null,
},
refStore,
borrowedTabs: new Map(),
}
: null,
@@ -30,6 +30,9 @@ function makeFakeCdp(handlers: Record<string, (params: unknown) => unknown>) {
const sendImpl = async (tabId: number, method: string, params?: object) => {
sent.push({ tabId, method, params });
const h = handlers[method];
if (!h && method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
}
if (!h) throw new Error(`unexpected CDP call ${method}`);
return h(params);
};
@@ -164,6 +167,63 @@ describe("handleClick", () => {
});
});
it("resolves frame refs in their CDP session and dispatches input in top coordinates", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const ctx = await sm.start("aa11");
ctx.refStore.set("e3", 1234, {
tabId: 4,
frameId: "child-frame",
cdpSessionId: "child-session",
});
const fake = makeFakeCdp({
"Input.dispatchMouseEvent": () => ({}),
"DOM.scrollIntoViewIfNeeded": () => ({}),
"DOM.getBoxModel": () => ({
model: { content: [204, 306, 604, 306, 604, 506, 204, 506] },
}),
});
fake.cdp.getFrameGraph = vi.fn(async () => ({
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "child-frame",
parentFrameId: "main",
ownerBackendNodeId: 99,
target: { tabId: 4, sessionId: "child-session" },
},
],
}));
const targetCalls: Array<{ sessionId?: string; method: string }> = [];
fake.cdp.sendToTarget = vi.fn(async (target, method) => {
targetCalls.push({ sessionId: target.sessionId, method });
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "DOM.getContentQuads") {
return { quads: [[10, 20, 110, 20, 110, 60, 10, 60]] };
}
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 200, clientHeight: 100 } };
}
throw new Error(`unexpected child CDP call ${method}`);
}) as CdpRunner["sendToTarget"];
const res = await handleClick(
sm,
{ session_id: "aa11", ref: "@e3" },
{ cdp: fake.cdp, tabsApi: fake.tabsApi },
);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
// The iframe content box starts at (204, 306) after its border and is scaled 2x.
expect(res).toMatchObject({ x: 324, y: 386 });
expect(targetCalls).toEqual([
{ sessionId: "child-session", method: "DOM.scrollIntoViewIfNeeded" },
{ sessionId: "child-session", method: "DOM.getContentQuads" },
{ sessionId: "child-session", method: "Page.getLayoutMetrics" },
]);
expect(fake.sent.filter((call) => call.method === "Input.dispatchMouseEvent")).toHaveLength(3);
});
it("enables overlay bypass before mouse events when overlay blocks the click point", async () => {
const order: string[] = [];
const bypassOverlay = vi.fn(async (_tabId: number, enabled: boolean) => {
@@ -8,6 +8,7 @@ import {
buildFrameVomScene,
buildVomScene,
type CdpAxNode,
captureVomObservation,
handleGetHtml,
handleObserve,
handleScreenshot,
@@ -63,6 +64,9 @@ function makeFakeCdp(handlers: Record<string, (params?: object) => unknown>) {
const send = vi.fn(async (_tabId: number, method: string, params?: object) => {
sent.push({ method, params });
const handler = handlers[method];
if (!handler && method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
}
if (!handler) throw new Error(`unexpected CDP call ${method}`);
return handler(params);
});
@@ -586,6 +590,7 @@ describe("buildVomScene", () => {
{
frameId: "main",
contextScopeId: "main",
target: { tabId: 7 },
domNodes: mainNodes,
axNodes: [
axNode("root", 1, "RootWebArea"),
@@ -598,6 +603,7 @@ describe("buildVomScene", () => {
contextScopeId: "first",
parentFrameId: "main",
ownerBackendNodeId: 10,
target: { tabId: 7, sessionId: "session-first" },
domNodes: firstNodes,
axNodes: [
axNode("root", 100, "RootWebArea"),
@@ -609,6 +615,7 @@ describe("buildVomScene", () => {
contextScopeId: "second",
parentFrameId: "main",
ownerBackendNodeId: 20,
target: { tabId: 7, sessionId: "session-second" },
domNodes: secondNodes,
axNodes: [
axNode("root", 200, "RootWebArea"),
@@ -621,6 +628,7 @@ describe("buildVomScene", () => {
contextScopeId: "nested",
parentFrameId: "second",
ownerBackendNodeId: 210,
target: { tabId: 7, sessionId: "session-nested" },
domNodes: nestedNodes,
axNodes: [
axNode("root", 300, "RootWebArea"),
@@ -673,6 +681,7 @@ describe("buildVomScene", () => {
{
frameId: "main",
contextScopeId: "main",
target: { tabId: 7 },
domNodes: [],
axNodes: [],
},
@@ -680,6 +689,7 @@ describe("buildVomScene", () => {
frameId: "child",
contextScopeId: "child",
parentFrameId: "main",
target: { tabId: 7 },
domNodes: [childNode],
axNodes: [
{
@@ -740,6 +750,7 @@ describe("buildVomScene", () => {
{
frameId: "main",
contextScopeId: "main",
target: { tabId: 7 },
domNodes: mainNodes,
axNodes: [
{
@@ -761,6 +772,7 @@ describe("buildVomScene", () => {
contextScopeId: "child",
parentFrameId: "main",
ownerBackendNodeId: 10,
target: { tabId: 7, sessionId: "child-session" },
domNodes: [],
axNodes: [
{
@@ -1318,7 +1330,7 @@ describe("buildVomScene", () => {
expect(rendered).toContain(' @e2 textbox "验证码"');
});
it("does not mark non-password inputs sensitive from password-like labels", () => {
it("marks current-password autocomplete as sensitive even for text inputs", () => {
const axNodes: CdpAxNode[] = [
{
nodeId: "1",
@@ -1353,7 +1365,7 @@ describe("buildVomScene", () => {
expect(buildVomScene(axNodes, captured).nodes[0]).toEqual(
expect.objectContaining({
id: 10,
sensitive: false,
sensitive: true,
}),
);
});
@@ -1648,7 +1660,9 @@ describe("buildVomScene", () => {
);
const rendered = renderVom(scene);
expect(rendered.text).toContain('@e1 button "close"');
expect(rendered.refs).toEqual([{ ref: "e1", backendNodeId: 20 }]);
expect(rendered.refs.map(({ ref, backendNodeId }) => ({ ref, backendNodeId }))).toEqual([
{ ref: "e1", backendNodeId: 20 },
]);
});
it("does not promote a clickable container that wraps a real interactive control", () => {
@@ -1792,7 +1806,9 @@ describe("buildVomScene", () => {
expect(scene.nodes.find((n) => n.id === 30)?.role).toBe("generic");
const rendered = renderVom(scene);
expect(rendered.text).toContain('@e1 button "收藏"');
expect(rendered.refs).toEqual([{ ref: "e1", backendNodeId: 20 }]);
expect(rendered.refs.map(({ ref, backendNodeId }) => ({ ref, backendNodeId }))).toEqual([
{ ref: "e1", backendNodeId: 20 },
]);
});
it("builds active scope blocks from active aria-controls relationships", () => {
@@ -2893,10 +2909,360 @@ describe("handleSnapshot", () => {
],
};
}
function secretsSnapshotReply(secrets: {
email: string;
password: string;
otp: string;
card: string;
}) {
const S = [
"html",
"body",
"input",
"position",
"static",
"pointer-events",
"auto",
"type",
"text",
"password",
"autocomplete",
"one-time-code",
"cc-number",
"value",
secrets.email,
secrets.password,
secrets.otp,
secrets.card,
];
const i = (s: string) => S.indexOf(s);
const style = [i("static"), i("auto"), i("auto")];
return {
strings: S,
documents: [
{
nodes: {
parentIndex: [-1, 0, 1, 1, 1, 1],
nodeName: [i("html"), i("body"), i("input"), i("input"), i("input"), i("input")],
backendNodeId: [10, 11, 12, 13, 14, 15],
attributes: [
[],
[],
[i("type"), i("text"), i("value"), i(secrets.email)],
[i("type"), i("password"), i("value"), i(secrets.password)],
[
i("type"),
i("text"),
i("autocomplete"),
i("one-time-code"),
i("value"),
i(secrets.otp),
],
[
i("type"),
i("text"),
i("autocomplete"),
i("cc-number"),
i("value"),
i(secrets.card),
],
],
inputValue: {
index: [2, 3, 4, 5],
value: [i(secrets.email), i(secrets.password), i(secrets.otp), i(secrets.card)],
},
},
layout: {
nodeIndex: [1, 2, 3, 4, 5],
styles: [style, style, style, style, style],
bounds: [
[0, 0, 1000, 800],
[10, 10, 200, 32],
[10, 50, 200, 32],
[10, 90, 200, 32],
[10, 130, 200, 32],
],
paintOrders: [0, 1, 2, 3, 4],
},
},
],
};
}
const VP_METRICS = {
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
};
function makeFrameAwareDeps() {
const strings = ["html", "body", "iframe", "button", "title", "Remote", "static", "auto"];
const i = (value: string) => strings.indexOf(value);
const styles = [i("static"), i("auto"), i("auto")];
const snapshot = {
strings,
documents: [
{
frameId: "main",
nodes: {
parentIndex: [-1, 0, 1],
nodeName: [i("html"), i("body"), i("iframe")],
backendNodeId: [10, 11, 12],
attributes: [[], [], [i("title"), i("Remote")]],
contentDocumentIndex: { index: [2], value: [1] },
},
layout: {
nodeIndex: [0, 1, 2],
styles: [styles, styles, styles],
bounds: [
[0, 0, 1000, 800],
[0, 0, 1000, 800],
[100, 100, 400, 300],
],
paintOrders: [0, 0, 1],
},
},
{
frameId: "child",
nodes: {
parentIndex: [-1, 0, 1],
nodeName: [i("html"), i("body"), i("button")],
backendNodeId: [20, 21, 22],
attributes: [[], [], []],
},
layout: {
nodeIndex: [0, 1, 2],
styles: [styles, styles, styles],
bounds: [
[0, 0, 400, 300],
[0, 0, 400, 300],
[20, 30, 120, 40],
],
paintOrders: [0, 0, 1],
},
},
],
};
const mainAx: CdpAxNode[] = [
{
nodeId: "main-root",
role: { type: "role", value: "RootWebArea" },
backendDOMNodeId: 11,
childIds: ["frame-owner"],
},
{
nodeId: "frame-owner",
parentId: "main-root",
role: { type: "role", value: "Iframe" },
name: { type: "computedString", value: "Remote" },
backendDOMNodeId: 12,
},
];
const childAx: CdpAxNode[] = [
{
nodeId: "child-root",
role: { type: "role", value: "RootWebArea" },
backendDOMNodeId: 21,
childIds: ["child-button"],
},
{
nodeId: "child-button",
parentId: "child-root",
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Frame action" },
backendDOMNodeId: 22,
},
];
const send = vi.fn(async (_tabId: number, method: string) => {
if (method === "Page.getLayoutMetrics") return VP_METRICS;
if (method === "DOMSnapshot.enable" || method === "Accessibility.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") return snapshot;
if (method === "Accessibility.getFullAXTree") return { nodes: mainAx };
throw new Error(`unexpected root CDP method ${method}`);
});
const sendToTarget = vi.fn(async (_target, method: string) => {
if (method === "Accessibility.enable") return {};
if (method === "Accessibility.getFullAXTree") return { nodes: childAx };
throw new Error(`unexpected child CDP method ${method}`);
});
const cdp = {
send: send as unknown as CdpRunner["send"],
sendToTarget: sendToTarget as unknown as NonNullable<CdpRunner["sendToTarget"]>,
getFrameGraph: vi.fn(async () => ({
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "child",
parentFrameId: "main",
ownerBackendNodeId: 12,
target: { tabId: 4, sessionId: "child-session" },
},
],
})),
trackSessionTab: vi.fn(),
} satisfies CdpRunner;
return {
cdp,
tabsApi: {
get: vi.fn(
async (tabId: number) => ({ id: tabId, windowId: 100, active: true }) as chrome.tabs.Tab,
),
query: vi.fn(async () => [{ id: 4, windowId: 100, active: true } as chrome.tabs.Tab]),
},
};
}
it("exposes the frame-aware observation result for recording", async () => {
const ax: CdpAxNode[] = [
{
nodeId: "root",
role: { type: "role", value: "RootWebArea" },
backendDOMNodeId: 11,
childIds: ["password"],
},
{
nodeId: "password",
parentId: "root",
role: { type: "role", value: "textbox" },
name: { type: "computedString", value: "Password" },
value: { value: "hunter2" },
backendDOMNodeId: 13,
},
];
const deps = makeOverlayDeps(ax, loginSnapshotReply(), VP_METRICS);
const result = await captureVomObservation(deps.cdp, 4, "https://example.com", {
redactValues: true,
});
expect(result.text).not.toContain("hunter2");
expect(result.text).toContain("•••");
expect(result.refs[0]).toMatchObject({
backendNodeId: 13,
role: "textbox",
name: "Password",
line: expect.any(Number),
});
expect(result.rootFrameId).toBe("root");
expect(result).not.toHaveProperty("frameDocuments");
expect(result.frames).toEqual([{ frameId: "root", target: { tabId: 4 } }]);
expect(result.matchNodes.find((node) => node.backendNodeId === 13)).toMatchObject({
frameId: "root",
backendNodeId: 13,
tag: "input",
rect: { x: 400, y: 300, w: 200, h: 40 },
localRect: { x: 400, y: 300, w: 200, h: 40 },
});
});
it("does not leak form secrets through the record-safe observation payload", async () => {
const secrets = {
email: "user@example.com",
password: "hunter2",
otp: "847291",
card: "4111111111111111",
};
const ax: CdpAxNode[] = [
{
nodeId: "root",
role: { type: "role", value: "RootWebArea" },
backendDOMNodeId: 11,
childIds: ["email", "password", "otp", "card"],
},
{
nodeId: "email",
parentId: "root",
role: { type: "role", value: "textbox" },
name: { type: "computedString", value: "Email" },
value: { value: secrets.email },
backendDOMNodeId: 12,
},
{
nodeId: "password",
parentId: "root",
role: { type: "role", value: "textbox" },
name: { type: "computedString", value: "Password" },
value: { value: secrets.password },
backendDOMNodeId: 13,
},
{
nodeId: "otp",
parentId: "root",
role: { type: "role", value: "textbox" },
name: { type: "computedString", value: "Code" },
value: { value: secrets.otp },
backendDOMNodeId: 14,
},
{
nodeId: "card",
parentId: "root",
role: { type: "role", value: "textbox" },
name: { type: "computedString", value: "Card" },
value: { value: secrets.card },
backendDOMNodeId: 15,
},
];
const deps = makeOverlayDeps(ax, secretsSnapshotReply(secrets), VP_METRICS);
const result = await captureVomObservation(deps.cdp, 4, "https://example.com", {
redactValues: true,
});
const dumped = JSON.stringify(result);
expect(result).not.toHaveProperty("frameDocuments");
expect(dumped).not.toContain(secrets.email);
expect(dumped).not.toContain(secrets.password);
expect(dumped).not.toContain(secrets.otp);
expect(dumped).not.toContain(secrets.card);
expect(dumped).not.toContain("formValue");
expect(dumped).not.toContain("formDefaultValue");
expect(dumped).not.toContain("axNodes");
expect(dumped).not.toContain("domNodes");
expect(dumped).not.toContain("attrs");
expect(result.text).toContain("•••");
});
it("keeps frames and rendered refs on the same frame identity", async () => {
const deps = makeFrameAwareDeps();
const result = await captureVomObservation(deps.cdp, 4, "https://example.com");
expect(result.rootFrameId).toBe("main");
expect(result.frames).toHaveLength(2);
expect(result.frames.find((frame) => frame.frameId === "child")).toEqual({
frameId: "child",
parentFrameId: "main",
ownerBackendNodeId: 12,
target: { tabId: 4, sessionId: "child-session" },
});
expect(result.matchNodes.find((node) => node.backendNodeId === 22)).toMatchObject({
frameId: "child",
tag: "button",
localRect: { x: 20, y: 30, w: 120, h: 40 },
});
expect(result.refs.find((ref) => ref.backendNodeId === 22)).toMatchObject({
frameId: "child",
name: "Frame action",
});
});
it("stores iframe refs with their owning CDP session", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const ctx = await sm.start("aa11");
const deps = makeFrameAwareDeps();
const result = await handleSnapshot(sm, { session_id: "aa11" }, deps);
if ("code" in result) throw new Error(`unexpected error: ${JSON.stringify(result)}`);
expect(result.text).toContain('@e1 button "Frame action"');
const frameRef = [...ctx.refStore.entries()].find(([, entry]) => entry.backendNodeId === 22);
expect(frameRef?.[1]).toMatchObject({
tabId: 4,
frameId: "child",
cdpSessionId: "child-session",
});
});
it("renders a blocking login overlay as the focused top layer", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
await sm.start("aa11");
@@ -3407,6 +3773,32 @@ describe("handleGetHtml", () => {
expect(deps.send).toHaveBeenCalledTimes(1);
});
it("reads a frame ref through its child CDP session", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const ctx = await sm.start("aa11");
ctx.refStore.set("e7", 4242, {
tabId: 4,
frameId: "child-frame",
cdpSessionId: "child-session",
});
const deps = makeDeps({});
const sendToTarget = vi.fn(async (target, method, params) => {
expect(target).toEqual({ tabId: 4, sessionId: "child-session" });
expect(method).toBe("DOM.getOuterHTML");
expect(params).toEqual({ backendNodeId: 4242 });
return { outerHTML: "<button>frame</button>" };
});
(deps.cdp as CdpRunner).sendToTarget = sendToTarget as unknown as NonNullable<
CdpRunner["sendToTarget"]
>;
const res = await handleGetHtml(sm, { session_id: "aa11", ref: "@e7" }, deps);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
expect(res.html).toBe("<button>frame</button>");
expect(deps.send).not.toHaveBeenCalled();
});
it("returns not_found when a ref belongs to another tab", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const ctx = await sm.start("aa11");
@@ -66,6 +66,52 @@ function makeApis(
}
describe("handleSessionStop with auto-return", () => {
it("leaves the session untouched when cancellation arrived before teardown", async () => {
const aw = fakeAgentWindow([100]);
const sm = new SessionManager({ agentWindow: aw });
await sm.start("aa11");
const cdp = { detachSession: vi.fn(async () => {}) };
const controller = new AbortController();
controller.abort();
const res = await handleSessionStop(
sm,
{ session_id: "aa11" },
{ cdp, signal: controller.signal },
);
expect(res).toMatchObject({ code: "cancelled" });
expect(sm.has("aa11")).toBe(true);
expect(cdp.detachSession).not.toHaveBeenCalled();
expect(aw.remove).not.toHaveBeenCalled();
});
it("does not close the Agent Window when cancellation lands during CDP detach", async () => {
const aw = fakeAgentWindow([100]);
const sm = new SessionManager({ agentWindow: aw });
await sm.start("aa11");
const controller = new AbortController();
let finishDetach: () => void = () => {};
const detach = new Promise<void>((resolve) => {
finishDetach = resolve;
});
const cdp = { detachSession: vi.fn(() => detach) };
const stopping = handleSessionStop(
sm,
{ session_id: "aa11" },
{ cdp, signal: controller.signal },
);
await vi.waitFor(() => expect(cdp.detachSession).toHaveBeenCalledWith("aa11"));
controller.abort();
finishDetach();
const res = await stopping;
expect(res).toMatchObject({ code: "cancelled" });
expect(sm.has("aa11")).toBe(true);
expect(aw.remove).not.toHaveBeenCalled();
});
it("returns every borrowed tab and closes the Agent Window in the right order", async () => {
const aw = fakeAgentWindow([100]);
const sm = new SessionManager({ agentWindow: aw });
@@ -46,6 +46,25 @@ describe("lookupSnapshotRef", () => {
});
describe("resolveSnapshotRef", () => {
it("preserves the frame and child session needed to route iframe refs", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const ctx = await sm.start("aa11");
ctx.refStore.set("e3", 1234, {
tabId: 4,
frameId: "child-frame",
cdpSessionId: "child-session",
});
const expected = {
backendNodeId: 1234,
refKey: "e3",
frameId: "child-frame",
cdpSessionId: "child-session",
};
expect(lookupSnapshotRef(ctx, "@e3", 4)).toEqual(expected);
expect(resolveSnapshotRef(ctx, "@e3", 4)).toEqual(expected);
});
it("returns not_found for unknown ref", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const ctx = await sm.start("aa11");
@@ -250,9 +250,10 @@ const DEFAULT_NOTIFICATION_COPY: BorrowNotificationCopy = {
// URL injection-eligibility test
// ---------------------------------------------------------------------------
const CHROME_WEB_STORE_RES = [
const EXTENSION_STORE_RES = [
/^https:\/\/chrome\.google\.com\/webstore/i,
/^https:\/\/chromewebstore\.google\.com/i,
/^https:\/\/microsoftedge\.microsoft\.com\/addons/i,
];
/**
@@ -264,14 +265,15 @@ const CHROME_WEB_STORE_RES = [
* - `ftp://` no longer hosts content scripts reliably in modern Chrome.
* - `about:` / `chrome:` / `chrome-extension:` / `edge:` / `devtools:` /
* `view-source:` / `data:` / `blob:` are blocked by the platform.
* - The Chrome Web Store (both legacy and new domains) is also blocked.
* - Extension storefronts are blocked by their own browser: the Chrome Web
* Store (legacy and new domains) under Chrome, and Edge Add-ons under Edge.
*
* Mirrors the implicit scheme list documented in
* https://developer.chrome.com/docs/extensions/develop/concepts/match-patterns.
*/
export function isInjectableContentScriptUrl(url: string | undefined): boolean {
if (!url) return false;
for (const re of CHROME_WEB_STORE_RES) {
for (const re of EXTENSION_STORE_RES) {
if (re.test(url)) return false;
}
return /^https?:\/\//i.test(url);
@@ -0,0 +1,9 @@
export {
type CaptureVomObservationOptions,
captureVomObservation,
} from "./observation";
export {
type CaptureVomFrame,
type CaptureVomMatchNode,
type CaptureVomObservationResult,
} from "./vom/record-safe-observation";
+1
View File
@@ -290,6 +290,7 @@ export class ToolDispatcher {
await this.releaseHoverLatch((req.params as SessionStopParams).session_id);
return handleSessionStop(this.sessions, req.params as SessionStopParams, {
cdp: this.cdp,
signal,
});
}
case "tool.tab_list":
+41 -96
View File
@@ -1,9 +1,15 @@
// Shared CDP helpers for scrolling nodes into view and computing
// viewport-space bounding rectangles. Used by interaction tools (click
// centre) and observation tools (element screenshot clip).
// Resolve and scroll nodes inside one CDP target. Cross-frame projection
// belongs to frame-geometry.ts.
import type { RpcError } from "@/transport/types";
import { rpcError } from "./errors";
import {
parseCdpQuad,
polygonBounds,
type Region,
rectPolygon,
type ViewportRect,
} from "./geometry";
import { type CdpRunner, isRpcError } from "./shared";
const ELEMENT_NOT_VISIBLE_MESSAGE =
@@ -13,45 +19,6 @@ function elementNotVisibleError(): RpcError {
return rpcError("permission_denied", "element_not_visible", ELEMENT_NOT_VISIBLE_MESSAGE);
}
/** Viewport-space axis-aligned bounding box for CDP `Page.captureScreenshot` clip. */
export interface ViewportRect {
x: number;
y: number;
width: number;
height: number;
}
/**
* Compute the axis-aligned bounding box of an 8-double CDP content
* quad / box-model polygon. Exported for unit tests.
*/
export function quadBoundingRect(quad: number[]): ViewportRect | null {
if (quad.length !== 8) return null;
const xs = [quad[0], quad[2], quad[4], quad[6]];
const ys = [quad[1], quad[3], quad[5], quad[7]];
const minX = Math.min(...xs);
const maxX = Math.max(...xs);
const minY = Math.min(...ys);
const maxY = Math.max(...ys);
const width = maxX - minX;
const height = maxY - minY;
if (width <= 0 || height <= 0) return null;
return { x: minX, y: minY, width, height };
}
function rectToQuad(rect: ViewportRect): number[] {
return [
rect.x,
rect.y,
rect.x + rect.width,
rect.y,
rect.x + rect.width,
rect.y + rect.height,
rect.x,
rect.y + rect.height,
];
}
/**
* Resolve `backendNodeId` → CDP `objectId` so we can invoke
* `Runtime.callFunctionOn` against the live JS object.
@@ -112,48 +79,24 @@ export async function scrollNodeIntoView(
}
}
/**
* Viewport-space bounding box for `backendNodeId`. Prefers
* `DOM.getContentQuads`, then `DOM.getBoxModel`, then the union of
* visible descendant client rects for zero-size overflow containers.
*/
export async function nodeBoundingRect(
function validCdpRegion(quads: number[][] | undefined): Region {
return (quads ?? []).flatMap((raw) => {
const polygon = parseCdpQuad(raw);
return polygon && polygonBounds(polygon) ? [polygon] : [];
});
}
async function resolveVisibleContentRegion(
cdp: CdpRunner,
tabId: number,
backendNodeId: number,
): Promise<ViewportRect | RpcError> {
const polygon = await visibleContentPolygon(cdp, tabId, backendNodeId);
if (isRpcError(polygon)) return polygon;
// visibleContentPolygon only returns quads with positive area.
return quadBoundingRect(polygon)!;
}
/**
* Compute the centroid of a CDP content quad. Quads are reported as
* 8 doubles in clockwise order: (x1, y1, x2, y2, x3, y3, x4, y4).
*/
export function quadCentre(quad: number[]): { x: number; y: number } {
const x = (quad[0] + quad[4]) / 2;
const y = (quad[1] + quad[5]) / 2;
return { x, y };
}
/** Centre of a `DOM.getBoxModel` `content` polygon (also 8 doubles). */
export function boxCentre(box: number[]): { x: number; y: number } {
return quadCentre(box);
}
async function visibleContentPolygon(
cdp: CdpRunner,
tabId: number,
backendNodeId: number,
): Promise<number[] | RpcError> {
): Promise<Region | RpcError> {
try {
const quads = await cdp.send<{ quads?: number[][] }>(tabId, "DOM.getContentQuads", {
backendNodeId,
});
const quad = quads.quads?.find((q) => q.length === 8 && quadBoundingRect(q) !== null);
if (quad) return quad;
const visible = validCdpRegion(quads.quads);
if (visible.length > 0) return visible;
} catch (err) {
console.debug("[bsk element-geometry] getContentQuads failed", err);
}
@@ -162,15 +105,34 @@ async function visibleContentPolygon(
backendNodeId,
});
const content = box.model?.content;
if (content && content.length === 8 && quadBoundingRect(content)) return content;
const fallback = validCdpRegion(content ? [content] : undefined);
if (fallback.length > 0) return fallback;
} catch (err) {
console.debug("[bsk element-geometry] getBoxModel failed", err);
}
const descendantRect = await descendantBoundingRect(cdp, tabId, backendNodeId);
if (descendantRect) return rectToQuad(descendantRect);
if (descendantRect) {
return [
rectPolygon({
x: descendantRect.x,
y: descendantRect.y,
w: descendantRect.width,
h: descendantRect.height,
}),
];
}
return elementNotVisibleError();
}
/** Resolve every visible content region in the node's target-local viewport. */
export async function nodeContentRegion(
cdp: CdpRunner,
tabId: number,
backendNodeId: number,
): Promise<Region | RpcError> {
return resolveVisibleContentRegion(cdp, tabId, backendNodeId);
}
async function descendantBoundingRect(
cdp: CdpRunner,
tabId: number,
@@ -241,20 +203,3 @@ function parseViewportRect(value: unknown): ViewportRect | null {
if (![x, y, width, height].every(Number.isFinite) || width <= 0 || height <= 0) return null;
return { x, y, width, height };
}
/**
* Get the click point for `backendNodeId`. Prefers
* `DOM.getContentQuads` (richer for rotated / transformed elements);
* falls back to `DOM.getBoxModel`, then visible descendant bounds.
* Returns `permission_denied` with "element not visible" when every
* geometry path comes back empty / fails.
*/
export async function nodeCentre(
cdp: CdpRunner,
tabId: number,
backendNodeId: number,
): Promise<{ x: number; y: number } | RpcError> {
const polygon = await visibleContentPolygon(cdp, tabId, backendNodeId);
if (isRpcError(polygon)) return polygon;
return quadCentre(polygon);
}
+290
View File
@@ -0,0 +1,290 @@
import { type CdpFrame, type CdpFrameGraph, type CdpTarget } from "@/browser-driver/frame-graph";
import type { RpcError } from "@/transport/types";
import { nodeContentRegion, scrollNodeIntoView } from "./element-geometry";
import {
clipPolygon,
type GeometryProjection,
type Point,
type Polygon,
type ProjectiveEdge,
parseCdpQuad,
polygonArea,
polygonCentroid,
projectRegionToViewport,
type Quad,
type Region,
rectPolygon,
regionBounds,
type Size,
type ViewportRect,
} from "./geometry";
import { type CdpRunner, cdpRunnerForTarget, isRpcError, sendToCdpTarget } from "./shared";
export interface NodeAddress {
target: CdpTarget;
backendNodeId: number;
frameId?: string;
}
export interface ResolvedNodeGeometry {
topVisibleRegions: Region;
topBounds: ViewportRect;
actionPoint: Point;
}
function geometryError(message: string): RpcError {
return { code: "cdp_failed", message };
}
function frameMap(graph: CdpFrameGraph): Map<string, CdpFrame> {
return new Map(graph.frames.map((frame) => [frame.frameId, frame]));
}
function targetRootFrame(byId: Map<string, CdpFrame>, frame: CdpFrame): CdpFrame | null {
const seen = new Set<string>();
let current = frame;
while (current.parentFrameId) {
if (seen.has(current.frameId)) return null;
seen.add(current.frameId);
const parent = byId.get(current.parentFrameId);
if (!parent || parent.target.sessionId !== current.target.sessionId) break;
current = parent;
}
return current;
}
function frameAncestry(graph: CdpFrameGraph, frameId: string): CdpFrame[] | null {
const byId = frameMap(graph);
const path: CdpFrame[] = [];
const seen = new Set<string>();
let current = byId.get(frameId);
if (!current) return null;
while (current.parentFrameId) {
if (seen.has(current.frameId)) return null;
seen.add(current.frameId);
path.push(current);
const parent = byId.get(current.parentFrameId);
if (!parent) return null;
current = parent;
}
return path;
}
async function targetViewport(cdp: CdpRunner, target: CdpTarget): Promise<Size | null> {
const metrics = await sendToCdpTarget<{
cssLayoutViewport?: { clientWidth?: number; clientHeight?: number };
layoutViewport?: { clientWidth?: number; clientHeight?: number };
}>(cdp, target, "Page.getLayoutMetrics", {});
const viewport = metrics.cssLayoutViewport ?? metrics.layoutViewport ?? {};
const width = viewport.clientWidth ?? 0;
const height = viewport.clientHeight ?? 0;
return width > 0 && height > 0 ? { width, height } : null;
}
async function ownerContentQuad(
cdp: CdpRunner,
parent: CdpFrame,
ownerBackendNodeId: number,
): Promise<Quad | null> {
const result = await sendToCdpTarget<{ model?: { content?: number[] } }>(
cdp,
parent.target,
"DOM.getBoxModel",
{ backendNodeId: ownerBackendNodeId },
);
return parseCdpQuad(result.model?.content);
}
async function sameTargetFrameClips(
cdp: CdpRunner,
byId: Map<string, CdpFrame>,
frame: CdpFrame,
targetRoot: CdpFrame,
): Promise<Polygon[] | null> {
const clips: Polygon[] = [];
let current = frame;
const seen = new Set<string>();
while (current.frameId !== targetRoot.frameId) {
if (seen.has(current.frameId) || !current.parentFrameId) return null;
seen.add(current.frameId);
const parent = byId.get(current.parentFrameId);
if (!parent || current.ownerBackendNodeId === undefined) return null;
const clip = await ownerContentQuad(cdp, parent, current.ownerBackendNodeId);
if (!clip) return null;
clips.push(clip);
current = parent;
}
return clips;
}
export async function resolveFrameProjection(
cdp: CdpRunner,
graph: CdpFrameGraph,
frameId: string,
): Promise<GeometryProjection | null> {
const byId = frameMap(graph);
const frame = byId.get(frameId);
if (!frame) return null;
let targetRoot = targetRootFrame(byId, frame);
if (!targetRoot) return null;
const sourceViewport = await targetViewport(cdp, targetRoot.target);
if (!sourceViewport) return null;
const sourceClips = await sameTargetFrameClips(cdp, byId, frame, targetRoot);
if (!sourceClips) return null;
const edges: ProjectiveEdge[] = [];
while (targetRoot.parentFrameId) {
const parent = byId.get(targetRoot.parentFrameId);
if (!parent || targetRoot.ownerBackendNodeId === undefined) return null;
const destinationQuad = await ownerContentQuad(cdp, parent, targetRoot.ownerBackendNodeId);
if (!destinationQuad) return null;
const source =
edges.length === 0 ? sourceViewport : await targetViewport(cdp, targetRoot.target);
if (!source) return null;
const parentTargetRoot = targetRootFrame(byId, parent);
if (!parentTargetRoot) return null;
const destinationClips = await sameTargetFrameClips(cdp, byId, parent, parentTargetRoot);
if (!destinationClips) return null;
edges.push({ sourceViewport: source, destinationQuad, destinationClips });
targetRoot = parentTargetRoot;
}
const topViewport =
edges.length === 0 ? sourceViewport : await targetViewport(cdp, targetRoot.target);
return topViewport ? { sourceClips, edges, topViewport } : null;
}
async function loadFrameGraph(cdp: CdpRunner, tabId: number): Promise<CdpFrameGraph | null> {
if (!cdp.getFrameGraph) return null;
try {
return await cdp.getFrameGraph(tabId);
} catch (error) {
console.debug("[bsk frame-geometry] frame graph resolution failed", error);
return null;
}
}
async function scrollFrameOwners(
cdp: CdpRunner,
tabId: number,
graph: CdpFrameGraph,
frameId: string,
): Promise<RpcError | null> {
const byId = frameMap(graph);
const ancestry = frameAncestry(graph, frameId);
if (!ancestry) return geometryError(`could not resolve frame ancestry for ${frameId}`);
for (const child of [...ancestry].reverse()) {
const parent = child.parentFrameId ? byId.get(child.parentFrameId) : undefined;
if (!parent || child.ownerBackendNodeId === undefined) {
return geometryError(`could not resolve frame owner for ${child.frameId}`);
}
const error = await scrollNodeIntoView(
cdpRunnerForTarget(cdp, parent.target),
tabId,
child.ownerBackendNodeId,
);
if (error) return error;
}
return null;
}
async function scrollElementWithFrameGraph(
cdp: CdpRunner,
tabId: number,
target: CdpTarget,
backendNodeId: number,
frameId: string | undefined,
graph: CdpFrameGraph | null,
): Promise<RpcError | null> {
if (frameId) {
if (!graph) return geometryError(`could not resolve frame graph for ${frameId}`);
const error = await scrollFrameOwners(cdp, tabId, graph, frameId);
if (error) return error;
} else if (target.sessionId) {
return geometryError("an OOPIF node address requires frameId");
}
return scrollNodeIntoView(cdpRunnerForTarget(cdp, target), tabId, backendNodeId);
}
export async function scrollElementAndFramesIntoView(
cdp: CdpRunner,
tabId: number,
target: CdpTarget,
backendNodeId: number,
frameId?: string,
): Promise<RpcError | null> {
const graph = frameId ? await loadFrameGraph(cdp, tabId) : null;
return scrollElementWithFrameGraph(cdp, tabId, target, backendNodeId, frameId, graph);
}
function largestRegion(regions: Region): Polygon | null {
let largest: { polygon: Polygon; area: number } | null = null;
for (const polygon of regions) {
const area = polygonArea(polygon);
if (area <= 0) continue;
if (!largest || area > largest.area) largest = { polygon, area };
}
return largest?.polygon ?? null;
}
export async function resolveNodeGeometry(
cdp: CdpRunner,
tabId: number,
address: NodeAddress,
options: { scrollIntoView?: boolean } = {},
): Promise<ResolvedNodeGeometry | RpcError> {
try {
if (address.target.sessionId && !address.frameId) {
return geometryError("an OOPIF node address requires frameId");
}
const graph = address.frameId ? await loadFrameGraph(cdp, tabId) : null;
if (address.frameId && !graph) {
return geometryError(`could not resolve frame graph for ${address.frameId}`);
}
if (options.scrollIntoView) {
const scrollError = await scrollElementWithFrameGraph(
cdp,
tabId,
address.target,
address.backendNodeId,
address.frameId,
graph,
);
if (scrollError) return scrollError;
}
const localRegion = await nodeContentRegion(
cdpRunnerForTarget(cdp, address.target),
tabId,
address.backendNodeId,
);
if (isRpcError(localRegion)) return localRegion;
let topVisibleRegions: Region;
if (address.frameId && graph) {
const projection = await resolveFrameProjection(cdp, graph, address.frameId);
if (!projection)
return geometryError(`could not resolve frame geometry for ${address.frameId}`);
topVisibleRegions = projectRegionToViewport(localRegion, projection);
} else {
const viewport = await targetViewport(cdp, address.target);
if (!viewport) return geometryError("could not resolve top viewport geometry");
topVisibleRegions = localRegion
.map((polygon) =>
clipPolygon(polygon, rectPolygon({ x: 0, y: 0, w: viewport.width, h: viewport.height })),
)
.filter((polygon) => polygon.length >= 3);
}
const topBounds = regionBounds(topVisibleRegions);
const actionRegion = largestRegion(topVisibleRegions);
const actionPoint = actionRegion ? polygonCentroid(actionRegion) : null;
if (!topBounds || !actionPoint) {
return { code: "permission_denied", message: "element not visible" };
}
return { topVisibleRegions, topBounds, actionPoint };
} catch (error) {
return geometryError(error instanceof Error ? error.message : String(error));
}
}
+266
View File
@@ -0,0 +1,266 @@
export interface Point {
x: number;
y: number;
}
export type Polygon = Point[];
export type Region = Polygon[];
export type Quad = [Point, Point, Point, Point];
export interface Size {
width: number;
height: number;
}
export interface ViewportRect {
x: number;
y: number;
width: number;
height: number;
}
export interface ProjectiveEdge {
sourceViewport: Size;
destinationQuad: Quad;
destinationClips?: Polygon[];
}
export interface GeometryProjection {
sourceClips: Polygon[];
edges: ProjectiveEdge[];
topViewport: Size;
}
export function parseCdpQuad(raw: number[] | undefined): Quad | null {
if (!raw || raw.length !== 8 || !raw.every(Number.isFinite)) return null;
return [
{ x: raw[0], y: raw[1] },
{ x: raw[2], y: raw[3] },
{ x: raw[4], y: raw[5] },
{ x: raw[6], y: raw[7] },
];
}
export function rectPolygon(rect: { x: number; y: number; w: number; h: number }): Quad {
return [
{ x: rect.x, y: rect.y },
{ x: rect.x + rect.w, y: rect.y },
{ x: rect.x + rect.w, y: rect.y + rect.h },
{ x: rect.x, y: rect.y + rect.h },
];
}
export function viewportPolygon(viewport: Size): Quad {
return rectPolygon({ x: 0, y: 0, w: viewport.width, h: viewport.height });
}
/** Project a normalized unit-square point into an arbitrary content quad. */
export function projectUnitPoint(point: Point, quad: Quad): Point {
const [p0, p1, p2, p3] = quad;
const dx1 = p1.x - p2.x;
const dx2 = p3.x - p2.x;
const dx3 = p0.x - p1.x + p2.x - p3.x;
const dy1 = p1.y - p2.y;
const dy2 = p3.y - p2.y;
const dy3 = p0.y - p1.y + p2.y - p3.y;
let a: number;
let b: number;
let c: number;
let d: number;
let e: number;
let f: number;
let g = 0;
let h = 0;
if (Math.abs(dx3) < 1e-9 && Math.abs(dy3) < 1e-9) {
a = p1.x - p0.x;
b = p3.x - p0.x;
c = p0.x;
d = p1.y - p0.y;
e = p3.y - p0.y;
f = p0.y;
} else {
const denominator = dx1 * dy2 - dx2 * dy1;
if (Math.abs(denominator) < 1e-9) {
return {
x: p0.x + (p1.x - p0.x) * point.x + (p3.x - p0.x) * point.y,
y: p0.y + (p1.y - p0.y) * point.x + (p3.y - p0.y) * point.y,
};
}
g = (dx3 * dy2 - dx2 * dy3) / denominator;
h = (dx1 * dy3 - dx3 * dy1) / denominator;
a = p1.x - p0.x + g * p1.x;
b = p3.x - p0.x + h * p3.x;
c = p0.x;
d = p1.y - p0.y + g * p1.y;
e = p3.y - p0.y + h * p3.y;
f = p0.y;
}
const scale = g * point.x + h * point.y + 1;
return {
x: (a * point.x + b * point.y + c) / scale,
y: (d * point.x + e * point.y + f) / scale,
};
}
export function projectPolygon(polygon: Polygon, edge: ProjectiveEdge): Polygon {
const { width, height } = edge.sourceViewport;
if (width <= 0 || height <= 0) return [];
return polygon.map((point) =>
projectUnitPoint({ x: point.x / width, y: point.y / height }, edge.destinationQuad),
);
}
function polygonSignedArea(points: Polygon): number {
let area = 0;
for (let index = 0; index < points.length; index += 1) {
const current = points[index];
const next = points[(index + 1) % points.length];
area += current.x * next.y - next.x * current.y;
}
return area / 2;
}
function edgeIntersection(start: Point, end: Point, clipStart: Point, clipEnd: Point): Point {
const subjectX = end.x - start.x;
const subjectY = end.y - start.y;
const clipX = clipEnd.x - clipStart.x;
const clipY = clipEnd.y - clipStart.y;
const denominator = subjectX * clipY - subjectY * clipX;
if (Math.abs(denominator) < 1e-9) return end;
const offsetX = clipStart.x - start.x;
const offsetY = clipStart.y - start.y;
const scale = (offsetX * clipY - offsetY * clipX) / denominator;
return { x: start.x + scale * subjectX, y: start.y + scale * subjectY };
}
/** Clip a polygon to a convex polygon, preserving every resulting vertex. */
export function clipPolygon(subject: Polygon, clip: Polygon): Polygon {
if (subject.length < 3 || clip.length < 3) return [];
const orientation = polygonSignedArea(clip) >= 0 ? 1 : -1;
let output = [...subject];
for (let edge = 0; edge < clip.length && output.length > 0; edge += 1) {
const clipStart = clip[edge];
const clipEnd = clip[(edge + 1) % clip.length];
const input = output;
output = [];
const inside = (point: Point) =>
orientation *
((clipEnd.x - clipStart.x) * (point.y - clipStart.y) -
(clipEnd.y - clipStart.y) * (point.x - clipStart.x)) >=
-1e-9;
let start = input[input.length - 1];
for (const end of input) {
const startInside = inside(start);
const endInside = inside(end);
if (endInside) {
if (!startInside) output.push(edgeIntersection(start, end, clipStart, clipEnd));
output.push(end);
} else if (startInside) {
output.push(edgeIntersection(start, end, clipStart, clipEnd));
}
start = end;
}
}
return output;
}
export function polygonBounds(points: Polygon): ViewportRect | null {
if (points.length < 3) return null;
const xs = points.map((point) => point.x);
const ys = points.map((point) => point.y);
const x = Math.min(...xs);
const y = Math.min(...ys);
const right = Math.max(...xs);
const bottom = Math.max(...ys);
if (right <= x || bottom <= y) return null;
return { x, y, width: right - x, height: bottom - y };
}
export function regionBounds(region: Region): ViewportRect | null {
return polygonBounds(region.flat());
}
export function polygonArea(points: Polygon): number {
return Math.abs(polygonSignedArea(points));
}
export function polygonCentroid(points: Polygon): Point | null {
if (points.length < 3) return null;
let twiceArea = 0;
let x = 0;
let y = 0;
for (let index = 0; index < points.length; index += 1) {
const current = points[index];
const next = points[(index + 1) % points.length];
const cross = current.x * next.y - next.x * current.y;
twiceArea += cross;
x += (current.x + next.x) * cross;
y += (current.y + next.y) * cross;
}
if (Math.abs(twiceArea) < 1e-9) return null;
return { x: x / (3 * twiceArea), y: y / (3 * twiceArea) };
}
export function projectAndClipRegion(
region: Region,
edges: ProjectiveEdge[],
topViewport: Size,
): Region {
let projected = region;
for (const edge of edges) {
const sourceClip = viewportPolygon(edge.sourceViewport);
projected = projected
.map((polygon) => clipPolygon(polygon, sourceClip))
.filter((polygon) => polygon.length >= 3)
.map((polygon) =>
[edge.destinationQuad, ...(edge.destinationClips ?? [])].reduce(
(current, clip) => clipPolygon(current, clip),
projectPolygon(polygon, edge),
),
)
.filter((polygon) => polygon.length >= 3);
}
const topClip = viewportPolygon(topViewport);
return projected
.map((polygon) => clipPolygon(polygon, topClip))
.filter((polygon) => polygon.length >= 3);
}
export function projectRegionToViewport(region: Region, projection: GeometryProjection): Region {
const clipped = region
.map((polygon) =>
projection.sourceClips.reduce((current, clip) => clipPolygon(current, clip), polygon),
)
.filter((polygon) => polygon.length >= 3);
return projectAndClipRegion(clipped, projection.edges, projection.topViewport);
}
export function projectRectToViewport(
rect: { x: number; y: number; w: number; h: number } | null,
projection: GeometryProjection,
): ViewportRect | null {
if (!rect) return null;
return regionBounds(projectRegionToViewport([rectPolygon(rect)], projection));
}
export function childFrameProjection(
parent: GeometryProjection,
ownerRectInParent: { x: number; y: number; w: number; h: number },
): GeometryProjection {
const destinationQuad = rectPolygon(ownerRectInParent);
return {
sourceClips: [],
edges: [
{
sourceViewport: { width: ownerRectInParent.w, height: ownerRectInParent.h },
destinationQuad,
destinationClips: parent.sourceClips,
},
...parent.edges,
],
topViewport: parent.topViewport,
};
}
+62 -68
View File
@@ -8,6 +8,7 @@ import {
HELP_REQUEST,
type HelpCancelMessage,
type HelpFinishMessage,
type HelpHighlightRect,
type HelpQueryResponse,
type HelpRequestMessage,
isHelpAckMessage,
@@ -25,6 +26,7 @@ import type {
ResolvedTarget,
RpcError,
} from "@/transport/types";
import { resolveNodeGeometry } from "./frame-geometry";
import {
type CdpRunner,
type ChromeTabsApi,
@@ -37,7 +39,6 @@ import {
import { lookupSnapshotRef } from "./snapshot-ref";
const DEFAULT_HELP_TIMEOUT_MS = 300_000;
const HELP_ATTR = "data-bsk-help";
const HELP_SEND_RETRIES = 3;
const HELP_SEND_RETRY_DELAY_MS = 350;
const HELP_REARM_DEBOUNCE_MS = 150;
@@ -74,6 +75,7 @@ interface ActiveHelpRequest {
title?: string;
targets: HelpTarget[];
selectors: string[];
rects: HelpHighlightRect[];
timeoutMs: number;
notificationId: string;
resolvedTargets?: ResolvedTarget[];
@@ -134,6 +136,7 @@ function helpRequestMessage(help: ActiveHelpRequest, tabId: number): HelpRequest
...(help.title ? { title: help.title } : {}),
displayMode: isPrimaryTab ? "full" : "compact",
selectors: isPrimaryTab ? help.selectors : [],
rects: isPrimaryTab ? help.rects : [],
timeoutMs: help.timeoutMs,
};
}
@@ -165,33 +168,6 @@ async function sendHelpRequestWithAck(
throw lastError ?? new Error("failed to show help overlay");
}
async function tagRefTarget(
cdp: CdpRunner,
tabId: number,
backendNodeId: number,
index: number,
): Promise<string | null> {
let objectId: string | undefined;
try {
const resolved = await cdp.send<{ object?: { objectId?: string } }>(tabId, "DOM.resolveNode", {
backendNodeId,
});
objectId = resolved.object?.objectId;
if (!objectId) return null;
await cdp.send(tabId, "Runtime.callFunctionOn", {
objectId,
functionDeclaration: `function(){ this.setAttribute("${HELP_ATTR}", "${index}"); }`,
});
return `[${HELP_ATTR}="${index}"]`;
} catch {
return null;
} finally {
if (objectId) {
await cdp.send(tabId, "Runtime.releaseObject", { objectId }).catch(() => {});
}
}
}
async function selectorExists(
cdp: CdpRunner,
tabId: number,
@@ -209,36 +185,22 @@ async function selectorExists(
}
}
async function clearRefTags(cdp: CdpRunner | undefined, tabId: number): Promise<void> {
if (!cdp) return;
try {
const doc = await cdp.send<{ root?: { nodeId?: number } }>(tabId, "DOM.getDocument", {
depth: 0,
});
const rootId = doc.root?.nodeId;
if (rootId === undefined) return;
const { nodeIds } = await cdp.send<{ nodeIds: number[] }>(tabId, "DOM.querySelectorAll", {
nodeId: rootId,
selector: `[${HELP_ATTR}]`,
});
for (const nodeId of nodeIds ?? []) {
await cdp.send(tabId, "DOM.removeAttribute", { nodeId, name: HELP_ATTR }).catch(() => {});
}
} catch {
// Best-effort cleanup.
}
}
async function resolveHelpTargets(
ctx: ReturnType<SessionManager["get"]>,
tabId: number,
targets: HelpTarget[],
deps: RequestHelpDeps,
): Promise<{ selectors: string[]; resolvedTargets?: ResolvedTarget[] }> {
options: { scrollIntoView: boolean },
): Promise<{
selectors: string[];
rects: HelpHighlightRect[];
resolvedTargets?: ResolvedTarget[];
}> {
const selectors: string[] = [];
const rects: HelpHighlightRect[] = [];
const resolved: ResolvedTarget[] = [];
let rootNodeId: number | undefined;
if (deps.cdp && targets.length > 0) {
if (deps.cdp && targets.some((target) => Boolean(target.selector))) {
try {
const doc = await deps.cdp.send<{ root?: { nodeId?: number } }>(tabId, "DOM.getDocument", {
depth: 0,
@@ -262,28 +224,55 @@ async function resolveHelpTargets(
resolved.push({ matched, selector: tgt.selector });
} else if (tgt.ref) {
const looked = ctx ? lookupSnapshotRef(ctx, tgt.ref, tabId) : null;
const backendNodeId = looked?.backendNodeId ?? null;
let sel: string | null = null;
if (backendNodeId !== null && deps.cdp) {
sel = await tagRefTarget(deps.cdp, tabId, backendNodeId, i);
let rect: HelpHighlightRect | null = null;
if (looked && deps.cdp) {
const target = {
tabId,
...(looked.cdpSessionId ? { sessionId: looked.cdpSessionId } : {}),
};
const geometry = await resolveNodeGeometry(
deps.cdp,
tabId,
{
target,
backendNodeId: looked.backendNodeId,
...(looked.frameId ? { frameId: looked.frameId } : {}),
},
{ scrollIntoView: options.scrollIntoView },
);
if (!isRpcError(geometry)) {
rect = {
top: geometry.topBounds.y,
left: geometry.topBounds.x,
width: geometry.topBounds.width,
height: geometry.topBounds.height,
};
}
}
if (sel) selectors.push(sel);
resolved.push({ matched: sel !== null, ref: tgt.ref });
if (rect) rects.push(rect);
resolved.push({
matched: rect !== null,
ref: tgt.ref,
});
}
}
return { selectors, resolvedTargets: resolved.length > 0 ? resolved : undefined };
return { selectors, rects, resolvedTargets: resolved.length > 0 ? resolved : undefined };
}
async function refreshHelpTargets(help: ActiveHelpRequest): Promise<void> {
await clearRefTags(help.deps.cdp, help.primaryTabId);
const { selectors, resolvedTargets } = await resolveHelpTargets(
async function refreshHelpTargets(
help: ActiveHelpRequest,
options: { scrollIntoView: boolean } = { scrollIntoView: true },
): Promise<void> {
const { selectors, rects, resolvedTargets } = await resolveHelpTargets(
help.ctx,
help.primaryTabId,
help.targets,
help.deps,
options,
);
help.selectors = selectors;
help.rects = rects;
help.resolvedTargets = resolvedTargets;
}
@@ -294,12 +283,7 @@ async function cleanupHelp(help: ActiveHelpRequest): Promise<void> {
const tabsToCancel = new Set([help.primaryTabId, ...help.overlayTabIds]);
await Promise.all(
[...tabsToCancel].map((tabId) =>
Promise.all([
clearRefTags(help.deps.cdp, tabId),
help.deps
.sendToTab(tabId, { type: HELP_CANCEL, requestId: help.requestId })
.catch(() => {}),
]),
help.deps.sendToTab(tabId, { type: HELP_CANCEL, requestId: help.requestId }).catch(() => {}),
),
);
}
@@ -476,7 +460,9 @@ function attachHelpRuntimeListener(deps: RequestHelpDeps): () => void {
sendResponse({ active: false });
return;
}
if (tabId === help.primaryTabId) await refreshHelpTargets(help);
if (tabId === help.primaryTabId) {
await refreshHelpTargets(help, { scrollIntoView: false });
}
help.overlayTabIds.add(tabId);
sendResponse({
active: true,
@@ -486,6 +472,7 @@ function attachHelpRuntimeListener(deps: RequestHelpDeps): () => void {
...(help.title ? { title: help.title } : {}),
displayMode: tabId === help.primaryTabId ? "full" : "compact",
selectors: tabId === help.primaryTabId ? help.selectors : [],
rects: tabId === help.primaryTabId ? help.rects : [],
timeoutMs: help.timeoutMs,
},
});
@@ -686,7 +673,13 @@ export async function handleRequestHelp(
if (deps.autoAttachLifecycle !== false) ensureHelpLifecycleListeners(deps);
const tabId = target.tabId;
const initialTargets = params.targets ?? [];
const { selectors, resolvedTargets } = await resolveHelpTargets(ctx, tabId, initialTargets, deps);
const { selectors, rects, resolvedTargets } = await resolveHelpTargets(
ctx,
tabId,
initialTargets,
deps,
{ scrollIntoView: true },
);
await deps.windows.update(target.windowId, { focused: true }).catch(() => {});
await deps.activateTab(tabId).catch(() => {});
@@ -729,6 +722,7 @@ export async function handleRequestHelp(
...(params.title ? { title: params.title } : {}),
targets: initialTargets,
selectors,
rects,
timeoutMs,
notificationId,
resolvedTargets,
+88 -56
View File
@@ -3,14 +3,15 @@
//
// All interaction tools:
// 1. Resolve target tab (sandbox: must be inside Agent Window).
// 2. Resolve target element by `ref` (RefStore.resolve with tabId
// binding) or `selector` (DOM.querySelector + describeNode).
// 3. Scroll the node into view, then dispatch the appropriate
// `Input.*` CDP events.
// 2. Resolve target element by `ref` (compound frame/session identity)
// or `selector` (DOM.querySelector + describeNode).
// 3. Scroll the node and its frame owners into view, project its live
// content quad into the top viewport, then dispatch `Input.*` events.
// 4. Honour `AbortSignal` so canceled calls don't issue follow-up CDP
// commands.
import { ChromiumCdp } from "@/browser-driver/chromium-cdp";
import type { CdpTarget } from "@/browser-driver/frame-graph";
import type { SessionContext, SessionManager } from "@/session-manager/manager";
import type {
ClickParams,
@@ -28,17 +29,13 @@ import type {
SelectResult,
} from "@/transport/types";
import { attachDialogs, markDialogCursor } from "./dialogs";
import {
backendNodeToObject,
boxCentre,
nodeCentre,
quadCentre,
scrollNodeIntoView,
} from "./element-geometry";
import { backendNodeToObject } from "./element-geometry";
import { rpcError } from "./errors";
import { resolveNodeGeometry, scrollElementAndFramesIntoView } from "./frame-geometry";
import {
type CdpRunner,
type ChromeTabsApi,
cdpRunnerForTarget,
chromeTabsApi,
enforceAgentWindow,
isRpcError,
@@ -137,7 +134,16 @@ async function resolveBackendNode(
target: { tabId: number },
params: { ref?: string; selector?: string },
toolName: string,
): Promise<{ backendNodeId: number; usedRef?: string; usedSelector?: string } | RpcError> {
): Promise<
| {
backendNodeId: number;
cdpTarget: CdpTarget;
frameId?: string;
usedRef?: string;
usedSelector?: string;
}
| RpcError
> {
const hasRef = typeof params.ref === "string" && params.ref.length > 0;
const hasSelector = typeof params.selector === "string" && params.selector.length > 0;
if (hasRef && hasSelector) {
@@ -155,7 +161,15 @@ async function resolveBackendNode(
if (hasRef) {
const resolved = resolveSnapshotRef(ctx, params.ref as string, target.tabId);
if (isRpcError(resolved)) return resolved;
return { backendNodeId: resolved.backendNodeId, usedRef: resolved.refKey };
return {
backendNodeId: resolved.backendNodeId,
cdpTarget: {
tabId: target.tabId,
...(resolved.cdpSessionId ? { sessionId: resolved.cdpSessionId } : {}),
},
...(resolved.frameId ? { frameId: resolved.frameId } : {}),
usedRef: resolved.refKey,
};
}
// selector path
try {
@@ -192,7 +206,7 @@ async function resolveBackendNode(
message: "DOM.describeNode returned no backendNodeId",
};
}
return { backendNodeId, usedSelector: params.selector };
return { backendNodeId, cdpTarget: { tabId: target.tabId }, usedSelector: params.selector };
} catch (err) {
return {
code: "cdp_failed",
@@ -228,19 +242,19 @@ export async function handleClick(
return { code: "cancelled", message: "click aborted" };
}
try {
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
const scrollErr = await scrollNodeIntoView(deps.cdp, target.tabId, node.backendNodeId);
if (scrollErr) return scrollErr;
} catch (err) {
return {
code: "cdp_failed",
message: err instanceof Error ? err.message : String(err),
};
}
const centre = await nodeCentre(deps.cdp, target.tabId, node.backendNodeId);
if (isRpcError(centre)) return centre;
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
const geometry = await resolveNodeGeometry(
deps.cdp,
target.tabId,
{
target: node.cdpTarget,
backendNodeId: node.backendNodeId,
...(node.frameId ? { frameId: node.frameId } : {}),
},
{ scrollIntoView: true },
);
if (isRpcError(geometry)) return geometry;
const centre = geometry.actionPoint;
if (throwIfAborted(deps.signal)) {
return { code: "cancelled", message: "click aborted" };
@@ -353,19 +367,19 @@ export async function handleHover(
const node = await resolveBackendNode(deps.cdp, ctx, target, params, "hover");
if (isRpcError(node)) return node;
try {
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
const scrollErr = await scrollNodeIntoView(deps.cdp, target.tabId, node.backendNodeId);
if (scrollErr) return scrollErr;
} catch (err) {
return {
code: "cdp_failed",
message: err instanceof Error ? err.message : String(err),
};
}
const centre = await nodeCentre(deps.cdp, target.tabId, node.backendNodeId);
if (isRpcError(centre)) return centre;
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
const geometry = await resolveNodeGeometry(
deps.cdp,
target.tabId,
{
target: node.cdpTarget,
backendNodeId: node.backendNodeId,
...(node.frameId ? { frameId: node.frameId } : {}),
},
{ scrollIntoView: true },
);
if (isRpcError(geometry)) return geometry;
const centre = geometry.actionPoint;
if (throwIfAborted(deps.signal)) {
return { code: "cancelled", message: "hover aborted" };
@@ -585,10 +599,11 @@ export async function handleFill(
const node = await resolveBackendNode(deps.cdp, ctx, target, params, "fill");
if (isRpcError(node)) return node;
const nodeCdp = cdpRunnerForTarget(deps.cdp, node.cdpTarget);
try {
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
const described = await deps.cdp.send<{ node?: DescribedNode }>(
const described = await nodeCdp.send<{ node?: DescribedNode }>(
target.tabId,
"DOM.describeNode",
{
@@ -602,12 +617,18 @@ export async function handleFill(
`element ${described.node?.nodeName ?? "?"} not fillable (need input/textarea/contenteditable)`,
);
}
const scrollErr = await scrollNodeIntoView(deps.cdp, target.tabId, node.backendNodeId);
const scrollErr = await scrollElementAndFramesIntoView(
deps.cdp,
target.tabId,
node.cdpTarget,
node.backendNodeId,
node.frameId,
);
if (scrollErr) return scrollErr;
if (throwIfAborted(deps.signal)) {
return { code: "cancelled", message: "fill aborted" };
}
await deps.cdp.send(target.tabId, "DOM.focus", { backendNodeId: node.backendNodeId });
await nodeCdp.send(target.tabId, "DOM.focus", { backendNodeId: node.backendNodeId });
} catch (err) {
return {
code: "cdp_failed",
@@ -619,7 +640,7 @@ export async function handleFill(
return { code: "cancelled", message: "fill aborted" };
}
const objectIdOrErr = await backendNodeToObject(deps.cdp, target.tabId, node.backendNodeId);
const objectIdOrErr = await backendNodeToObject(nodeCdp, target.tabId, node.backendNodeId);
if (isRpcError(objectIdOrErr)) return objectIdOrErr;
const objectId = objectIdOrErr;
const clearBefore = params.clear_before ?? true;
@@ -628,7 +649,7 @@ export async function handleFill(
if (clearBefore) {
// Clear input/textarea value or wipe contenteditable innerText,
// then fire `input` so frameworks observe the empty state.
await deps.cdp.send(target.tabId, "Runtime.callFunctionOn", {
await nodeCdp.send(target.tabId, "Runtime.callFunctionOn", {
objectId,
functionDeclaration: `function() {
if (this.isContentEditable) { this.textContent = ''; }
@@ -655,7 +676,7 @@ export async function handleFill(
return { code: "cancelled", message: "fill aborted" };
}
// Fire `input` + `change` so React / Vue controlled inputs commit.
await deps.cdp.send(target.tabId, "Runtime.callFunctionOn", {
await nodeCdp.send(target.tabId, "Runtime.callFunctionOn", {
objectId,
functionDeclaration: `function() {
this.dispatchEvent(new Event('input', { bubbles: true }));
@@ -853,14 +874,21 @@ export async function handlePress(
if (params.ref || params.selector) {
const node = await resolveBackendNode(deps.cdp, ctx, target, params, "press");
if (isRpcError(node)) return node;
const nodeCdp = cdpRunnerForTarget(deps.cdp, node.cdpTarget);
try {
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
const scrollErr = await scrollNodeIntoView(deps.cdp, target.tabId, node.backendNodeId);
const scrollErr = await scrollElementAndFramesIntoView(
deps.cdp,
target.tabId,
node.cdpTarget,
node.backendNodeId,
node.frameId,
);
if (scrollErr) return scrollErr;
if (throwIfAborted(deps.signal)) {
return { code: "cancelled", message: "press aborted" };
}
await deps.cdp.send(target.tabId, "DOM.focus", { backendNodeId: node.backendNodeId });
await nodeCdp.send(target.tabId, "DOM.focus", { backendNodeId: node.backendNodeId });
} catch (err) {
return {
code: "cdp_failed",
@@ -965,10 +993,11 @@ export async function handleSelect(
const node = await resolveBackendNode(deps.cdp, ctx, target, params, "select");
if (isRpcError(node)) return node;
const nodeCdp = cdpRunnerForTarget(deps.cdp, node.cdpTarget);
try {
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
const described = await deps.cdp.send<{ node?: DescribedNode }>(
const described = await nodeCdp.send<{ node?: DescribedNode }>(
target.tabId,
"DOM.describeNode",
{ backendNodeId: node.backendNodeId },
@@ -991,12 +1020,18 @@ export async function handleSelect(
"single-select <select> requires exactly one value",
);
}
const scrollErr = await scrollNodeIntoView(deps.cdp, target.tabId, node.backendNodeId);
const scrollErr = await scrollElementAndFramesIntoView(
deps.cdp,
target.tabId,
node.cdpTarget,
node.backendNodeId,
node.frameId,
);
if (scrollErr) return scrollErr;
if (throwIfAborted(deps.signal)) {
return { code: "cancelled", message: "select aborted" };
}
await deps.cdp.send(target.tabId, "DOM.focus", { backendNodeId: node.backendNodeId });
await nodeCdp.send(target.tabId, "DOM.focus", { backendNodeId: node.backendNodeId });
} catch (err) {
return {
code: "cdp_failed",
@@ -1008,12 +1043,12 @@ export async function handleSelect(
return { code: "cancelled", message: "select aborted" };
}
const objectIdOrErr = await backendNodeToObject(deps.cdp, target.tabId, node.backendNodeId);
const objectIdOrErr = await backendNodeToObject(nodeCdp, target.tabId, node.backendNodeId);
if (isRpcError(objectIdOrErr)) return objectIdOrErr;
const objectId = objectIdOrErr;
try {
const evaluated = await deps.cdp.send<{
const evaluated = await nodeCdp.send<{
result?: { value?: SelectMutationResult | null };
}>(target.tabId, "Runtime.callFunctionOn", {
objectId,
@@ -1072,9 +1107,6 @@ export async function handleSelect(
export const __testing__ = {
DEFAULT_TIMEOUT_MS,
quadCentre,
boxCentre,
resolveBackendNode,
nodeCentre,
isFillable,
};
+136 -64
View File
@@ -10,9 +10,11 @@ import {
isVomReferenceNode,
renderVom,
type VomNode,
type VomOptions,
type VomScene,
} from "@browser-skill/vom";
import { ChromiumCdp } from "@/browser-driver/chromium-cdp";
import type { CdpTarget } from "@/browser-driver/frame-graph";
import {
type CaptureSuppressSendToTab,
withOverlaysHiddenForCapture,
@@ -30,8 +32,8 @@ import type {
SnapshotResult,
} from "@/transport/types";
import { attachDialogs, markDialogCursor } from "./dialogs";
import { nodeBoundingRect, scrollNodeIntoView } from "./element-geometry";
import { rpcError } from "./errors";
import { resolveNodeGeometry } from "./frame-geometry";
import {
type ChromeTabsApi,
enforceToolTargetScope,
@@ -40,6 +42,7 @@ import {
type ResolvedTargetTab,
resolveCdpAccessibleTargetTab,
type CdpRunner as SharedCdpRunner,
sendToCdpTarget,
normaliseRef as sharedNormaliseRef,
type ToolEffect,
} from "./shared";
@@ -51,6 +54,10 @@ import {
collectOverlayExcludedBackendIds,
} from "./vom/capture";
import { type CapturedFrameDocument, captureFrameData } from "./vom/frame-capture";
import {
type CaptureVomObservationResult,
projectRecordSafeObservation,
} from "./vom/record-safe-observation";
// ---------------------------------------------------------------------------
// Shared helpers (legacy aliases — observation.ts kept exporting these
@@ -163,26 +170,30 @@ function throwIfAborted(signal: AbortSignal | undefined, tool: string): void {
async function captureElementScreenshot(
cdp: SharedCdpRunner,
tabId: number,
target: CdpTarget,
backendNodeId: number,
frameId?: string,
signal?: AbortSignal,
): Promise<{ image_base64: string; width: number; height: number } | RpcError> {
if (signal?.aborted) return cancelled("screenshot");
const scrollErr = await scrollNodeIntoView(cdp, tabId, backendNodeId);
if (scrollErr) return scrollErr;
if (signal?.aborted) return cancelled("screenshot");
const rectOrErr = await nodeBoundingRect(cdp, tabId, backendNodeId);
if (isRpcError(rectOrErr)) return rectOrErr;
const geometry = await resolveNodeGeometry(
cdp,
tabId,
{ target, backendNodeId, ...(frameId ? { frameId } : {}) },
{ scrollIntoView: true },
);
if (isRpcError(geometry)) return geometry;
if (signal?.aborted) return cancelled("screenshot");
const rect = geometry.topBounds;
try {
const shot = await cdp.send<{ data?: string }>(tabId, "Page.captureScreenshot", {
format: "png",
clip: {
x: rectOrErr.x,
y: rectOrErr.y,
width: rectOrErr.width,
height: rectOrErr.height,
x: rect.x,
y: rect.y,
width: rect.width,
height: rect.height,
scale: 1,
},
});
@@ -192,8 +203,8 @@ async function captureElementScreenshot(
return { code: "cdp_failed", message: "Page.captureScreenshot returned no data" };
}
const dims = parsePngDimensions(image_base64) ?? {
width: Math.round(rectOrErr.width),
height: Math.round(rectOrErr.height),
width: Math.round(rect.width),
height: Math.round(rect.height),
};
return { image_base64, width: dims.width, height: dims.height };
} catch (err) {
@@ -297,9 +308,21 @@ export async function handleScreenshot(
await deps.cdp.ensureAttachedToUrl?.(target.tabId, target.url);
if (signal?.aborted) return cancelled("screenshot");
const cdp = deps.cdp;
const nodeTarget = {
tabId: target.tabId,
...(node.cdpSessionId ? { sessionId: node.cdpSessionId } : {}),
};
const captured = await withOverlaysHiddenForCapture(
target.tabId,
() => captureElementScreenshot(cdp, target.tabId, node.backendNodeId, signal),
() =>
captureElementScreenshot(
cdp,
target.tabId,
nodeTarget,
node.backendNodeId,
node.frameId,
signal,
),
deps.sendToTab,
);
if (isRpcError(captured)) return captured;
@@ -393,7 +416,15 @@ function isModalSignal(
function isSensitive(capturedNode: CapturedNode | undefined): boolean {
const attrs = capturedNode?.attrs ?? {};
return (attrs.type ?? "").toLowerCase() === "password";
const type = (attrs.type ?? "").toLowerCase();
if (type === "password") return true;
const autocomplete = (attrs.autocomplete ?? "").toLowerCase();
return (
autocomplete.startsWith("cc-") ||
autocomplete === "one-time-code" ||
autocomplete === "current-password" ||
autocomplete === "new-password"
);
}
interface AxNodeSignals {
@@ -443,7 +474,13 @@ const AX_TEXT_STOP_ROLES = new Set([
"columnheader",
"rowheader",
]);
const SENSITIVE_AX_INPUT_TYPES = new Set(["password", "credit-card"]);
const SENSITIVE_AX_INPUT_TYPES = new Set([
"password",
"credit-card",
"one-time-code",
"current-password",
"new-password",
]);
function axPropertyString(axNode: CdpAxNode, name: string): string | undefined {
const prop = axNode.properties?.find((item) => item.name === name);
@@ -1280,7 +1317,7 @@ export function buildFrameVomScene(
excludedBackendNodeIds:
frameId === rootFrameId ? captured.excludedBackendNodeIds : new Set(),
},
{ pageUrl: options.pageUrl },
{ pageUrl: document.url ?? options.pageUrl },
),
);
}
@@ -1452,9 +1489,15 @@ export async function handleGetHtml(
if (params.ref) {
const resolved = resolveSnapshotRef(ctx, params.ref, target.tabId);
if (isRpcError(resolved)) return resolved;
const resp = await deps.cdp.send<{ outerHTML?: string }>(target.tabId, "DOM.getOuterHTML", {
backendNodeId: resolved.backendNodeId,
});
const resp = await sendToCdpTarget<{ outerHTML?: string }>(
deps.cdp,
{
tabId: target.tabId,
...(resolved.cdpSessionId ? { sessionId: resolved.cdpSessionId } : {}),
},
"DOM.getOuterHTML",
{ backendNodeId: resolved.backendNodeId },
);
throwIfAborted(signal, "get_html");
html = resp.outerHTML ?? "";
} else {
@@ -1513,14 +1556,13 @@ async function fallbackCapturedViewModel(
try {
const metrics = await cdp.send<LayoutMetricsViewportReply>(tabId, "Page.getLayoutMetrics", {});
throwIfAborted(signal, "observation");
const vpSrc = metrics.cssLayoutViewport ?? metrics.layoutViewport ?? {};
const source = metrics.cssLayoutViewport ?? metrics.layoutViewport ?? {};
viewport = {
width: vpSrc.clientWidth ?? 0,
height: vpSrc.clientHeight ?? 0,
width: source.clientWidth ?? 0,
height: source.clientHeight ?? 0,
};
} catch (err) {
if (isAbortError(err)) throw err;
// viewport stays zero-sized
} catch (error) {
if (isAbortError(error)) throw error;
}
const excludedBackendNodeIds = await collectOverlayExcludedBackendIds(cdp, tabId, signal);
throwIfAborted(signal, "observation");
@@ -1530,22 +1572,58 @@ async function fallbackCapturedViewModel(
async function captureForVom(
cdp: CdpRunner,
tabId: number,
conditionalSurfaceProbe: boolean,
hoverProbeBypassOverlay?: (tabId: number, enabled: boolean) => Promise<void>,
signal?: AbortSignal,
options: CaptureVomObservationOptions,
): Promise<CapturedViewModel> {
try {
return await captureViewModel(cdp, tabId, {
conditionalSurfaceProbe,
hoverProbeBypassOverlay,
signal,
conditionalSurfaceProbe: options.conditionalSurfaceProbe ?? false,
hoverProbeBypassOverlay: options.hoverProbeBypassOverlay,
signal: options.signal,
});
} catch (err) {
if (isAbortError(err)) throw err;
return fallbackCapturedViewModel(cdp, tabId, signal);
} catch (error) {
if (isAbortError(error)) throw error;
return fallbackCapturedViewModel(cdp, tabId, options.signal);
}
}
export interface CaptureVomObservationOptions extends VomOptions {
conditionalSurfaceProbe?: boolean;
hoverProbeBypassOverlay?: (tabId: number, enabled: boolean) => Promise<void>;
signal?: AbortSignal;
}
export async function captureVomObservation(
cdp: CdpRunner,
tabId: number,
url: string | undefined,
options: CaptureVomObservationOptions = {},
): Promise<CaptureVomObservationResult> {
throwIfAborted(options.signal, "observation");
await cdp.ensureAttachedToUrl?.(tabId, url);
throwIfAborted(options.signal, "observation");
const captured = await captureForVom(cdp, tabId, options);
throwIfAborted(options.signal, "observation");
const documents = await captureFrameData<CdpAxNode>(cdp, tabId, captured, options.signal);
throwIfAborted(options.signal, "observation");
const scene =
captured.rootFrameId && captured.frameNodes?.size
? buildFrameVomScene(documents, captured, { pageUrl: url })
: buildVomScene(documents[0]?.axNodes ?? [], captured, { pageUrl: url });
const rendered = renderVom(scene, {
maxDepth: options.maxDepth,
maxTokens: options.maxTokens,
redactValues: options.redactValues,
activeRegionPolicy: options.activeRegionPolicy,
});
throwIfAborted(options.signal, "observation");
return projectRecordSafeObservation({
rootFrameId: captured.rootFrameId ?? documents[0]?.frameId ?? "root",
frameDocuments: documents,
rendered,
surfaceProbes: captured.surfaceProbes,
});
}
async function handleVomObservation(
manager: SessionManager,
params: SnapshotParams | ObserveParams,
@@ -1575,49 +1653,43 @@ async function handleVomObservation(
try {
throwIfAborted(signal, toolName);
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
await deps.cdp.ensureAttachedToUrl?.(target.tabId, target.url);
throwIfAborted(signal, toolName);
const effectiveConditionalSurfaceProbe =
deps.conditionalSurfaceProbe ?? conditionalSurfaceProbe;
const captured = await captureForVom(
deps.cdp,
target.tabId,
effectiveConditionalSurfaceProbe,
deps.hoverProbeBypassOverlay,
signal,
);
throwIfAborted(signal, toolName);
const frameDocuments = await captureFrameData<CdpAxNode>(
deps.cdp,
target.tabId,
captured,
signal,
);
throwIfAborted(signal, toolName);
const scene =
captured.rootFrameId && captured.frameNodes?.size
? buildFrameVomScene(frameDocuments, captured, { pageUrl: target.url })
: buildVomScene(frameDocuments[0]?.axNodes ?? [], captured, { pageUrl: target.url });
const rendered = renderVom(scene, {
const observation = await captureVomObservation(deps.cdp, target.tabId, target.url, {
maxDepth: params.max_depth,
maxTokens: params.max_tokens,
activeRegionPolicy: true,
conditionalSurfaceProbe: effectiveConditionalSurfaceProbe,
hoverProbeBypassOverlay: deps.hoverProbeBypassOverlay,
signal,
});
throwIfAborted(signal, toolName);
const targetByFrameId = new Map(
observation.frames.map((frame) => [frame.frameId, frame.target]),
);
ctx.refStore.replace(
rendered.refs.map(
(ref) => [ref.ref, { backendNodeId: ref.backendNodeId, tabId: target.tabId }] as const,
),
observation.refs.map((ref) => {
const refTarget = ref.frameId ? targetByFrameId.get(ref.frameId) : undefined;
return [
ref.ref,
{
backendNodeId: ref.backendNodeId,
tabId: target.tabId,
...(ref.frameId ? { frameId: ref.frameId } : {}),
...(refTarget?.sessionId ? { cdpSessionId: refTarget.sessionId } : {}),
},
] as const;
}),
);
return attachDialogs(deps.cdp, target.tabId, dialogCursor, {
text: rendered.text,
ref_count: rendered.refs.length,
text: observation.text,
ref_count: observation.refs.length,
tab_id: target.tabId,
truncated: rendered.truncated,
truncated: observation.truncated,
...(toolName === "observe" && (params as ObserveParams).debug_surfaces
? {
debug: {
surface_probes: (captured.surfaceProbes ?? []).map((probe) => ({
surface_probes: (observation.surfaceProbes ?? []).map((probe) => ({
trigger_backend_node_id: probe.triggerBackendNodeId,
...(probe.triggerPoint ? { trigger_point: probe.triggerPoint } : {}),
trigger_action: probe.triggerAction,
+18 -22
View File
@@ -17,11 +17,11 @@ import {
type RecordStartMessage,
type RecordStopMessage,
} from "@/lib/record-bridge";
import { appendRecordedPayload, observeRecordedNavigation } from "@/lib/recording-step-buffer";
import { reduceTraceSteps, resolveTraceStartUrl } from "@/lib/trace-reducer";
import { appendRecordedPayload, observeRecordedNavigation } from "@/lib/recording/step-buffer";
import { buildTraceV2 } from "@/lib/recording/trace-reducer-v2";
import type { RecordingDraftStep } from "@/lib/recording/types";
import type { SessionManager } from "@/session-manager/manager";
import type {
DraftTraceStep,
RecordAwaitParams,
RecordAwaitResult,
RecordStartParams,
@@ -29,7 +29,7 @@ import type {
RecordStopParams,
RecordStopResult,
RpcError,
Trace,
TraceV2,
} from "@/transport/types";
import { handleNavigate } from "./navigation";
import {
@@ -47,11 +47,11 @@ interface ActiveRecording {
agentWindowId: number;
startUrl?: string;
purpose?: string;
steps: DraftTraceStep[];
steps: RecordingDraftStep[];
startedAt: string;
startedAtMs: number;
finishPromise: Promise<Trace>;
resolveFinish: (trace: Trace) => void;
finishPromise: Promise<TraceV2>;
resolveFinish: (trace: TraceV2) => void;
rejectFinish: (err: Error) => void;
settled: boolean;
finishing: boolean;
@@ -154,17 +154,13 @@ async function sendRecordStartWithAck(
throw lastError ?? new Error("failed to start recording in content script");
}
function buildTrace(recording: ActiveRecording): Trace {
const { pages, steps } = reduceTraceSteps(recording.steps, recording.startUrl);
const startUrl = resolveTraceStartUrl(recording.steps, recording.startUrl, pages);
return {
function buildTrace(recording: ActiveRecording): TraceV2 {
return buildTraceV2({
steps: recording.steps,
startedAt: recording.startedAt,
...(recording.startUrl ? { startUrl: recording.startUrl } : {}),
...(recording.purpose ? { purpose: recording.purpose } : {}),
recorded_at: new Date().toISOString(),
started_at: recording.startedAt,
entry: { start_url: startUrl },
pages,
steps,
};
});
}
export interface RecordDeps {
@@ -532,7 +528,7 @@ async function finishRecordingByRequest(
}
}
async function finishRecording(sessionId: string, deps: RecordDeps): Promise<Trace | null> {
async function finishRecording(sessionId: string, deps: RecordDeps): Promise<TraceV2 | null> {
const recording = recordings.get(sessionId);
if (!recording || recording.settled || recording.finishing) return null;
recording.finishing = true;
@@ -574,9 +570,9 @@ export async function handleRecordStart(
// on the destination page can RECORD_QUERY → rearm → show RecordOverlay
// instead of flashing ControlOverlay ("Agent 正在控制").
const requestId = makeRequestId(target.tabId);
let resolveFinish!: (trace: Trace) => void;
let resolveFinish!: (trace: TraceV2) => void;
let rejectFinish!: (err: Error) => void;
const finishPromise = new Promise<Trace>((resolve, reject) => {
const finishPromise = new Promise<TraceV2>((resolve, reject) => {
resolveFinish = resolve;
rejectFinish = reject;
});
@@ -785,9 +781,9 @@ export async function handleRecordAwait(
return { code: "cancelled", message: "record_await aborted" };
}
const outcome = await new Promise<{ trace: Trace } | { error: RpcError }>((resolve) => {
const outcome = await new Promise<{ trace: TraceV2 } | { error: RpcError }>((resolve) => {
let settled = false;
const finish = (result: { trace: Trace } | { error: RpcError }) => {
const finish = (result: { trace: TraceV2 } | { error: RpcError }) => {
if (settled) return;
settled = true;
if (timer) clearTimeout(timer);
+20
View File
@@ -76,6 +76,7 @@ export interface SessionStopResult {
}
export interface SessionStopDeps {
signal?: AbortSignal;
cdp?: {
detachSession(sessionId: string): Promise<void>;
};
@@ -177,6 +178,9 @@ export async function handleSessionStop(
message: `session ${params.session_id} unknown`,
};
}
if (deps.signal?.aborted) {
return { code: "cancelled", message: "session_stop aborted before teardown" };
}
// Step 1: auto-return borrowed tabs. Iterate over a snapshot of the
// ids so deletions during iteration do not break the Map iterator.
@@ -187,6 +191,7 @@ export async function handleSessionStop(
try {
const tabManagement = {
...(deps.tabManagement ?? {}),
signal: deps.signal,
isAgentWindowId:
deps.tabManagement?.isAgentWindowId ??
((windowId: number) => manager.findByWindowId(windowId) !== null),
@@ -214,6 +219,17 @@ export async function handleSessionStop(
}
}
if (deps.signal?.aborted) {
const nonCancelFailures = returnFailures?.filter((failure) => failure.code !== "cancelled");
if (nonCancelFailures && nonCancelFailures.length > 0) {
return {
...(returnedTabIds.length > 0 ? { returned_tab_ids: returnedTabIds } : {}),
return_failures: nonCancelFailures,
};
}
return { code: "cancelled", message: "session_stop aborted during tab return" };
}
const result: SessionStopResult = {};
if (returnedTabIds.length > 0) result.returned_tab_ids = returnedTabIds;
if (returnFailures && returnFailures.length > 0) {
@@ -233,6 +249,10 @@ export async function handleSessionStop(
// Step 3: detach CDP sessions this session opened (no-op if none).
await deps.cdp?.detachSession(params.session_id);
if (deps.signal?.aborted) {
return { code: "cancelled", message: "session_stop aborted before window close" };
}
// Step 4: close the Agent Window and drop the context.
await manager.stop(params.session_id);
+24 -2
View File
@@ -6,7 +6,8 @@
// exactly the same sandbox + visibility rules as the M6 observation
// handlers (review parity).
import type { DialogCursor } from "@/browser-driver/chromium-cdp";
import type { CdpDebuggee, DialogCursor } from "@/browser-driver/chromium-cdp";
import type { CdpFrameGraph, CdpTarget } from "@/browser-driver/frame-graph";
import type { SessionContext, SessionManager } from "@/session-manager/manager";
import { normaliseRef } from "@/session-manager/ref-store";
import type { ConsoleResult, JavaScriptDialogInfo, RpcError } from "@/transport/types";
@@ -51,9 +52,11 @@ export type { DialogCursor };
*/
export interface CdpRunner {
send<T = unknown>(tabId: number, method: string, params?: object): Promise<T>;
sendToTarget?<T = unknown>(target: CdpTarget, method: string, params?: object): Promise<T>;
getFrameGraph?(tabId: number): Promise<CdpFrameGraph>;
ensureAttachedToUrl?(tabId: number, expectedUrl: string | undefined): Promise<void>;
trackSessionTab?(sessionId: string, tabId: number): void;
onEvent?(handler: (source: chrome.debugger.Debuggee, method: string, params: unknown) => void): {
onEvent?(handler: (source: CdpDebuggee, method: string, params: unknown) => void): {
dispose(): void;
};
dialogCursor?(tabId: number): DialogCursor;
@@ -68,6 +71,25 @@ export interface CdpRunner {
): ConsoleResult;
}
export function sendToCdpTarget<T = unknown>(
cdp: CdpRunner,
target: CdpTarget,
method: string,
params?: object,
): Promise<T> {
if (target.sessionId && cdp.sendToTarget) {
return cdp.sendToTarget<T>(target, method, params);
}
return cdp.send<T>(target.tabId, method, params);
}
export function cdpRunnerForTarget(cdp: CdpRunner, target: CdpTarget): CdpRunner {
return {
send: <T = unknown>(_tabId: number, method: string, params?: object) =>
sendToCdpTarget<T>(cdp, target, method, params),
};
}
export interface BufferedReadBounds {
since: number | undefined;
limit: number;
+17 -5
View File
@@ -1,5 +1,5 @@
// Snapshot ref resolution — normalise `@eN` / `eN`, look up
// `backendNodeId` via the session RefStore, and build stable
// compound CDP node identity via the session RefStore, and build stable
// `ref_not_found` errors for hard-failure tool paths.
import type { SessionContext } from "@/session-manager/manager";
@@ -10,6 +10,13 @@ import { rpcError } from "./errors";
export interface SnapshotRefLookup {
backendNodeId: number;
refKey: string;
frameId?: string;
cdpSessionId?: string;
}
function refEntryForTab(ctx: SessionContext, refKey: string, tabId: number) {
const entry = ctx.refStore.resolveEntry(refKey);
return entry?.tabId === tabId ? entry : null;
}
/**
@@ -23,9 +30,14 @@ export function lookupSnapshotRef(
tabId: number,
): SnapshotRefLookup | null {
const refKey = normaliseRef(ref);
const backendNodeId = ctx.refStore.resolve(refKey, { tabId });
if (backendNodeId === null) return null;
return { backendNodeId, refKey };
const entry = refEntryForTab(ctx, refKey, tabId);
if (!entry) return null;
return {
backendNodeId: entry.backendNodeId,
refKey,
...(entry.frameId ? { frameId: entry.frameId } : {}),
...(entry.cdpSessionId ? { cdpSessionId: entry.cdpSessionId } : {}),
};
}
/**
@@ -37,7 +49,7 @@ export function resolveSnapshotRef(
ctx: SessionContext,
ref: string,
tabId: number,
): { backendNodeId: number; refKey: string } | RpcError {
): SnapshotRefLookup | RpcError {
const looked = lookupSnapshotRef(ctx, ref, tabId);
if (looked === null) {
return rpcError(
@@ -698,7 +698,7 @@ describe("captureViewModel", () => {
const { nodes } = await captureViewModel(cdp, 4);
const div = nodes.find((n) => n.backendNodeId === 12);
expect(div?.localRect?.y).toBe(-200);
expect(div?.rect).toMatchObject({ y: -200, h: 800 });
expect(div?.rect).toMatchObject({ y: 0, h: 600 });
});
it("normalizes device-pixel bounds by devicePixelRatio", async () => {
@@ -919,7 +919,7 @@ describe("captureViewModel", () => {
expect(input?.attrs.type).toBe("text");
expect(input?.ownerFrameBackendNodeId).toBe(13);
expect(input?.localRect).toEqual({ x: 0, y: 0, w: 200, h: 40 });
expect(input?.rect).toBeNull();
expect(input?.rect).toEqual({ x: 100, y: 300, w: 200, h: 40 });
expect(rootFrameId).toBe("main-frame");
expect(frameNodes?.get("child-frame")).toBe(subNodes);
expect(frameOwnerBackendNodeIds?.get("child-frame")).toBe(13);
@@ -1091,10 +1091,10 @@ describe("captureViewModel", () => {
const input = iframeNodes.get(23)?.find((n) => n.backendNodeId === 31);
expect(nestedIframe?.localRect).toEqual({ x: 5, y: 6, w: 100, h: 80 });
expect(nestedIframe?.rect).toBeNull();
expect(nestedIframe?.rect).toEqual({ x: 15, y: 26, w: 100, h: 80 });
expect(input?.ownerFrameBackendNodeId).toBe(23);
expect(input?.localRect).toEqual({ x: 1, y: 2, w: 40, h: 20 });
expect(input?.rect).toBeNull();
expect(input?.rect).toEqual({ x: 16, y: 28, w: 40, h: 20 });
});
it("normalizes bounds then subtracts CSS scroll at dpr>1", async () => {
@@ -3,61 +3,133 @@ import type { CdpRunner } from "../../shared";
import type { CapturedNode, CapturedViewModel } from "../capture";
import { captureFrameData } from "../frame-capture";
function node(frameId: string, backendNodeId: number): CapturedNode {
function ownerNode(backendNodeId: number, x: number): CapturedNode {
return {
backendNodeId,
parentBackendNodeId: null,
frameId,
tag: "div",
parentBackendNodeId: 1,
frameId: "main",
ownerFrameBackendNodeId: null,
tag: "iframe",
attrs: {},
rect: null,
localRect: { x: 0, y: 0, w: 20, h: 20 },
paintOrder: 0,
rect: { x, y: 100, w: 300, h: 200 },
localRect: { x, y: 100, w: 300, h: 200 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
};
}
function childSnapshot(frameId: string, backendNodeId: number) {
const strings = [frameId, "body", "button", "static", "auto", "pointer"];
return {
strings,
documents: [
{
frameId,
nodes: {
parentIndex: [-1, 0],
nodeName: [1, 2],
backendNodeId: [backendNodeId - 1, backendNodeId],
attributes: [[], []],
},
layout: {
nodeIndex: [0, 1],
styles: [
[3, 4, 4],
[3, 4, 5],
],
bounds: [
[0, 0, 300, 200],
[10, 20, 100, 40],
],
paintOrders: [0, 1],
},
},
],
};
}
describe("captureFrameData", () => {
it("captures AX trees for every same-target frame found in the DOM snapshot", async () => {
it("captures and positions multiple OOPIF documents missing from the root snapshot", async () => {
const leftOwner = ownerNode(10, 50);
const rightOwner = ownerNode(20, 500);
const captured: CapturedViewModel = {
nodes: [node("main", 1)],
nodes: [leftOwner, rightOwner],
viewport: { width: 1000, height: 800 },
iframeNodes: new Map([[10, [node("child", 101)]]]),
frameNodes: new Map([
["main", [node("main", 1)]],
["child", [node("child", 101)]],
]),
frameOwnerBackendNodeIds: new Map([["child", 10]]),
frameParentIds: new Map([["child", "main"]]),
iframeNodes: new Map(),
frameNodes: new Map([["main", [leftOwner, rightOwner]]]),
frameOwnerBackendNodeIds: new Map(),
rootFrameId: "main",
excludedBackendNodeIds: new Set(),
};
const send = vi.fn(async (_tabId: number, method: string, params?: object) => {
if (method === "Accessibility.enable") return {};
if (method === "Accessibility.getFullAXTree") {
const frameId = (params as { frameId?: string })?.frameId ?? "main";
return {
nodes: [
{
nodeId: `${frameId}-root`,
frameId,
backendDOMNodeId: frameId === "main" ? 1 : 101,
},
],
};
const sendToTarget = vi.fn(async (target, method) => {
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 300, clientHeight: 200, pageX: 0, pageY: 0 } };
}
if (method === "DOMSnapshot.enable" || method === "Accessibility.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") {
return target.sessionId === "left-session"
? childSnapshot("left", 101)
: childSnapshot("right", 201);
}
if (method === "Accessibility.getFullAXTree") return { nodes: [] };
throw new Error(`unexpected ${method}`);
});
const cdp: CdpRunner = {
send: vi.fn(async (_tabId, method, params) => {
if (method === "Accessibility.enable") return {};
if (method === "Accessibility.getFullAXTree") return { nodes: [] };
if (method === "DOM.getBoxModel") {
const backendNodeId = (params as { backendNodeId?: number })?.backendNodeId;
const x = backendNodeId === 10 ? 50 : 500;
return { model: { content: [x, 100, x + 300, 100, x + 300, 300, x, 300] } };
}
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1000, clientHeight: 800 } };
}
throw new Error(`unexpected root ${method}`);
}) as CdpRunner["send"],
sendToTarget: sendToTarget as unknown as NonNullable<CdpRunner["sendToTarget"]>,
getFrameGraph: vi.fn(async () => ({
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "left",
parentFrameId: "main",
ownerBackendNodeId: 10,
target: { tabId: 4, sessionId: "left-session" },
},
{
frameId: "right",
parentFrameId: "main",
ownerBackendNodeId: 20,
target: { tabId: 4, sessionId: "right-session" },
},
],
})),
};
const documents = await captureFrameData({ send } as CdpRunner, 4, captured);
const trees = await captureFrameData(cdp, 4, captured);
expect(send).toHaveBeenCalledWith(4, "Accessibility.enable", {});
expect(documents.map((document) => document.frameId)).toEqual(["main", "child"]);
expect(documents.find((document) => document.frameId === "child")).toMatchObject({
parentFrameId: "main",
ownerBackendNodeId: 10,
axNodes: [{ nodeId: "child-root", frameId: "child", backendDOMNodeId: 101 }],
});
expect(trees.map((tree) => tree.frameId)).toEqual(["main", "left", "right"]);
expect(
captured.frameNodes?.get("left")?.find((node) => node.backendNodeId === 101)?.rect,
).toEqual({ x: 60, y: 120, w: 100, h: 40 });
expect(
captured.frameNodes?.get("right")?.find((node) => node.backendNodeId === 201)?.rect,
).toEqual({ x: 510, y: 120, w: 100, h: 40 });
expect(captured.frameOwnerBackendNodeIds).toEqual(
new Map([
["left", 10],
["right", 20],
]),
);
expect(captured.frameParentIds).toEqual(
new Map([
["left", "main"],
["right", "main"],
]),
);
});
});
@@ -1,4 +1,5 @@
import { describe, expect, it } from "vitest";
import type { CdpFrameGraph } from "@/browser-driver/frame-graph";
import type { CapturedNode, CapturedViewModel } from "../capture";
import { buildFrameDocuments, type FrameAxBatch, type FrameOwnedAxNode } from "../frame-document";
@@ -9,8 +10,7 @@ function domNode(frameId: string, backendNodeId: number): CapturedNode {
frameId,
tag: "div",
attrs: {},
rect: null,
localRect: { x: 0, y: 0, w: 20, h: 20 },
rect: { x: 0, y: 0, w: 20, h: 20 },
paintOrder: 0,
position: "static",
pointerEvents: "auto",
@@ -30,9 +30,16 @@ function captured(frameNodes: Map<string, CapturedNode[]>): CapturedViewModel {
describe("buildFrameDocuments", () => {
it("partitions mixed AX results by node ownership and cuts cross-frame parent edges", () => {
const graph: CdpFrameGraph = {
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{ frameId: "child", parentFrameId: "main", target: { tabId: 4 } },
],
};
const batches: FrameAxBatch<FrameOwnedAxNode>[] = [
{
frameId: "main",
frame: graph.frames[0],
nodes: [
{ nodeId: "root", frameId: "main", backendDOMNodeId: 1, childIds: ["h"] },
{
@@ -48,6 +55,7 @@ describe("buildFrameDocuments", () => {
];
const documents = buildFrameDocuments(
graph,
batches,
captured(
new Map([
@@ -64,14 +72,28 @@ describe("buildFrameDocuments", () => {
expect(childNodes.find((node) => node.nodeId === "virtual")?.frameId).toBe("child");
});
it("deduplicates overlapping AX results using the strongest ownership", () => {
it("deduplicates overlapping same-target AX results using the strongest ownership", () => {
const graph: CdpFrameGraph = {
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{ frameId: "child", parentFrameId: "main", target: { tabId: 4 } },
],
};
const childNode = { nodeId: "shared", backendDOMNodeId: 100 };
const batches: FrameAxBatch<FrameOwnedAxNode>[] = [
{ frameId: "main", nodes: [{ ...childNode, frameId: "child" }] },
{ frameId: "child", nodes: [childNode] },
{
frame: graph.frames[0],
nodes: [{ ...childNode, frameId: "child" }],
},
{
frame: graph.frames[1],
nodes: [childNode],
},
];
const documents = buildFrameDocuments(
graph,
batches,
captured(
new Map([
@@ -85,22 +107,64 @@ describe("buildFrameDocuments", () => {
expect(documents.find((document) => document.frameId === "child")?.axNodes).toHaveLength(1);
});
it("uses target-scoped backend ownership when OOPIF backend ids collide", () => {
const graph: CdpFrameGraph = {
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "left",
parentFrameId: "main",
target: { tabId: 4, sessionId: "left-session" },
},
{
frameId: "right",
parentFrameId: "main",
target: { tabId: 4, sessionId: "right-session" },
},
],
};
const batches: FrameAxBatch<FrameOwnedAxNode>[] = graph.frames.slice(1).map((frame) => ({
frame,
nodes: [{ nodeId: `${frame.frameId}-node`, backendDOMNodeId: 7 }],
}));
const documents = buildFrameDocuments(
graph,
batches,
captured(
new Map([
["main", []],
["left", [domNode("left", 7)]],
["right", [domNode("right", 7)]],
]),
),
);
expect(documents.find((document) => document.frameId === "left")?.axNodes[0]?.nodeId).toBe(
"left-node",
);
expect(documents.find((document) => document.frameId === "right")?.axNodes[0]?.nodeId).toBe(
"right-node",
);
});
it("uses captured frame parent identity instead of guessing from backend ids", () => {
const frameNodes = new Map([
["main", [domNode("main", 1)]],
["left", [domNode("left", 7)]],
["right", [domNode("right", 8)]],
["right", [domNode("right", 7)]],
["child", [domNode("child", 9)]],
]);
const capture = captured(frameNodes);
capture.frameOwnerBackendNodeIds = new Map([["child", 7]]);
capture.frameParentIds = new Map([["child", "left"]]);
const batches = ["main", "left", "right", "child"].map((frameId) => ({
frameId,
const frames = ["main", "left", "right", "child"].map((frameId) => ({
frame: { frameId, target: { tabId: 4 } },
nodes: [],
}));
const documents = buildFrameDocuments(batches, capture);
const documents = buildFrameDocuments(null, frames, capture);
expect(documents.find((document) => document.frameId === "child")).toMatchObject({
parentFrameId: "left",
+23 -4
View File
@@ -7,6 +7,7 @@
import type { Rect, Viewport } from "@browser-skill/vom";
import { evaluateHoverTrigger } from "@/lib/hover-trigger-policy";
import { isOverlayHostNode, OVERLAY_HOST_SELECTOR } from "../../lib/overlay-bridge";
import { childFrameProjection, type GeometryProjection, projectRectToViewport } from "../geometry";
import type { CdpRunner } from "../shared";
const REQUESTED_STYLES = ["position", "pointer-events", "cursor"] as const;
@@ -22,9 +23,9 @@ export interface CapturedNode {
ownerFrameBackendNodeId?: number | null;
tag: string;
attrs: Record<string, string>;
/** Top-level viewport geometry when this node belongs to the root document. */
/** Top-level viewport-relative CSS px, clipped to the owning frame viewport. */
rect: Rect | null;
/** Owning document viewport-relative CSS px. */
/** Frame-local viewport-relative CSS px before top-level projection. */
localRect?: Rect | null;
paintOrder: number;
position: string;
@@ -317,6 +318,7 @@ interface ParseDocumentResult {
interface FrameContext {
frameId?: string;
ownerFrameBackendNodeId: number | null;
projection: GeometryProjection | null;
scrollX: number;
scrollY: number;
}
@@ -869,7 +871,10 @@ function parseDocumentNodes(
w: b[2] / dpr,
h: b[3] / dpr,
};
rect = context.ownerFrameBackendNodeId === null ? localRect : null;
const bounds = context.projection
? projectRectToViewport(localRect, context.projection)
: null;
rect = bounds ? { x: bounds.x, y: bounds.y, w: bounds.width, h: bounds.height } : null;
}
paintOrder = dl?.paintOrders?.[li] ?? 0;
const styleRow = dl?.styles?.[li] ?? [];
@@ -939,6 +944,7 @@ function parseChildFrameDocuments(
strings: string[],
dpr: number,
parentDocIndex: number,
parentNodes: CapturedNode[],
parentContext: FrameContext,
visited = new Set<number>(),
): ParseFrameDocumentsResult {
@@ -953,6 +959,7 @@ function parseChildFrameDocuments(
}
const parentBackendIds = parentDoc?.nodes?.backendNodeId ?? [];
const parentNodeByBackendId = new Map(parentNodes.map((node) => [node.backendNodeId, node]));
for (const [nodeArrayIdx, childDocIndex] of cdi) {
if (visited.has(childDocIndex)) continue;
@@ -960,10 +967,16 @@ function parseChildFrameDocuments(
if (!childDoc) continue;
const iframeBackendId = parentBackendIds[nodeArrayIdx];
if (iframeBackendId === undefined) continue;
const iframeNode = parentNodeByBackendId.get(iframeBackendId);
const projection =
parentContext.projection && iframeNode?.localRect
? childFrameProjection(parentContext.projection, iframeNode.localRect)
: null;
const childContext: FrameContext = {
frameId: snapshotFrameId(childDoc, strings),
ownerFrameBackendNodeId: iframeBackendId,
projection,
scrollX: childDoc.scrollOffsetX ?? 0,
scrollY: childDoc.scrollOffsetY ?? 0,
};
@@ -982,6 +995,7 @@ function parseChildFrameDocuments(
strings,
dpr,
childDocIndex,
parsed.nodes,
childContext,
nextVisited,
);
@@ -1042,6 +1056,11 @@ export async function captureViewModel(
const topContext: FrameContext = {
frameId: snapshotFrameId(doc0, strings),
ownerFrameBackendNodeId: null,
projection: {
sourceClips: [],
edges: [],
topViewport: viewport,
},
scrollX,
scrollY,
};
@@ -1049,7 +1068,7 @@ export async function captureViewModel(
const nodes = mainParsed.nodes;
const excludedBackendNodeIds = new Set(mainParsed.excludedBackendNodeIds);
const frameParsed = parseChildFrameDocuments(documents, strings, dpr, 0, topContext);
const frameParsed = parseChildFrameDocuments(documents, strings, dpr, 0, nodes, topContext);
const iframeNodes = frameParsed.iframeNodes;
for (const id of frameParsed.excludedBackendNodeIds) {
excludedBackendNodeIds.add(id);
+123 -19
View File
@@ -1,5 +1,8 @@
import type { CdpRunner } from "../shared";
import type { CapturedViewModel } from "./capture";
import type { CdpFrame, CdpFrameGraph } from "@/browser-driver/frame-graph";
import { resolveFrameProjection } from "../frame-geometry";
import { type GeometryProjection, projectRectToViewport } from "../geometry";
import { type CdpRunner, cdpRunnerForTarget, sendToCdpTarget } from "../shared";
import { type CapturedNode, type CapturedViewModel, captureViewModel } from "./capture";
import {
buildFrameDocuments,
type FrameAxBatch,
@@ -18,39 +21,136 @@ function isAbortError(err: unknown): boolean {
return err instanceof Error && err.name === "AbortError";
}
function capturedFrameIds(captured: CapturedViewModel): string[] {
const frameIds = new Set<string>();
if (captured.rootFrameId) frameIds.add(captured.rootFrameId);
for (const frameId of captured.frameNodes?.keys() ?? []) frameIds.add(frameId);
if (frameIds.size === 0) frameIds.add("root");
return [...frameIds];
async function discoverFrameGraph(cdp: CdpRunner, tabId: number): Promise<CdpFrameGraph | null> {
if (!cdp.getFrameGraph) return null;
try {
return await cdp.getFrameGraph(tabId);
} catch (err) {
console.debug("[bsk observation] frame graph capture failed", err);
return null;
}
}
function transformFrameNodes(
nodes: CapturedNode[],
projection: GeometryProjection,
): CapturedNode[] {
return nodes.map((node) => ({
...node,
rect: (() => {
const bounds = projectRectToViewport(node.rect, projection);
return bounds ? { x: bounds.x, y: bounds.y, w: bounds.width, h: bounds.height } : null;
})(),
}));
}
function frameOwnerNode(captured: CapturedViewModel, frame: CdpFrame): CapturedNode | undefined {
if (frame.ownerBackendNodeId === undefined) return undefined;
const parentNodes = frame.parentFrameId
? captured.frameNodes?.get(frame.parentFrameId)
: captured.nodes;
return parentNodes?.find((node) => node.backendNodeId === frame.ownerBackendNodeId);
}
async function captureMissingFrameDocuments(
cdp: CdpRunner,
tabId: number,
graph: CdpFrameGraph,
captured: CapturedViewModel,
signal?: AbortSignal,
): Promise<void> {
if (!captured.frameNodes) captured.frameNodes = new Map();
if (!captured.frameOwnerBackendNodeIds) captured.frameOwnerBackendNodeIds = new Map();
if (!captured.frameParentIds) captured.frameParentIds = new Map();
for (const frame of graph.frames) {
if (!frame.target.sessionId || captured.frameNodes.has(frame.frameId)) continue;
const owner = frameOwnerNode(captured, frame);
if (!owner?.rect || frame.ownerBackendNodeId === undefined) continue;
try {
const child = await captureViewModel(cdpRunnerForTarget(cdp, frame.target), tabId, {
signal,
});
const projection = await resolveFrameProjection(cdp, graph, frame.frameId);
if (!projection) continue;
const rootFrameId = child.rootFrameId ?? frame.frameId;
const childFrames = child.frameNodes ?? new Map<string, CapturedNode[]>();
if (!childFrames.has(rootFrameId)) childFrames.set(rootFrameId, child.nodes);
if (rootFrameId !== frame.frameId && !childFrames.has(frame.frameId)) {
childFrames.set(
frame.frameId,
child.nodes.map((node) => ({ ...node, frameId: frame.frameId })),
);
}
for (const [childFrameId, nodes] of childFrames) {
captured.frameNodes.set(childFrameId, transformFrameNodes(nodes, projection));
}
captured.iframeNodes.set(
frame.ownerBackendNodeId,
captured.frameNodes.get(frame.frameId) ?? [],
);
captured.frameOwnerBackendNodeIds.set(frame.frameId, frame.ownerBackendNodeId);
if (frame.parentFrameId) captured.frameParentIds.set(frame.frameId, frame.parentFrameId);
for (const [childFrameId, ownerBackendNodeId] of child.frameOwnerBackendNodeIds ?? []) {
captured.frameOwnerBackendNodeIds.set(childFrameId, ownerBackendNodeId);
}
for (const [childFrameId, parentFrameId] of child.frameParentIds ?? []) {
captured.frameParentIds.set(childFrameId, parentFrameId);
}
} catch (err) {
if (isAbortError(err)) throw err;
console.debug("[bsk observation] child frame DOM capture failed", {
frameId: frame.frameId,
err,
});
}
}
}
async function captureAxTrees<T extends FrameAxNode>(
cdp: CdpRunner,
tabId: number,
discoveredFrames: CdpFrame[],
captured: CapturedViewModel,
signal?: AbortSignal,
): Promise<FrameAxBatch<T>[]> {
const frameIds = capturedFrameIds(captured);
let frames = [...discoveredFrames];
const capturedFrameIds = [...(captured.frameNodes?.keys() ?? [])];
if (frames.length === 0) {
const rootFrameId = captured.rootFrameId ?? capturedFrameIds[0] ?? "root";
frames = [{ frameId: rootFrameId, target: { tabId } }];
}
for (const frameId of capturedFrameIds) {
if (!frames.some((frame) => frame.frameId === frameId)) {
frames.push({ frameId, target: { tabId } });
}
}
let firstFailure: unknown;
await cdp.send(tabId, "Accessibility.enable", {});
const trees = await Promise.all(
frameIds.map(async (frameId): Promise<FrameAxBatch<T>> => {
frames.map(async (frame): Promise<FrameAxBatch<T>> => {
try {
throwIfAborted(signal);
const result = await cdp.send<{ nodes?: T[] }>(
tabId,
await sendToCdpTarget(cdp, frame.target, "Accessibility.enable", {});
const result = await sendToCdpTarget<{ nodes?: T[] }>(
cdp,
frame.target,
"Accessibility.getFullAXTree",
frameId === "root" ? {} : { frameId },
frame.frameId === "root" ? {} : { frameId: frame.frameId },
);
throwIfAborted(signal);
return { frameId, nodes: result.nodes ?? [] };
return {
frame,
nodes: result.nodes ?? [],
};
} catch (err) {
if (isAbortError(err)) throw err;
firstFailure ??= err;
console.debug("[bsk observation] frame AX capture failed", { frameId, err });
return { frameId, nodes: [] };
console.debug("[bsk observation] frame AX capture failed", {
frameId: frame.frameId,
err,
});
return { frame, nodes: [] };
}
}),
);
@@ -70,6 +170,10 @@ export async function captureFrameData<T extends FrameAxNode>(
captured: CapturedViewModel,
signal?: AbortSignal,
): Promise<CapturedFrameDocument<T>[]> {
const batches = await captureAxTrees<T>(cdp, tabId, captured, signal);
return buildFrameDocuments(batches, captured);
const graph = await discoverFrameGraph(cdp, tabId);
const frames = graph?.frames ?? [];
if (graph) await captureMissingFrameDocuments(cdp, tabId, graph, captured, signal);
throwIfAborted(signal);
const batches = await captureAxTrees<T>(cdp, tabId, frames, captured, signal);
return buildFrameDocuments(graph, batches, captured);
}
+72 -41
View File
@@ -1,3 +1,9 @@
import {
type CdpFrame,
type CdpFrameGraph,
type CdpTarget,
cdpTargetKey,
} from "@/browser-driver/frame-graph";
import type { CapturedNode, CapturedViewModel } from "./capture";
export interface FrameOwnedAxNode {
@@ -9,14 +15,11 @@ export interface FrameOwnedAxNode {
}
export interface FrameAxBatch<T extends FrameOwnedAxNode> {
frameId: string;
frame: CdpFrame;
nodes: T[];
}
export interface FrameDocument<T extends FrameOwnedAxNode> {
frameId: string;
parentFrameId?: string;
ownerBackendNodeId?: number;
export interface FrameDocument<T extends FrameOwnedAxNode> extends CdpFrame {
contextScopeId: string;
axNodes: T[];
domNodes: CapturedNode[];
@@ -29,30 +32,58 @@ interface Ownership {
interface OwnedCandidate<T> {
node: T;
target: CdpTarget;
ownership: Ownership;
}
function frameIds<T extends FrameOwnedAxNode>(
function targetNodeKey(target: CdpTarget, nodeId: string): string {
return `${cdpTargetKey(target)}:${nodeId}`;
}
function targetBackendKey(target: CdpTarget, backendNodeId: number): string {
return `${cdpTargetKey(target)}:${backendNodeId}`;
}
function frameList<T extends FrameOwnedAxNode>(
graph: CdpFrameGraph | null,
batches: FrameAxBatch<T>[],
captured: CapturedViewModel,
): string[] {
const ids = new Set<string>();
if (captured.rootFrameId) ids.add(captured.rootFrameId);
for (const frameId of captured.frameNodes?.keys() ?? []) ids.add(frameId);
for (const batch of batches) ids.add(batch.frameId);
return [...ids];
): CdpFrame[] {
const frames = new Map<string, CdpFrame>();
for (const frame of graph?.frames ?? []) frames.set(frame.frameId, frame);
for (const batch of batches) {
if (!frames.has(batch.frame.frameId)) frames.set(batch.frame.frameId, batch.frame);
}
for (const frameId of captured.frameNodes?.keys() ?? []) {
if (!frames.has(frameId))
frames.set(frameId, { frameId, target: { tabId: batches[0]?.frame.target.tabId ?? 0 } });
}
return [...frames.values()].map((frame) => {
const ownerBackendNodeId =
frame.ownerBackendNodeId ?? captured.frameOwnerBackendNodeIds?.get(frame.frameId);
const parentFrameId = frame.parentFrameId ?? captured.frameParentIds?.get(frame.frameId);
return {
...frame,
...(ownerBackendNodeId !== undefined ? { ownerBackendNodeId } : {}),
...(parentFrameId ? { parentFrameId } : {}),
};
});
}
export function buildFrameDocuments<T extends FrameOwnedAxNode>(
graph: CdpFrameGraph | null,
batches: FrameAxBatch<T>[],
captured: CapturedViewModel,
): FrameDocument<T>[] {
const frames = frameIds(batches, captured);
const knownFrames = new Set(frames);
const backendOwner = new Map<number, string>();
for (const frameId of frames) {
for (const node of captured.frameNodes?.get(frameId) ?? []) {
backendOwner.set(node.backendNodeId, frameId);
const frames = frameList(graph, batches, captured);
const frameById = new Map(frames.map((frame) => [frame.frameId, frame]));
const rootFrameId = graph?.rootFrameId ?? captured.rootFrameId ?? frames[0]?.frameId;
const domNodesForFrame = (frameId: string): CapturedNode[] =>
captured.frameNodes?.get(frameId) ?? (frameId === rootFrameId ? captured.nodes : []);
const backendOwner = new Map<string, string>();
for (const frame of frames) {
for (const node of domNodesForFrame(frame.frameId)) {
backendOwner.set(targetBackendKey(frame.target, node.backendNodeId), frame.frameId);
}
}
@@ -64,15 +95,17 @@ export function buildFrameDocuments<T extends FrameOwnedAxNode>(
const resolveOwnership = (node: T): Ownership => {
const cached = ownershipByNodeId.get(node.nodeId);
if (cached) return cached;
if (node.frameId && knownFrames.has(node.frameId)) {
if (node.frameId && frameById.has(node.frameId)) {
const ownership = { frameId: node.frameId, strength: 4 };
ownershipByNodeId.set(node.nodeId, ownership);
return ownership;
}
if (typeof node.backendDOMNodeId === "number") {
const owner = backendOwner.get(node.backendDOMNodeId);
if (owner) {
const ownership = { frameId: owner, strength: 3 };
const frameId = backendOwner.get(
targetBackendKey(batch.frame.target, node.backendDOMNodeId),
);
if (frameId) {
const ownership = { frameId, strength: 3 };
ownershipByNodeId.set(node.nodeId, ownership);
return ownership;
}
@@ -91,28 +124,32 @@ export function buildFrameDocuments<T extends FrameOwnedAxNode>(
return ownership;
}
}
const ownership = { frameId: batch.frameId, strength: 1 };
const ownership = { frameId: batch.frame.frameId, strength: 1 };
ownershipByNodeId.set(node.nodeId, ownership);
return ownership;
};
for (const node of batch.nodes) {
const ownership = resolveOwnership(node);
const existing = candidates.get(node.nodeId);
const key = targetNodeKey(batch.frame.target, node.nodeId);
const existing = candidates.get(key);
if (!existing || ownership.strength > existing.ownership.strength) {
candidates.set(node.nodeId, { node, ownership });
candidates.set(key, { node, target: batch.frame.target, ownership });
}
}
}
const ownershipByNodeId = new Map(
[...candidates].map(([nodeId, candidate]) => [nodeId, candidate.ownership.frameId]),
const ownershipByTargetNode = new Map(
[...candidates].map(([key, candidate]) => [key, candidate.ownership.frameId]),
);
const axNodesByFrame = new Map<string, T[]>();
for (const { node, ownership } of candidates.values()) {
const parentFrameId = node.parentId ? ownershipByNodeId.get(node.parentId) : undefined;
for (const candidate of candidates.values()) {
const { node, target, ownership } = candidate;
const parentFrameId = node.parentId
? ownershipByTargetNode.get(targetNodeKey(target, node.parentId))
: undefined;
const childIds = node.childIds?.filter(
(childId) => ownershipByNodeId.get(childId) === ownership.frameId,
(childId) => ownershipByTargetNode.get(targetNodeKey(target, childId)) === ownership.frameId,
);
const ownedNode = {
...node,
@@ -127,16 +164,10 @@ export function buildFrameDocuments<T extends FrameOwnedAxNode>(
axNodesByFrame.set(ownership.frameId, nodes);
}
return frames.map((frameId) => ({
frameId,
...(captured.frameParentIds?.get(frameId)
? { parentFrameId: captured.frameParentIds.get(frameId) }
: {}),
...(captured.frameOwnerBackendNodeIds?.get(frameId) !== undefined
? { ownerBackendNodeId: captured.frameOwnerBackendNodeIds.get(frameId) }
: {}),
contextScopeId: frameId,
axNodes: axNodesByFrame.get(frameId) ?? [],
domNodes: captured.frameNodes?.get(frameId) ?? [],
return frames.map((frame) => ({
...frame,
contextScopeId: frame.frameId,
axNodes: axNodesByFrame.get(frame.frameId) ?? [],
domNodes: domNodesForFrame(frame.frameId),
}));
}
@@ -0,0 +1,77 @@
import type { Rect, RenderedRef, VomResult } from "@browser-skill/vom";
import type { CdpTarget } from "@/browser-driver/frame-graph";
import type { CapturedSurfaceProbe } from "./capture";
import type { CapturedFrameDocument, FrameAxNode } from "./frame-capture";
/** Frame identity needed to resolve an `@eN` that lives in an iframe. */
export interface CaptureVomFrame {
frameId: string;
target: CdpTarget;
parentFrameId?: string;
ownerBackendNodeId?: number;
}
/** Allowlisted geometry used to match a recorded action to a rendered ref. */
export interface CaptureVomMatchNode {
frameId: string;
backendNodeId: number;
tag: string;
/** Top-level viewport-relative geometry. */
rect: Rect | null;
/** Frame-local viewport-relative geometry, when available. */
localRect?: Rect | null;
}
/**
* Raw AX/DOM trees are never returned. When `redactValues` is enabled by the
* caller, form values in `text` are masked as well.
*/
export interface CaptureVomObservationResult {
text: string;
refs: RenderedRef[];
truncated: boolean;
rootFrameId: string;
frames: CaptureVomFrame[];
matchNodes: CaptureVomMatchNode[];
surfaceProbes?: CapturedSurfaceProbe[];
}
function projectFrames(documents: CapturedFrameDocument<FrameAxNode>[]): CaptureVomFrame[] {
return documents.map((document) => ({
frameId: document.frameId,
target: document.target,
...(document.parentFrameId ? { parentFrameId: document.parentFrameId } : {}),
...(document.ownerBackendNodeId !== undefined
? { ownerBackendNodeId: document.ownerBackendNodeId }
: {}),
}));
}
function projectMatchNodes(documents: CapturedFrameDocument<FrameAxNode>[]): CaptureVomMatchNode[] {
return documents.flatMap((document) =>
document.domNodes.map((node) => ({
frameId: document.frameId,
backendNodeId: node.backendNodeId,
tag: node.tag,
rect: node.rect,
...(node.localRect !== undefined ? { localRect: node.localRect } : {}),
})),
);
}
export function projectRecordSafeObservation(input: {
rootFrameId: string;
frameDocuments: CapturedFrameDocument<FrameAxNode>[];
rendered: VomResult;
surfaceProbes?: CapturedSurfaceProbe[];
}): CaptureVomObservationResult {
return {
text: input.rendered.text,
refs: input.rendered.refs,
truncated: input.rendered.truncated,
rootFrameId: input.rootFrameId,
frames: projectFrames(input.frameDocuments),
matchNodes: projectMatchNodes(input.frameDocuments),
surfaceProbes: input.surfaceProbes,
};
}
+114 -65
View File
@@ -657,10 +657,60 @@ export interface EmulateResult {
}
// --------------------------------------------------------------------------
// Semantic record payloads — mirror bsk-protocol record.rs (Trace v2)
// Semantic record payloads mirror the versioned Rust protocol models.
// --------------------------------------------------------------------------
export interface TargetDescriptor {
export const TRACE_VERSION_V3 = 3;
/** Logical v2 identifier. Not a wire field — v2 envelopes omit `version`. */
export const TRACE_VERSION_V2 = 2;
export const VOM_FORMAT_VERSION = 1;
export interface TargetDescriptorV3 {
ref?: string;
role?: string;
name?: string;
ctx?: string;
unmatched?: boolean;
}
export interface RecorderInfo {
bsk: string;
vom: number;
}
export type StopReason = "user_finish" | "cli_stop";
export interface TraceStateV3 {
id: string;
url: string;
title?: string;
body: string;
truncated?: boolean;
}
export interface StepResultV3 {
state: string;
}
export interface StepCommonV3 {
id: number;
state: string;
result: StepResultV3;
}
export type NavigationCause =
| "user_typed"
| "link"
| "form_submit"
| "reload"
| "history"
| "script"
| "browser";
export type FillCommit = "enter" | "suggestion" | "blur";
/** Legacy v2 target shape retained for existing record producers. */
export interface TargetDescriptorV2 {
role?: string;
name?: string;
tag: string;
@@ -673,98 +723,97 @@ export interface TraceEntry {
start_url: string;
}
export interface PageRef {
export interface PageRefV2 {
id: string;
url: string;
title?: string;
}
export interface SelectedOption {
export interface SelectedOptionV2 {
value: string;
label?: string;
}
export interface StepEffect {
export interface StepEffectV2 {
navigated_to: string;
}
export interface StepCommon {
export interface StepCommonV2 {
id: number;
page: string;
effect?: StepEffect;
effect?: StepEffectV2;
}
/** Capture/buffer draft before v2 reduction. */
export type DraftTraceStep =
| {
op: "click";
target: TargetDescriptor;
navigated_to?: string;
page_url?: string;
}
| {
op: "hover";
target: TargetDescriptor;
page_url?: string;
}
| {
op: "fill";
target: TargetDescriptor;
value: string;
redacted?: boolean;
page_url?: string;
}
| {
op: "press";
key: string;
target?: TargetDescriptor;
modifiers?: KeyModifier[];
navigated_to?: string;
page_url?: string;
}
| {
op: "select";
target: TargetDescriptor;
values: string[];
labels?: string[];
navigated_to?: string;
page_url?: string;
}
| {
op: "navigate";
url: string;
page_url?: string;
};
/** Exported record-only step (trace v2). */
export type Step =
| ({ op: "navigate" } & StepCommon & { to: string })
| ({ op: "click" } & StepCommon & { target: TargetDescriptor })
| ({ op: "hover" } & StepCommon & { target: TargetDescriptor })
| ({ op: "fill" } & StepCommon & {
target: TargetDescriptor;
export type StepV2 =
| ({ op: "navigate" } & StepCommonV2 & { to: string })
| ({ op: "click" } & StepCommonV2 & { target: TargetDescriptorV2 })
| ({ op: "hover" } & StepCommonV2 & { target: TargetDescriptorV2 })
| ({ op: "fill" } & StepCommonV2 & {
target: TargetDescriptorV2;
value: string;
redacted?: boolean;
})
| ({ op: "select" } & StepCommon & {
target: TargetDescriptor;
selection: SelectedOption[];
| ({ op: "select" } & StepCommonV2 & {
target: TargetDescriptorV2;
selection: SelectedOptionV2[];
})
| ({ op: "press" } & StepCommon & {
| ({ op: "press" } & StepCommonV2 & {
key: string;
modifiers?: KeyModifier[];
target?: TargetDescriptor;
target?: TargetDescriptorV2;
});
export interface Trace {
export interface TraceV2 {
recorded_at: string;
started_at?: string;
purpose?: string;
entry: TraceEntry;
pages: PageRef[];
steps: Step[];
pages: PageRefV2[];
steps: StepV2[];
}
export interface SelectedOptionV3 {
value: string;
label?: string;
}
export type StepV3 =
| ({ op: "navigate" } & StepCommonV3 & { to: string; cause: NavigationCause })
| ({ op: "click" } & StepCommonV3 & { target: TargetDescriptorV3 })
| ({ op: "hover" } & StepCommonV3 & { target: TargetDescriptorV3 })
| ({ op: "fill" } & StepCommonV3 & {
target: TargetDescriptorV3;
value: string;
commit: FillCommit;
redacted?: boolean;
})
| ({ op: "select" } & StepCommonV3 & {
target: TargetDescriptorV3;
selection?: SelectedOptionV3[];
})
| ({ op: "press" } & StepCommonV3 & {
key: string;
modifiers?: KeyModifier[];
target?: TargetDescriptorV3;
})
| ({ op: "scroll" } & StepCommonV3);
export interface TraceV3 {
version: typeof TRACE_VERSION_V3;
recorded_at: string;
started_at?: string;
purpose?: string;
stopped_by: StopReason;
entry: TraceEntry;
recorder: RecorderInfo;
states: TraceStateV3[];
steps: StepV3[];
}
export type RecordedTrace = TraceV2 | TraceV3;
export type RecordedStep = StepV2 | StepV3;
export interface RecordStartParams {
session_id: string;
tab_id?: number;
@@ -782,7 +831,7 @@ export interface RecordStopParams {
}
export interface RecordStopResult {
trace: Trace;
trace: RecordedTrace;
}
export interface RecordAwaitParams {
@@ -791,5 +840,5 @@ export interface RecordAwaitParams {
}
export interface RecordAwaitResult {
trace: Trace;
trace: RecordedTrace;
}
+2 -4
View File
@@ -25,10 +25,8 @@ export default defineConfig({
name: "BrowserSkill",
description:
"Let AI agents use your logged-in browser in a separate Agent Window—without interrupting your work. Powered by the bsk CLI.",
// Chrome 116 is the first version where WebSocket activity resets the
// service-worker idle timer, which the 20s heartbeat relies on to
// keep the daemon link alive.
minimum_chrome_version: "116",
// Flat debugger sessions are required to address out-of-process iframes.
minimum_chrome_version: "125",
permissions: [
"alarms",
"debugger",
+37 -11
View File
@@ -18,6 +18,12 @@ use crate::daemon::state::PROTOCOL_VERSION;
const EXTENSION_STORE_URL: &str =
"https://chromewebstore.google.com/detail/hhcmgoofomhgciiibhipgmgkgnoenaoi";
/// Edge Add-ons listing for the browser-skill extension.
const EXTENSION_STORE_URL_EDGE: &str = "https://microsoftedge.microsoft.com/addons/detail/browserskill/emacgiaaaiojkkpkddmmdfhmokgmnikg";
/// Store listings highlighted in repair hints, in the order they appear.
const EXTENSION_STORE_URLS: [&str; 2] = [EXTENSION_STORE_URL, EXTENSION_STORE_URL_EDGE];
/// Status of a single doctor check. `Ok` / `Fail` are the legacy two
/// states; `NotApplicable` (review M2) is reported as "N/A" in human
/// output and as `"status": "na"` in `--json` output, so a check that
@@ -420,7 +426,10 @@ fn check_extension_connected(status: Option<&StatusResult>) -> CheckResult {
CheckResult::fail(
name,
"0 browsers connected",
format!("install the extension from {EXTENSION_STORE_URL} and load it in Chromium"),
format!(
"install the extension from {EXTENSION_STORE_URL} (Chrome) \
or {EXTENSION_STORE_URL_EDGE} (Edge) and load it in the browser"
),
)
}
}
@@ -428,17 +437,14 @@ fn check_extension_connected(status: Option<&StatusResult>) -> CheckResult {
/// Highlight known URLs in repair hints for terminal output. Plain
/// text is preserved in `--json` and in stored [`CheckResult::hint`].
fn style_hint(hint: &str) -> String {
if !hint.contains(EXTENSION_STORE_URL) {
return hint.to_string();
let mut styled = hint.to_string();
for url in EXTENSION_STORE_URLS {
if !styled.contains(url) {
continue;
}
styled = styled.replace(url, &style(url).cyan().bold().underlined().to_string());
}
hint.replace(
EXTENSION_STORE_URL,
&style(EXTENSION_STORE_URL)
.cyan()
.bold()
.underlined()
.to_string(),
)
styled
}
fn render_human(checks: &[CheckResult]) {
@@ -504,6 +510,26 @@ mod m2_tests {
hint.contains(EXTENSION_STORE_URL),
"hint should include Chrome Web Store URL: {hint}"
);
assert!(
hint.contains(EXTENSION_STORE_URL_EDGE),
"hint should include Edge Add-ons URL: {hint}"
);
}
#[test]
fn style_hint_preserves_every_store_url() {
let hint = check_extension_connected(Some(&fake_status(Vec::new(), Vec::new())))
.hint
.expect("extension disconnected should include a hint");
let styled = style_hint(&hint);
// Whether or not the terminal accepts colors, styling must never drop or
// mangle a URL the user has to click.
for url in EXTENSION_STORE_URLS {
assert!(
styled.contains(url),
"styled hint should still contain {url}: {styled}"
);
}
}
#[test]
+8 -4
View File
@@ -8,7 +8,7 @@ use anyhow::Context;
use bsk_protocol::Method;
use bsk_protocol::tools::{
RecordAwaitParams, RecordAwaitResult, RecordStartParams, RecordStartResult, RecordStopParams,
RecordStopResult, Trace,
RecordStopResult, RecordedTrace,
};
use clap::{Args, Subcommand};
@@ -190,7 +190,7 @@ fn record_await_ipc_timeout(timeout_ms: u32) -> Duration {
.unwrap_or(Duration::from_secs(u64::from(timeout_ms / 1_000) + 15))
}
fn write_trace_file(output: &PathBuf, trace: &Trace) -> Result<(), CliError> {
fn write_trace_file(output: &PathBuf, trace: &RecordedTrace) -> Result<(), CliError> {
let json = serde_json::to_string_pretty(trace)
.context("serialize trace JSON")
.map_err(CliError::Local)?;
@@ -207,7 +207,7 @@ fn write_trace_file(output: &PathBuf, trace: &Trace) -> Result<(), CliError> {
Ok(())
}
fn render_finish(trace: &Trace, output: &PathBuf, format: Format) -> Result<(), CliError> {
fn render_finish(trace: &RecordedTrace, output: &PathBuf, format: Format) -> Result<(), CliError> {
match format {
Format::Json => {
println!(
@@ -221,7 +221,11 @@ fn render_finish(trace: &Trace, output: &PathBuf, format: Format) -> Result<(),
);
}
Format::Human => {
println!("saved {} steps to {}", trace.steps.len(), output.display());
let step_count = match trace {
RecordedTrace::V2(trace) => trace.steps.len(),
RecordedTrace::V3(trace) => trace.steps.len(),
};
println!("saved {step_count} steps to {}", output.display());
}
}
Ok(())
+88 -9
View File
@@ -206,15 +206,15 @@ pub fn full_handler(status: DaemonStatus, state: Arc<DaemonState>) -> RpcHandler
Ok(v) => ResponseBody::Ok(v),
Err(e) => ResponseBody::Err(e),
},
Method::SessionStart => match handle_session_start(&state, params).await {
Method::SessionStart => match handle_session_start(&state, rpc_id, params).await {
Ok(v) => ResponseBody::Ok(v),
Err(e) => ResponseBody::Err(e),
},
Method::SessionStop => match handle_session_stop(&state, params).await {
Method::SessionStop => match handle_session_stop(&state, rpc_id, params).await {
Ok(v) => ResponseBody::Ok(v),
Err(e) => ResponseBody::Err(e),
},
Method::SessionStopAll => match handle_session_stop_all(&state).await {
Method::SessionStopAll => match handle_session_stop_all_rpc(&state, rpc_id).await {
Ok(v) => ResponseBody::Ok(v),
Err(e) => ResponseBody::Err(e),
},
@@ -434,8 +434,9 @@ async fn handle_wait_ms(
/// cancellation surfaces (M10.2 + review C2):
///
/// 1. [`AbortRegistry`] — answers daemon-local cancellable runners
/// (currently `tool.wait_ms`). If a token is registered we trip it
/// and stop.
/// (`tool.wait_ms` and `session.*` lifecycle calls). If a token is
/// registered we trip it and stop. Lifecycle handlers forward their
/// own WS cancel after preserving request-before-cancel ordering.
/// 2. [`super::inflight::ToolInflightRegistry`] — every IPC-tracked
/// `tool.*` RPC, registered the moment the IPC handler accepts the
/// request. Trips the entry's cancel token regardless of whether
@@ -634,7 +635,20 @@ fn clamp_browser_wait(wait_ms: Option<u64>) -> Option<Duration> {
))
}
async fn handle_session_start(state: &Arc<DaemonState>, params: Value) -> Result<Value, RpcError> {
async fn handle_session_start(
state: &Arc<DaemonState>,
rpc_id: RpcId,
params: Value,
) -> Result<Value, RpcError> {
let abort_guard = state
.abort_registry
.register(rpc_id)
.map_err(|err| RpcError {
code: ErrorCode::ProtocolError,
message: format!("session.start cancellation registration failed: {err:?}"),
data: None,
})?;
let cancel = abort_guard.token().clone();
let params: CliSessionStartParams = if params.is_null() {
CliSessionStartParams {
browser_instance_id: None,
@@ -673,6 +687,7 @@ async fn handle_session_start(state: &Arc<DaemonState>, params: Value) -> Result
},
state.config.extension_connect_wait,
DEFAULT_RPC_TIMEOUT,
Some(cancel),
)
.await
{
@@ -696,6 +711,8 @@ fn map_start_error(err: StartSessionError) -> RpcError {
StartSessionError::AmbiguousBrowserLabel { .. } => ErrorCode::InvalidParams,
StartSessionError::IdExhausted => ErrorCode::ProtocolError,
StartSessionError::Timeout => ErrorCode::Timeout,
StartSessionError::Cancelled => ErrorCode::Cancelled,
StartSessionError::CleanupFailed { .. } => ErrorCode::ProtocolError,
StartSessionError::TransportClosed => ErrorCode::ProtocolError,
StartSessionError::ExtensionError(inner) => inner.code,
};
@@ -711,6 +728,16 @@ fn map_start_error(err: StartSessionError) -> RpcError {
"label": label,
"instance_ids": instance_ids,
})),
StartSessionError::CleanupFailed {
session_id,
agent_window_id,
..
} => Some(serde_json::json!({
"reason": "cleanup_failed",
"session_id": session_id.0,
"agent_window_id": agent_window_id,
})),
StartSessionError::ExtensionError(inner) => inner.data.clone(),
_ => None,
};
RpcError {
@@ -720,7 +747,20 @@ fn map_start_error(err: StartSessionError) -> RpcError {
}
}
async fn handle_session_stop(state: &Arc<DaemonState>, params: Value) -> Result<Value, RpcError> {
async fn handle_session_stop(
state: &Arc<DaemonState>,
rpc_id: RpcId,
params: Value,
) -> Result<Value, RpcError> {
let abort_guard = state
.abort_registry
.register(rpc_id)
.map_err(|err| RpcError {
code: ErrorCode::ProtocolError,
message: format!("session.stop cancellation registration failed: {err:?}"),
data: None,
})?;
let cancel = abort_guard.token().clone();
let params: CliSessionStopParams = if params.is_null() {
CliSessionStopParams {
session_id: None,
@@ -734,7 +774,7 @@ async fn handle_session_stop(state: &Arc<DaemonState>, params: Value) -> Result<
})?
};
if params.all {
return handle_session_stop_all(state).await;
return handle_session_stop_all(state, Some(cancel)).await;
}
let session_id = match params.session_id {
Some(s) => SessionId(s),
@@ -753,6 +793,7 @@ async fn handle_session_stop(state: &Arc<DaemonState>, params: Value) -> Result<
&state.session_interrupts,
&session_id,
DEFAULT_SESSION_STOP_TIMEOUT,
Some(cancel),
)
.await
{
@@ -804,7 +845,25 @@ fn map_stop_error(err: StopSessionError) -> RpcError {
}
}
async fn handle_session_stop_all(state: &Arc<DaemonState>) -> Result<Value, RpcError> {
async fn handle_session_stop_all_rpc(
state: &Arc<DaemonState>,
rpc_id: RpcId,
) -> Result<Value, RpcError> {
let abort_guard = state
.abort_registry
.register(rpc_id)
.map_err(|err| RpcError {
code: ErrorCode::ProtocolError,
message: format!("session.stop_all cancellation registration failed: {err:?}"),
data: None,
})?;
handle_session_stop_all(state, Some(abort_guard.token().clone())).await
}
async fn handle_session_stop_all(
state: &Arc<DaemonState>,
cancel: Option<super::abort::AbortToken>,
) -> Result<Value, RpcError> {
let ids: Vec<SessionId> = state
.sessions
.snapshot()
@@ -816,6 +875,16 @@ async fn handle_session_stop_all(state: &Arc<DaemonState>) -> Result<Value, RpcE
let mut returned_tab_ids = Vec::new();
let mut return_failures = Vec::new();
for id in ids {
if cancel
.as_ref()
.is_some_and(super::abort::AbortToken::is_cancelled)
{
return Err(RpcError {
code: ErrorCode::Cancelled,
message: "session.stop_all was cancelled".into(),
data: None,
});
}
match stop_session(
&state.browsers,
&state.sessions,
@@ -823,6 +892,7 @@ async fn handle_session_stop_all(state: &Arc<DaemonState>) -> Result<Value, RpcE
&state.session_interrupts,
&id,
DEFAULT_SESSION_STOP_TIMEOUT,
cancel.clone(),
)
.await
{
@@ -848,6 +918,15 @@ async fn handle_session_stop_all(state: &Arc<DaemonState>) -> Result<Value, RpcE
returned_tab_ids.extend(stop.returned_tab_ids);
return_failures.extend(stop.return_failures);
}
Err(err)
if matches!(
&err,
StopSessionError::ExtensionError(inner)
if inner.code == ErrorCode::Cancelled
) =>
{
return Err(map_stop_error(err));
}
Err(err) => {
debug!(session = %id, ?err, "session.stop_all: failure (continuing)");
let rpc = map_stop_error(err);
+158 -17
View File
@@ -43,6 +43,7 @@ use serde_json::Value;
use tokio::sync::{mpsc, oneshot};
use tracing::{debug, warn};
use super::abort::AbortToken;
use super::browsers::BrowserRegistry;
use super::inflight::{PromoteOutcome, ToolInflightEntry};
use super::sessions::{SessionId, SessionRegistry};
@@ -83,6 +84,14 @@ pub struct ToolJob {
/// IPC request id (e.g. `session.stop`'s queued teardown call —
/// those already carry their own bespoke retry / abort path).
pub inflight: Option<Arc<ToolInflightEntry>>,
/// Cancellation token for session-lifecycle work registered in the
/// daemon-local abort registry. Unlike `inflight`, lifecycle callers
/// forward their own WS cancel after the request has been queued.
pub lifecycle_cancel: Option<AbortToken>,
/// How long cancellation keeps the worker busy while the extension
/// finishes compensation. Session teardown uses its full RPC budget
/// so daemon state is reconciled even after the CLI exits.
pub cancel_cleanup_timeout: Duration,
/// Owning session — used by `ToolInflightRegistry::cancel_session`
/// to drain queued jobs that have not yet been promoted to
/// "forwarded". Worker callers also have access to it via the
@@ -322,6 +331,8 @@ impl ToolQueueRegistry {
timeout,
false,
inflight,
None,
CANCEL_CLEANUP_TIMEOUT,
)
.await;
// A long-running request may outlive the idle threshold. Touching
@@ -360,6 +371,8 @@ impl ToolQueueRegistry {
timeout,
respond: respond_tx,
inflight,
lifecycle_cancel: None,
cancel_cleanup_timeout: CANCEL_CLEANUP_TIMEOUT,
session_id: sid.clone(),
};
match forward_one(sid, &self.browsers, &self.sessions, &job).await {
@@ -378,6 +391,7 @@ impl ToolQueueRegistry {
method: Method,
params: Value,
timeout: Duration,
cancel: Option<AbortToken>,
) -> Result<Value, DispatchError> {
let (sender, state) = {
let mut guard = self.queues.lock().expect("tool queue registry poisoned");
@@ -404,6 +418,8 @@ impl ToolQueueRegistry {
timeout,
true,
None,
cancel,
timeout,
)
.await
}
@@ -419,6 +435,8 @@ async fn dispatch_with_sender(
timeout: Duration,
wait_for_capacity: bool,
inflight: Option<Arc<ToolInflightEntry>>,
lifecycle_cancel: Option<AbortToken>,
cancel_cleanup_timeout: Duration,
) -> Result<Value, DispatchError> {
let (respond_tx, respond_rx) = oneshot::channel();
let job = ToolJob {
@@ -427,8 +445,11 @@ async fn dispatch_with_sender(
timeout,
respond: respond_tx,
inflight,
lifecycle_cancel,
cancel_cleanup_timeout,
session_id,
};
let lifecycle_cancellable = job.lifecycle_cancel.is_some();
if wait_for_capacity {
let waited = tokio::time::timeout(
timeout.saturating_add(Duration::from_secs(1)),
@@ -453,8 +474,15 @@ async fn dispatch_with_sender(
mpsc::error::TrySendError::Closed(_) => DispatchError::QueueClosed,
});
}
let waited =
tokio::time::timeout(timeout.saturating_add(Duration::from_secs(1)), respond_rx).await;
let mut response_timeout = timeout.saturating_add(Duration::from_secs(1));
if lifecycle_cancellable {
// A lifecycle cancel can arrive at the original RPC deadline and
// intentionally keeps the worker alive while extension cleanup
// settles. Do not let this outer waiter abandon that reconciliation
// early, which could reopen a daemon session whose window is gone.
response_timeout = response_timeout.saturating_add(cancel_cleanup_timeout);
}
let waited = tokio::time::timeout(response_timeout, respond_rx).await;
match waited {
Ok(Ok(Ok(v))) => Ok(v),
Ok(Ok(Err(rpc))) => Err(DispatchError::Rpc(rpc)),
@@ -492,11 +520,17 @@ async fn forward_one(
// / browser resolution work, and before any WS frame leaves the
// daemon (review C2). We keep the `tokio::select!` later so
// cancels arriving mid-WS-hop also unblock the worker.
if let Some(entry) = job.inflight.as_ref()
&& entry.is_cancelled()
if job
.inflight
.as_ref()
.is_some_and(|entry| entry.is_cancelled())
|| job
.lifecycle_cancel
.as_ref()
.is_some_and(AbortToken::is_cancelled)
{
return Err(cancelled_error(
Some(entry),
job.inflight.as_deref(),
"tool dispatch cancelled before forwarding",
));
}
@@ -560,11 +594,20 @@ async fn forward_one(
}
}
None => {
// Daemon-internal callers (e.g. queued session.stop drain)
// don't carry an inflight entry; they cannot be cancelled
// through the IPC `cancel` surface, so a plain sink send
// is sufficient and keeps the existing transport-error
// behaviour.
// Daemon-internal callers do not carry a tool-inflight entry.
// Session lifecycle jobs instead use the abort-registry token
// below; other internal callers keep the plain send path.
if job
.lifecycle_cancel
.as_ref()
.is_some_and(AbortToken::is_cancelled)
{
client.pending.lock().unwrap().cancel(&rpc_id);
return Err(cancelled_error(
None,
"session lifecycle cancelled before forwarding",
));
}
if client.sink.send(request).is_err() {
client.pending.lock().unwrap().cancel(&rpc_id);
return Err(RpcError {
@@ -573,19 +616,45 @@ async fn forward_one(
data: None,
});
}
None
job.lifecycle_cancel.clone()
}
};
let forward_lifecycle_cancel = || {
if job.lifecycle_cancel.is_none() {
return;
}
let cancel = Frame::Request(RequestFrame {
id: format!("cancel-{rpc_id}"),
method: Method::Cancel,
params: Some(serde_json::json!({ "rpc_id": rpc_id })),
});
if let Err(err) = client.sink.send(cancel) {
warn!(%rpc_id, ?err, "failed to forward session lifecycle cancel");
}
};
let on_abort: Option<&(dyn Fn() + Sync)> = job
.lifecycle_cancel
.as_ref()
.map(|_| &forward_lifecycle_cancel as &(dyn Fn() + Sync));
let waited = await_with_optional_cancel(
job.timeout,
CANCEL_CLEANUP_TIMEOUT,
job.cancel_cleanup_timeout,
waiter,
cancel_token.as_ref(),
on_abort,
)
.await;
let response = match waited {
WaitOutcome::Response(resp) => resp,
WaitOutcome::CancelledAfterResponse(resp) => {
if job.method == Method::ToolSessionStop
&& let ResponseBody::Ok(value) = resp.body
{
// Teardown crossed its final irreversible boundary before
// the cancel landed. Commit the real extension result so the
// daemon does not retain a session whose window is gone.
return Ok(value);
}
// Cancellation wins the external verdict, but only after the
// extension's original RPC has settled. Preserve any non-cancel
// error so compensation failures remain explicit instead of
@@ -600,13 +669,33 @@ async fn forward_one(
"tool dispatch cancelled after extension cleanup",
));
}
WaitOutcome::TimedOutAfterResponse(resp) => match resp.body {
ResponseBody::Ok(value) if job.method == Method::ToolSessionStop => {
// The close crossed its irreversible boundary during the
// timeout cleanup grace. Reconcile daemon state instead of
// reopening a session whose window is already gone.
return Ok(value);
}
ResponseBody::Err(err)
if !matches!(err.code, ErrorCode::Cancelled | ErrorCode::UserAborted) =>
{
return Err(err);
}
_ => {
return Err(RpcError {
code: ErrorCode::Timeout,
message: format!("tool RPC timed out after {:?}", job.timeout),
data: None,
});
}
},
WaitOutcome::CleanupTimeout => {
client.pending.lock().unwrap().cancel(&rpc_id);
return Err(RpcError {
code: ErrorCode::Timeout,
message: format!(
"cancelled tool did not finish cleanup within {:?}",
CANCEL_CLEANUP_TIMEOUT
job.cancel_cleanup_timeout
),
data: Some(serde_json::json!({ "reason": "cancel_cleanup_timeout" })),
});
@@ -638,6 +727,7 @@ async fn forward_one(
enum WaitOutcome {
Response(bsk_protocol::ResponseFrame),
CancelledAfterResponse(bsk_protocol::ResponseFrame),
TimedOutAfterResponse(bsk_protocol::ResponseFrame),
CleanupTimeout,
WaiterClosed,
Timeout,
@@ -648,6 +738,7 @@ async fn await_with_optional_cancel(
cleanup_timeout: Duration,
mut waiter: oneshot::Receiver<bsk_protocol::ResponseFrame>,
cancel: Option<&super::abort::AbortToken>,
on_abort: Option<&(dyn Fn() + Sync)>,
) -> WaitOutcome {
match cancel {
Some(token) => {
@@ -659,6 +750,9 @@ async fn await_with_optional_cancel(
// after compensation or the cleanup deadline expires.
biased;
_ = token.cancelled() => {
if let Some(on_abort) = on_abort {
on_abort();
}
match tokio::time::timeout(cleanup_timeout, &mut waiter).await {
Ok(Ok(resp)) => WaitOutcome::CancelledAfterResponse(resp),
Ok(Err(_)) => WaitOutcome::WaiterClosed,
@@ -669,7 +763,17 @@ async fn await_with_optional_cancel(
Ok(resp) => WaitOutcome::Response(resp),
Err(_) => WaitOutcome::WaiterClosed,
},
_ = &mut deadline => WaitOutcome::Timeout,
_ = &mut deadline => {
let Some(on_abort) = on_abort else {
return WaitOutcome::Timeout;
};
on_abort();
match tokio::time::timeout(cleanup_timeout, &mut waiter).await {
Ok(Ok(resp)) => WaitOutcome::TimedOutAfterResponse(resp),
Ok(Err(_)) => WaitOutcome::WaiterClosed,
Err(_) => WaitOutcome::CleanupTimeout,
}
},
}
}
None => match tokio::time::timeout(timeout, &mut waiter).await {
@@ -759,6 +863,7 @@ mod await_with_optional_cancel_tests {
Duration::from_secs(1),
rx,
Some(&token),
None,
)
.await;
assert!(
@@ -781,6 +886,7 @@ mod await_with_optional_cancel_tests {
Duration::from_secs(1),
rx,
Some(&token),
None,
)
.await;
match outcome {
@@ -795,9 +901,14 @@ mod await_with_optional_cancel_tests {
async fn no_cancel_token_still_returns_response() {
let (tx, rx) = oneshot::channel();
tx.send(dummy_response()).unwrap();
let outcome =
await_with_optional_cancel(Duration::from_secs(10), Duration::from_secs(1), rx, None)
.await;
let outcome = await_with_optional_cancel(
Duration::from_secs(10),
Duration::from_secs(1),
rx,
None,
None,
)
.await;
assert!(matches!(outcome, WaitOutcome::Response(_)));
}
@@ -812,6 +923,7 @@ mod await_with_optional_cancel_tests {
Duration::from_secs(1),
rx,
Some(&token),
None,
)
.await
});
@@ -835,10 +947,37 @@ mod await_with_optional_cancel_tests {
Duration::from_millis(10),
rx,
Some(&token),
None,
)
.await;
assert!(matches!(outcome, WaitOutcome::CleanupTimeout));
}
#[tokio::test]
async fn lifecycle_timeout_forwards_abort_and_waits_for_cleanup_response() {
use std::sync::atomic::{AtomicBool, Ordering};
let token = AbortToken::new();
let (tx, rx) = oneshot::channel();
let abort_forwarded = AtomicBool::new(false);
let forward_abort = || abort_forwarded.store(true, Ordering::SeqCst);
tokio::spawn(async move {
tokio::time::sleep(Duration::from_millis(20)).await;
tx.send(dummy_response()).unwrap();
});
let outcome = await_with_optional_cancel(
Duration::from_millis(5),
Duration::from_millis(100),
rx,
Some(&token),
Some(&forward_abort),
)
.await;
assert!(abort_forwarded.load(Ordering::SeqCst));
assert!(matches!(outcome, WaitOutcome::TimedOutAfterResponse(_)));
}
}
#[cfg(test)]
@@ -901,6 +1040,8 @@ mod tool_job_session_id_tests {
timeout: Duration::from_secs(1),
respond: tx,
inflight: None,
lifecycle_cancel: None,
cancel_cleanup_timeout: CANCEL_CLEANUP_TIMEOUT,
session_id: SessionId("sess-A".into()),
};
assert_eq!(job.session_id.0, "sess-A");
+223 -26
View File
@@ -12,13 +12,14 @@ use bsk_protocol::system::{BrowserStatusEntry, SessionStatusEntry};
use bsk_protocol::tools::{
SessionStartParams, SessionStartResult, SessionStopParams, SessionStopResult,
};
use bsk_protocol::{Frame, RequestFrame, ResponseBody, RpcError, RpcId};
use bsk_protocol::{ErrorCode, Frame, Method, RequestFrame, ResponseBody, RpcError, RpcId};
use rand::Rng;
use serde::{Deserialize, Serialize};
use tokio::time::{Duration, timeout};
use super::abort::AbortToken;
use super::browsers::{BrowserClient, BrowserId, BrowserRegistry, SelectError};
use super::queue::{DispatchError, ToolQueueRegistry};
use super::queue::{CANCEL_CLEANUP_TIMEOUT, DispatchError, ToolQueueRegistry};
use super::session_interrupt::SessionInterruptRegistry;
#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)]
@@ -339,6 +340,14 @@ pub enum StartSessionError {
IdExhausted,
#[error("session creation timed out waiting for extension")]
Timeout,
#[error("session creation was cancelled")]
Cancelled,
#[error("session creation cancellation cleanup failed for {session_id}: {message}")]
CleanupFailed {
session_id: SessionId,
agent_window_id: Option<i64>,
message: String,
},
#[error("extension rejected tool.session_start: {0:?}")]
ExtensionError(RpcError),
#[error("transport closed while waiting for extension response")]
@@ -354,6 +363,8 @@ impl StartSessionError {
StartSessionError::AmbiguousBrowserLabel { .. } => "invalid_params",
StartSessionError::IdExhausted => "protocol_error",
StartSessionError::Timeout => "timeout",
StartSessionError::Cancelled => "cancelled",
StartSessionError::CleanupFailed { .. } => "protocol_error",
StartSessionError::TransportClosed => "protocol_error",
StartSessionError::ExtensionError(err) => match err.code {
bsk_protocol::ErrorCode::Timeout => "timeout",
@@ -407,6 +418,7 @@ pub struct AgentWindowOptions {
/// On success the matching per-session dispatch queue (M6.5,
/// design §5) is also spawned so that subsequent `tool.*` RPCs serialise
/// against this session's Agent Window / ref-store.
#[allow(clippy::too_many_arguments)]
pub async fn start_session(
registry: &Arc<BrowserRegistry>,
sessions: &Arc<SessionRegistry>,
@@ -415,24 +427,32 @@ pub async fn start_session(
window: AgentWindowOptions,
connect_wait: Duration,
timeout_dur: Duration,
cancel: Option<AbortToken>,
) -> Result<Session, StartSessionError> {
let client: Arc<BrowserClient> = registry
.select_with_connect_wait(requested, connect_wait)
.await
.map_err(|e| match e {
SelectError::NoBrowserConnected => StartSessionError::NoBrowserConnected,
SelectError::MultipleBrowsersOnline => StartSessionError::MultipleBrowsersOnline {
browsers: snapshot_status_entries(registry, sessions),
},
SelectError::NotFound => StartSessionError::BrowserNotFound,
SelectError::AmbiguousLabel {
label,
instance_ids,
} => StartSessionError::AmbiguousBrowserLabel {
label,
instance_ids,
},
})?;
let selection = registry.select_with_connect_wait(requested, connect_wait);
tokio::pin!(selection);
let selected = match cancel.as_ref() {
Some(token) => tokio::select! {
biased;
_ = token.cancelled() => return Err(StartSessionError::Cancelled),
selected = &mut selection => selected,
},
None => selection.await,
};
let client: Arc<BrowserClient> = selected.map_err(|e| match e {
SelectError::NoBrowserConnected => StartSessionError::NoBrowserConnected,
SelectError::MultipleBrowsersOnline => StartSessionError::MultipleBrowsersOnline {
browsers: snapshot_status_entries(registry, sessions),
},
SelectError::NotFound => StartSessionError::BrowserNotFound,
SelectError::AmbiguousLabel {
label,
instance_ids,
} => StartSessionError::AmbiguousBrowserLabel {
label,
instance_ids,
},
})?;
let session_id = sessions
.reserve_id(client.id.clone(), SESSION_ID_MAX_RESERVE_ATTEMPTS, now_ms)
.ok_or(StartSessionError::IdExhausted)?;
@@ -453,22 +473,71 @@ pub async fn start_session(
let mut pending = client.pending.lock().unwrap();
pending.register(rpc_id.clone())
};
if cancel.as_ref().is_some_and(AbortToken::is_cancelled) {
sessions.cancel_reservation(&session_id);
client.pending.lock().unwrap().cancel(&rpc_id);
return Err(StartSessionError::Cancelled);
}
if client.sink.send(Frame::Request(request)).is_err() {
sessions.cancel_reservation(&session_id);
client.pending.lock().unwrap().cancel(&rpc_id);
return Err(StartSessionError::TransportClosed);
}
let response = match timeout(timeout_dur, waiter).await {
Ok(Ok(resp)) => resp,
Ok(Err(_)) => {
let mut waiter = waiter;
enum StartWaitOutcome {
Response(bsk_protocol::ResponseFrame),
WaiterClosed,
Cancelled,
Timeout,
}
let wait_outcome = match cancel.as_ref() {
Some(token) => {
let deadline = tokio::time::sleep(timeout_dur);
tokio::pin!(deadline);
tokio::select! {
biased;
_ = token.cancelled() => StartWaitOutcome::Cancelled,
response = &mut waiter => match response {
Ok(response) => StartWaitOutcome::Response(response),
Err(_) => StartWaitOutcome::WaiterClosed,
},
_ = &mut deadline => StartWaitOutcome::Timeout,
}
}
None => match timeout(timeout_dur, &mut waiter).await {
Ok(Ok(response)) => StartWaitOutcome::Response(response),
Ok(Err(_)) => StartWaitOutcome::WaiterClosed,
Err(_) => StartWaitOutcome::Timeout,
},
};
let response = match wait_outcome {
StartWaitOutcome::Response(response) => response,
StartWaitOutcome::WaiterClosed => {
sessions.cancel_reservation(&session_id);
client.pending.lock().unwrap().cancel(&rpc_id);
return Err(StartSessionError::TransportClosed);
}
Err(_) => {
sessions.cancel_reservation(&session_id);
client.pending.lock().unwrap().cancel(&rpc_id);
return Err(StartSessionError::Timeout);
StartWaitOutcome::Cancelled => {
return Err(finish_aborted_start(
&client,
sessions,
&session_id,
&rpc_id,
waiter,
StartAbortReason::Cancelled,
)
.await);
}
StartWaitOutcome::Timeout => {
return Err(finish_aborted_start(
&client,
sessions,
&session_id,
&rpc_id,
waiter,
StartAbortReason::Timeout,
)
.await);
}
};
let agent_window_id = match response.body {
@@ -504,6 +573,124 @@ pub async fn start_session(
Ok(session)
}
#[derive(Clone, Copy)]
enum StartAbortReason {
Cancelled,
Timeout,
}
impl StartAbortReason {
fn error(self) -> StartSessionError {
match self {
Self::Cancelled => StartSessionError::Cancelled,
Self::Timeout => StartSessionError::Timeout,
}
}
}
async fn finish_aborted_start(
client: &Arc<BrowserClient>,
sessions: &Arc<SessionRegistry>,
session_id: &SessionId,
rpc_id: &RpcId,
mut waiter: tokio::sync::oneshot::Receiver<bsk_protocol::ResponseFrame>,
reason: StartAbortReason,
) -> StartSessionError {
sessions.cancel_reservation(session_id);
let cancel = Frame::Request(RequestFrame {
id: format!("cancel-{rpc_id}"),
method: Method::Cancel,
params: Some(serde_json::json!({ "rpc_id": rpc_id })),
});
if client.sink.send(cancel).is_err() {
client.pending.lock().unwrap().cancel(rpc_id);
return StartSessionError::TransportClosed;
}
match timeout(CANCEL_CLEANUP_TIMEOUT, &mut waiter).await {
Ok(Ok(response)) => match response.body {
ResponseBody::Err(err)
if matches!(err.code, ErrorCode::Cancelled | ErrorCode::UserAborted) =>
{
reason.error()
}
ResponseBody::Err(err) => StartSessionError::ExtensionError(err),
ResponseBody::Ok(value) => {
let agent_window_id = serde_json::from_value::<SessionStartResult>(value)
.ok()
.and_then(|result| result.agent_window_id);
match rollback_extension_session(client, session_id).await {
Ok(()) => reason.error(),
Err(message) => StartSessionError::CleanupFailed {
session_id: session_id.clone(),
agent_window_id,
message,
},
}
}
},
Ok(Err(_)) => {
client.pending.lock().unwrap().cancel(rpc_id);
StartSessionError::TransportClosed
}
Err(_) => {
// The cancel frame is already queued behind session_start. Even
// if chrome.windows.create is still blocked, the extension will
// observe the aborted signal and compensate before replying.
client.pending.lock().unwrap().cancel(rpc_id);
reason.error()
}
}
}
async fn rollback_extension_session(
client: &Arc<BrowserClient>,
session_id: &SessionId,
) -> Result<(), String> {
let rollback_id = next_rpc_id("sess-start-rollback");
let waiter = {
let mut pending = client.pending.lock().unwrap();
pending.register(rollback_id.clone())
};
let request = RequestFrame {
id: rollback_id.clone(),
method: Method::ToolSessionStop,
params: Some(
serde_json::to_value(SessionStopParams {
session_id: session_id.0.clone(),
})
.expect("serialize session_start rollback params"),
),
};
if client.sink.send(Frame::Request(request)).is_err() {
client.pending.lock().unwrap().cancel(&rollback_id);
return Err("browser disconnected before rollback was queued".into());
}
let response = match timeout(CANCEL_CLEANUP_TIMEOUT, waiter).await {
Ok(Ok(response)) => response,
Ok(Err(_)) => return Err("browser disconnected during rollback".into()),
Err(_) => {
client.pending.lock().unwrap().cancel(&rollback_id);
return Err(format!(
"extension did not finish rollback within {CANCEL_CLEANUP_TIMEOUT:?}"
));
}
};
match response.body {
ResponseBody::Ok(value) => {
let result: SessionStopResult = serde_json::from_value(value)
.map_err(|err| format!("invalid rollback response: {err}"))?;
if result.return_failures.is_empty() {
Ok(())
} else {
Err("rollback reported borrowed-tab return failures".into())
}
}
ResponseBody::Err(err) if err.code == ErrorCode::NotFound => Ok(()),
ResponseBody::Err(err) => Err(format!("rollback failed: {} ({:?})", err.message, err.code)),
}
}
/// Tear down a single session id (round-trips `tool.session_stop` to the
/// owning extension and unregisters on success). Also tears down the
/// per-session dispatch queue so the worker task exits.
@@ -514,6 +701,7 @@ pub async fn stop_session(
interrupts: &Arc<SessionInterruptRegistry>,
session_id: &SessionId,
timeout_dur: Duration,
cancel: Option<AbortToken>,
) -> Result<SessionStopResult, StopSessionError> {
let session = sessions.get(session_id).ok_or(StopSessionError::NotFound)?;
if registry.get(&session.browser_id).is_none() {
@@ -524,12 +712,17 @@ pub async fn stop_session(
session_id: session_id.0.clone(),
})
.unwrap();
// Internal callers (idle reaping and shutdown) do not have a CLI-owned
// token, but still need timeout-driven WS cancellation so a late window
// close cannot leave daemon state pointing at a vanished session.
let lifecycle_cancel = Some(cancel.unwrap_or_default());
match queues
.dispatch_after_closing(
session_id,
bsk_protocol::Method::ToolSessionStop,
params,
timeout_dur,
lifecycle_cancel,
)
.await
{
@@ -567,6 +760,10 @@ pub async fn stop_session(
drop_session_local(queues, interrupts, session_id);
return Ok(SessionStopResult::default());
}
if err.code == bsk_protocol::ErrorCode::Timeout {
queues.reopen(session_id);
return Err(StopSessionError::Timeout);
}
queues.reopen(session_id);
Err(StopSessionError::ExtensionError(err))
}
+96 -27
View File
@@ -15,7 +15,6 @@ use std::net::{IpAddr, Ipv4Addr, SocketAddr};
use std::path::Path;
use std::sync::Arc;
use std::sync::Mutex;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::{Duration, Instant};
use anyhow::{Context, Result};
@@ -291,8 +290,7 @@ pub fn run_foreground(cfg: DaemonConfig) -> Result<()> {
});
let started_at = Instant::now();
let activity: Arc<Mutex<Instant>> = Arc::new(Mutex::new(started_at));
let active_ipc_connections = Arc::new(AtomicUsize::new(0));
let activity = Arc::new(Mutex::new(IpcActivityState::new(started_at)));
let (ipc_shutdown_tx, ipc_shutdown_rx) = tokio::sync::oneshot::channel::<()>();
let status = ipc::DaemonStatus {
started_at,
@@ -314,18 +312,14 @@ pub fn run_foreground(cfg: DaemonConfig) -> Result<()> {
let handler = handler.clone();
let ipc_open = {
let activity = activity.clone();
let active = active_ipc_connections.clone();
move || {
active.fetch_add(1, Ordering::SeqCst);
record_activity(&activity);
record_ipc_open(&activity);
}
};
let ipc_close = {
let activity = activity.clone();
let active = active_ipc_connections.clone();
move || {
active.fetch_sub(1, Ordering::SeqCst);
record_activity(&activity);
record_ipc_close(&activity);
}
};
tokio::spawn(ipc::serve(
@@ -344,7 +338,6 @@ pub fn run_foreground(cfg: DaemonConfig) -> Result<()> {
let idle_task = {
let activity = activity.clone();
let state = Arc::clone(&state);
let active_ipc_connections = active_ipc_connections.clone();
let daemon_idle = cfg.daemon_idle;
tokio::spawn(async move {
let tick = (daemon_idle / 4).max(Duration::from_millis(250));
@@ -352,22 +345,19 @@ pub fn run_foreground(cfg: DaemonConfig) -> Result<()> {
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
loop {
ticker.tick().await;
let last = match activity.lock() {
Ok(a) => *a,
Err(p) => *p.into_inner(),
};
if last.elapsed() < daemon_idle {
let ipc_is_idle =
lock_activity(&activity).should_exit_for_idle(Instant::now(), daemon_idle);
if !ipc_is_idle {
continue;
}
// Bridge from M2/M3 (which only knew about connection
// counts) to M4/M5 (browser + session registries):
// hold the daemon alive while any IPC client is
// connected, any browser is paired, or any session
// is live (design §3.2).
if active_ipc_connections.load(Ordering::SeqCst) > 0
|| !state.browsers.is_empty()
|| !state.sessions.is_empty()
{
// IPC liveness (open-connection count + idle window) is already
// enforced above by `should_exit_for_idle`, which reads
// `IpcActivityState` under one lock and returns true only when
// there are zero connections AND the idle interval has elapsed.
// Here we additionally hold the daemon alive while any browser
// is paired or any session is live (design §3.2, M4/M5
// registries).
if !state.browsers.is_empty() || !state.sessions.is_empty() {
continue;
}
info!(
@@ -491,6 +481,7 @@ pub(crate) fn spawn_session_idle_reaper(state: Arc<DaemonState>) -> tokio::task:
&state.session_interrupts,
&session_id,
Duration::from_secs(10),
None,
)
.await
{
@@ -681,10 +672,55 @@ fn restart_start_args(cfg: &DaemonConfig) -> StartArgs {
}
}
fn record_activity(activity: &Arc<Mutex<Instant>>) {
if let Ok(mut a) = activity.lock() {
*a = Instant::now();
#[derive(Debug)]
struct IpcActivityState {
last_activity: Instant,
active_connections: usize,
}
impl IpcActivityState {
fn new(now: Instant) -> Self {
Self {
last_activity: now,
active_connections: 0,
}
}
fn touch(&mut self, now: Instant) {
self.last_activity = now;
}
fn opened(&mut self, now: Instant) {
self.active_connections = self.active_connections.saturating_add(1);
self.last_activity = now;
}
fn closed(&mut self, now: Instant) {
self.active_connections = self.active_connections.saturating_sub(1);
self.last_activity = now;
}
fn should_exit_for_idle(&self, now: Instant, idle: Duration) -> bool {
self.active_connections == 0 && now.saturating_duration_since(self.last_activity) >= idle
}
}
fn lock_activity(
activity: &Arc<Mutex<IpcActivityState>>,
) -> std::sync::MutexGuard<'_, IpcActivityState> {
activity.lock().unwrap_or_else(|poison| poison.into_inner())
}
fn record_activity(activity: &Arc<Mutex<IpcActivityState>>) {
lock_activity(activity).touch(Instant::now());
}
fn record_ipc_open(activity: &Arc<Mutex<IpcActivityState>>) {
lock_activity(activity).opened(Instant::now());
}
fn record_ipc_close(activity: &Arc<Mutex<IpcActivityState>>) {
lock_activity(activity).closed(Instant::now());
}
/// Initialise tracing for the daemon: write a daily-rotated file in
@@ -1067,4 +1103,37 @@ mod tests {
assert_eq!(args.session_idle, Some(Duration::from_secs(11)));
assert_eq!(args.daemon_idle, Some(Duration::from_secs(22)));
}
#[test]
fn ipc_close_restarts_the_idle_window_atomically() {
let started = Instant::now();
let idle = Duration::from_secs(10);
let mut activity = IpcActivityState::new(started);
activity.opened(started);
let closed_at = started + Duration::from_secs(30);
assert!(
!activity.should_exit_for_idle(closed_at, idle),
"an open IPC connection must hold the daemon alive"
);
activity.closed(closed_at);
assert_eq!(activity.active_connections, 0);
assert_eq!(activity.last_activity, closed_at);
assert!(!activity.should_exit_for_idle(closed_at, idle));
assert!(!activity.should_exit_for_idle(closed_at + idle - Duration::from_millis(1), idle));
assert!(activity.should_exit_for_idle(closed_at + idle, idle));
}
#[test]
fn ipc_activity_keeps_the_daemon_alive_until_a_full_idle_interval_passes() {
let started = Instant::now();
let idle = Duration::from_secs(10);
let mut activity = IpcActivityState::new(started);
let request_at = started + Duration::from_secs(20);
activity.touch(request_at);
assert!(!activity.should_exit_for_idle(request_at + Duration::from_secs(9), idle));
assert!(activity.should_exit_for_idle(request_at + idle, idle));
}
}
+2 -2
View File
@@ -34,8 +34,8 @@ pub struct DaemonState {
/// `stop_session` / browser disconnect.
pub tool_queues: Arc<ToolQueueRegistry>,
/// Per-rpc-id cancellation tokens for daemon-side long-runners
/// (M9.3 — currently only `tool.wait_ms`). The CLI's `cancel
/// { rpc_id }` consults this registry first.
/// (`tool.wait_ms` plus `session.*` lifecycle calls). The CLI's
/// `cancel { rpc_id }` consults this registry first.
pub abort_registry: Arc<AbortRegistry>,
/// Tracks `tool.*` RPCs that have been forwarded to an extension
/// over WS but have not yet received a response. Indexed by the
@@ -325,6 +325,7 @@ async fn session_stop_fast_fails_while_tool_is_in_flight() {
&state.session_interrupts,
&sid,
Duration::from_secs(5),
None,
)
.await;
assert!(
@@ -366,6 +367,7 @@ async fn session_stop_fast_fails_while_tool_is_in_flight() {
&state.session_interrupts,
&sid,
Duration::from_secs(5),
None,
)
.await
.expect("session.stop ok after tool completes");
+368 -10
View File
@@ -10,9 +10,14 @@ use std::time::Duration;
use bsk::daemon::{self, DaemonConfig};
use bsk::ipc_client::IpcClient;
use bsk_protocol::cancel::{CancelParams, CancelResult};
use bsk_protocol::system::{HandshakeParams, HandshakeResult, StatusResult};
use bsk_protocol::tools::{SessionStartParams, SessionStartResult, SessionStopParams};
use bsk_protocol::{BrowserPeerInfo, Frame, Method, RequestFrame, ResponseBody, ResponseFrame};
use bsk_protocol::tools::{
SessionStartParams, SessionStartResult, SessionStopParams, SessionStopResult,
};
use bsk_protocol::{
BrowserPeerInfo, ErrorCode, Frame, Method, RequestFrame, ResponseBody, ResponseFrame, RpcError,
};
use futures_util::{SinkExt, StreamExt};
use rand::Rng;
use tokio_tungstenite::tungstenite::handshake::client::generate_key;
@@ -23,6 +28,9 @@ use support::{wait_for_browser_count, wait_for_no_sessions};
const TEST_EXT_ID: &str = "abcdefghijklmnopabcdefghijklmnop";
type TestWs =
tokio_tungstenite::WebSocketStream<tokio_tungstenite::MaybeTlsStream<tokio::net::TcpStream>>;
fn tempfile_path(prefix: &str) -> PathBuf {
let mut p = std::env::temp_dir();
let mut rng = rand::thread_rng();
@@ -56,9 +64,7 @@ async fn spawn_daemon_with_session_idle(session_idle: Duration) -> (daemon::Daem
(handle, sock)
}
async fn connect_ext(
addr: std::net::SocketAddr,
) -> tokio_tungstenite::WebSocketStream<tokio_tungstenite::MaybeTlsStream<tokio::net::TcpStream>> {
async fn connect_ext(addr: std::net::SocketAddr) -> TestWs {
let origin = format!("chrome-extension://{TEST_EXT_ID}");
let url = format!("ws://{addr}/");
let req = Request::builder()
@@ -76,11 +82,7 @@ async fn connect_ext(
ws
}
async fn handshake_as_ext(
ws: &mut tokio_tungstenite::WebSocketStream<
tokio_tungstenite::MaybeTlsStream<tokio::net::TcpStream>,
>,
) -> HandshakeResult {
async fn handshake_as_ext(ws: &mut TestWs) -> HandshakeResult {
let params = HandshakeParams {
client: "browser-skill-extension".into(),
version: "0.1.0-dev.0".parse().unwrap(),
@@ -114,6 +116,128 @@ async fn handshake_as_ext(
}
}
async fn next_extension_request(ws: &mut TestWs) -> RequestFrame {
loop {
let message = ws.next().await.expect("daemon closed websocket").unwrap();
match message {
Message::Text(text) => match serde_json::from_str::<Frame>(&text).unwrap() {
Frame::Request(request) => return request,
Frame::Response(_) | Frame::Event(_) => continue,
},
Message::Ping(payload) => ws.send(Message::Pong(payload)).await.unwrap(),
Message::Close(_) => panic!("daemon closed websocket before sending request"),
_ => {}
}
}
}
async fn send_extension_response(ws: &mut TestWs, response: ResponseFrame) {
ws.send(Message::Text(serde_json::to_string(&response).unwrap()))
.await
.unwrap();
}
async fn acknowledge_extension_cancel(ws: &mut TestWs, cancel: RequestFrame, target_rpc_id: &str) {
assert_eq!(cancel.method, Method::Cancel);
let params: CancelParams = serde_json::from_value(cancel.params.unwrap()).unwrap();
assert_eq!(params.rpc_id, target_rpc_id);
send_extension_response(
ws,
ResponseFrame {
id: cancel.id,
body: ResponseBody::Ok(serde_json::to_value(CancelResult { cancelled: true }).unwrap()),
},
)
.await;
}
async fn respond_to_aborted_start(
ws: &mut TestWs,
start_seen: Option<tokio::sync::oneshot::Sender<()>>,
agent_window_id: i64,
) {
let start = next_extension_request(ws).await;
assert_eq!(start.method, Method::ToolSessionStart);
let params: SessionStartParams = serde_json::from_value(start.params.clone().unwrap()).unwrap();
if let Some(start_seen) = start_seen {
start_seen.send(()).unwrap();
}
let cancel = next_extension_request(ws).await;
acknowledge_extension_cancel(ws, cancel, &start.id).await;
// Model the hardest race: chrome.windows.create completed just as
// cancellation arrived, so the extension's original call succeeds.
send_extension_response(
ws,
ResponseFrame {
id: start.id,
body: ResponseBody::Ok(
serde_json::to_value(SessionStartResult {
agent_window_id: Some(agent_window_id),
})
.unwrap(),
),
},
)
.await;
let rollback = next_extension_request(ws).await;
assert_eq!(rollback.method, Method::ToolSessionStop);
let rollback_params: SessionStopParams =
serde_json::from_value(rollback.params.unwrap()).unwrap();
assert_eq!(rollback_params.session_id, params.session_id);
send_extension_response(
ws,
ResponseFrame {
id: rollback.id,
body: ResponseBody::Ok(serde_json::to_value(SessionStopResult::default()).unwrap()),
},
)
.await;
}
async fn respond_to_aborted_stop(
ws: &mut TestWs,
stop_seen: Option<tokio::sync::oneshot::Sender<()>>,
) {
let start = next_extension_request(ws).await;
assert_eq!(start.method, Method::ToolSessionStart);
send_extension_response(
ws,
ResponseFrame {
id: start.id,
body: ResponseBody::Ok(
serde_json::to_value(SessionStartResult {
agent_window_id: Some(4242),
})
.unwrap(),
),
},
)
.await;
let stop = next_extension_request(ws).await;
assert_eq!(stop.method, Method::ToolSessionStop);
if let Some(stop_seen) = stop_seen {
stop_seen.send(()).unwrap();
}
let cancel = next_extension_request(ws).await;
acknowledge_extension_cancel(ws, cancel, &stop.id).await;
send_extension_response(
ws,
ResponseFrame {
id: stop.id,
body: ResponseBody::Err(RpcError {
code: ErrorCode::Cancelled,
message: "session_stop aborted before window close".into(),
data: None,
}),
},
)
.await;
}
#[tokio::test]
async fn session_start_stop_round_trip_via_ipc() {
let (handle, sock) = spawn_daemon().await;
@@ -250,6 +374,240 @@ async fn session_start_stop_round_trip_via_ipc() {
handle.shutdown().await;
}
#[tokio::test]
async fn cancelling_session_start_rolls_back_a_late_extension_success() {
const START_RPC_ID: &str = "session-start-cancel";
let (handle, sock) = spawn_daemon().await;
let mut ws = connect_ext(handle.ws_addr()).await;
let _ = handshake_as_ext(&mut ws).await;
let (start_seen_tx, start_seen_rx) = tokio::sync::oneshot::channel();
let (release_tx, release_rx) = tokio::sync::oneshot::channel();
let responder = tokio::spawn(async move {
respond_to_aborted_start(&mut ws, Some(start_seen_tx), 4242).await;
let _ = release_rx.await;
});
let mut start_ipc = IpcClient::connect(&sock).await.unwrap();
let start_call = tokio::spawn(async move {
let outcome: Result<serde_json::Value, RpcError> = start_ipc
.call_with_id(
START_RPC_ID.to_string(),
Method::SessionStart,
Some(serde_json::json!({ "focused": false })),
Duration::from_secs(5),
)
.await
.unwrap();
outcome
});
start_seen_rx.await.unwrap();
let mut cancel_ipc = IpcClient::connect(&sock).await.unwrap();
let cancel: CancelResult = cancel_ipc
.call(
"cancel-start",
Method::Cancel,
Some(CancelParams {
rpc_id: START_RPC_ID.to_string(),
}),
Duration::from_secs(2),
)
.await
.unwrap()
.unwrap();
assert!(cancel.cancelled);
let start_error = start_call.await.unwrap().unwrap_err();
assert_eq!(start_error.code, ErrorCode::Cancelled);
let mut status_ipc = IpcClient::connect(&sock).await.unwrap();
let status: StatusResult = status_ipc
.call::<(), _>(
"status-after-start-cancel",
Method::SystemStatus,
None,
Duration::from_secs(2),
)
.await
.unwrap()
.unwrap();
assert!(status.sessions.is_empty());
assert_eq!(status.browsers[0].session_count, 0);
let _ = release_tx.send(());
responder.await.unwrap();
handle.shutdown().await;
}
#[tokio::test]
async fn timing_out_session_start_rolls_back_a_late_extension_success() {
let (handle, _sock) = spawn_daemon().await;
let state = handle.state();
let mut ws = connect_ext(handle.ws_addr()).await;
let _ = handshake_as_ext(&mut ws).await;
let (release_tx, release_rx) = tokio::sync::oneshot::channel();
let responder = tokio::spawn(async move {
respond_to_aborted_start(&mut ws, None, 4343).await;
let _ = release_rx.await;
});
let start = bsk::daemon::sessions::start_session(
&state.browsers,
&state.sessions,
&state.tool_queues,
None,
bsk::daemon::sessions::AgentWindowOptions::default(),
Duration::ZERO,
Duration::from_millis(20),
None,
)
.await;
assert!(matches!(
start,
Err(bsk::daemon::sessions::StartSessionError::Timeout)
));
assert!(state.sessions.is_empty());
let _ = release_tx.send(());
responder.await.unwrap();
handle.shutdown().await;
}
#[tokio::test]
async fn cancelling_session_stop_keeps_the_session_retryable() {
const STOP_RPC_ID: &str = "session-stop-cancel";
let (handle, sock) = spawn_daemon().await;
let mut ws = connect_ext(handle.ws_addr()).await;
let _ = handshake_as_ext(&mut ws).await;
let (stop_seen_tx, stop_seen_rx) = tokio::sync::oneshot::channel();
let (release_tx, release_rx) = tokio::sync::oneshot::channel();
let responder = tokio::spawn(async move {
respond_to_aborted_stop(&mut ws, Some(stop_seen_tx)).await;
let _ = release_rx.await;
});
let mut start_ipc = IpcClient::connect(&sock).await.unwrap();
let start: serde_json::Value = start_ipc
.call(
"start-before-stop-cancel",
Method::SessionStart,
Some(serde_json::json!({ "focused": false })),
Duration::from_secs(5),
)
.await
.unwrap()
.unwrap();
let session_id = start["session_id"].as_str().unwrap().to_string();
let mut stop_ipc = IpcClient::connect(&sock).await.unwrap();
let stop_session_id = session_id.clone();
let stop_call = tokio::spawn(async move {
let outcome: Result<serde_json::Value, RpcError> = stop_ipc
.call_with_id(
STOP_RPC_ID.to_string(),
Method::SessionStop,
Some(serde_json::json!({
"session_id": stop_session_id,
"all": false,
})),
Duration::from_secs(5),
)
.await
.unwrap();
outcome
});
stop_seen_rx.await.unwrap();
let mut cancel_ipc = IpcClient::connect(&sock).await.unwrap();
let cancel: CancelResult = cancel_ipc
.call(
"cancel-stop",
Method::Cancel,
Some(CancelParams {
rpc_id: STOP_RPC_ID.to_string(),
}),
Duration::from_secs(2),
)
.await
.unwrap()
.unwrap();
assert!(cancel.cancelled);
let stop_error = stop_call.await.unwrap().unwrap_err();
assert_eq!(stop_error.code, ErrorCode::Cancelled);
let mut status_ipc = IpcClient::connect(&sock).await.unwrap();
let status: StatusResult = status_ipc
.call::<(), _>(
"status-after-stop-cancel",
Method::SystemStatus,
None,
Duration::from_secs(2),
)
.await
.unwrap()
.unwrap();
assert_eq!(status.sessions.len(), 1);
assert_eq!(status.sessions[0].session_id, session_id);
let _ = release_tx.send(());
responder.await.unwrap();
handle.shutdown().await;
}
#[tokio::test]
async fn timing_out_session_stop_cancels_extension_teardown_and_keeps_session() {
let (handle, sock) = spawn_daemon().await;
let state = handle.state();
let mut ws = connect_ext(handle.ws_addr()).await;
let _ = handshake_as_ext(&mut ws).await;
let (release_tx, release_rx) = tokio::sync::oneshot::channel();
let responder = tokio::spawn(async move {
respond_to_aborted_stop(&mut ws, None).await;
let _ = release_rx.await;
});
let mut start_ipc = IpcClient::connect(&sock).await.unwrap();
let start: serde_json::Value = start_ipc
.call(
"start-before-stop-timeout",
Method::SessionStart,
Some(serde_json::json!({ "focused": false })),
Duration::from_secs(5),
)
.await
.unwrap()
.unwrap();
let session_id =
bsk::daemon::sessions::SessionId(start["session_id"].as_str().unwrap().to_string());
let stop = bsk::daemon::sessions::stop_session(
&state.browsers,
&state.sessions,
&state.tool_queues,
&state.session_interrupts,
&session_id,
Duration::from_millis(20),
None,
)
.await;
assert!(matches!(
stop,
Err(bsk::daemon::sessions::StopSessionError::Timeout)
));
assert!(state.sessions.get(&session_id).is_some());
let _ = release_tx.send(());
responder.await.unwrap();
handle.shutdown().await;
}
#[tokio::test]
async fn session_idle_timeout_stops_and_unregisters_session() {
let (handle, _sock) = spawn_daemon_with_session_idle(Duration::from_millis(50)).await;
@@ -7,10 +7,18 @@
],
"properties": {
"trace": {
"$ref": "#/definitions/Trace"
"$ref": "#/definitions/RecordedTrace"
}
},
"definitions": {
"FillCommit": {
"type": "string",
"enum": [
"enter",
"suggestion",
"blur"
]
},
"KeyModifier": {
"description": "Keyboard modifier flags. Multiple flags may be combined; the extension folds them into CDP's bitfield (`alt=1, ctrl=2, meta=4, shift=8`).",
"type": "string",
@@ -21,7 +29,19 @@
"shift"
]
},
"PageRef": {
"NavigationCause": {
"type": "string",
"enum": [
"user_typed",
"link",
"form_submit",
"reload",
"history",
"script",
"browser"
]
},
"PageRefV2": {
"description": "Page context dictionary entry — referenced by steps via `page` id.",
"type": "object",
"required": [
@@ -43,7 +63,42 @@
}
}
},
"SelectedOption": {
"RecordedTrace": {
"if": {
"required": [
"version"
],
"properties": {
"version": {
"type": "integer"
}
}
},
"then": {
"$ref": "#/definitions/TraceV3"
},
"else": {
"$ref": "#/definitions/TraceV2"
}
},
"RecorderInfo": {
"type": "object",
"required": [
"bsk",
"vom"
],
"properties": {
"bsk": {
"type": "string"
},
"vom": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
}
}
},
"SelectedOptionV2": {
"description": "One selected option (`select` op).",
"type": "object",
"required": [
@@ -61,11 +116,53 @@
}
}
},
"Step": {
"description": "One recorded user action — discriminated union by `op`.",
"SelectedOptionV3": {
"description": "One selected option (`select` op).",
"type": "object",
"required": [
"value"
],
"properties": {
"label": {
"type": [
"string",
"null"
]
},
"value": {
"type": "string"
}
}
},
"StepEffectV2": {
"description": "Observed navigation after a step (objective fact only).",
"type": "object",
"required": [
"navigated_to"
],
"properties": {
"navigated_to": {
"description": "Reference into `pages[]` for the destination page.",
"type": "string"
}
}
},
"StepResultV3": {
"type": "object",
"required": [
"state"
],
"properties": {
"state": {
"type": "string"
}
}
},
"StepV2": {
"description": "One recorded user action — discriminated union by `op` (v2).",
"oneOf": [
{
"description": "Fields shared by every step variant (flattened in JSON).",
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
@@ -77,7 +174,7 @@
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
@@ -105,7 +202,7 @@
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
@@ -117,7 +214,7 @@
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
@@ -140,12 +237,12 @@
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
"$ref": "#/definitions/TargetDescriptorV2"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
@@ -157,7 +254,7 @@
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
@@ -180,12 +277,12 @@
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
"$ref": "#/definitions/TargetDescriptorV2"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
@@ -198,7 +295,7 @@
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
@@ -227,7 +324,7 @@
]
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
"$ref": "#/definitions/TargetDescriptorV2"
},
"value": {
"type": "string"
@@ -235,7 +332,7 @@
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
@@ -248,7 +345,7 @@
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
@@ -273,16 +370,16 @@
"selection": {
"type": "array",
"items": {
"$ref": "#/definitions/SelectedOption"
"$ref": "#/definitions/SelectedOptionV2"
}
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
"$ref": "#/definitions/TargetDescriptorV2"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
@@ -294,7 +391,7 @@
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
@@ -331,7 +428,7 @@
"target": {
"anyOf": [
{
"$ref": "#/definitions/TargetDescriptor"
"$ref": "#/definitions/TargetDescriptorV2"
},
{
"type": "null"
@@ -342,21 +439,294 @@
}
]
},
"StepEffect": {
"description": "Observed navigation after a step (objective fact only).",
"type": "object",
"required": [
"navigated_to"
],
"properties": {
"navigated_to": {
"description": "Reference into `pages[]` for the destination page.",
"type": "string"
"StepV3": {
"description": "One recorded user action — discriminated union by `op`.",
"oneOf": [
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"cause",
"id",
"op",
"result",
"state",
"to"
],
"properties": {
"cause": {
"$ref": "#/definitions/NavigationCause"
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"navigate"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"to": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state",
"target"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"click"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state",
"target"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"hover"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"commit",
"id",
"op",
"result",
"state",
"target",
"value"
],
"properties": {
"commit": {
"$ref": "#/definitions/FillCommit"
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"fill"
]
},
"redacted": {
"type": "boolean"
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
},
"value": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state",
"target"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"select"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"selection": {
"type": "array",
"items": {
"$ref": "#/definitions/SelectedOptionV3"
}
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"key",
"op",
"result",
"state"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"key": {
"type": "string"
},
"modifiers": {
"type": [
"array",
"null"
],
"items": {
"$ref": "#/definitions/KeyModifier"
}
},
"op": {
"type": "string",
"enum": [
"press"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"anyOf": [
{
"$ref": "#/definitions/TargetDescriptorV3"
},
{
"type": "null"
}
]
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"scroll"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
}
}
}
}
]
},
"TargetDescriptor": {
"description": "Stable semantic handle for an interacted element.\n\n`name` and `nearby_label` are **untrusted page text**.",
"StopReason": {
"type": "string",
"enum": [
"user_finish",
"cli_stop"
]
},
"TargetDescriptorV2": {
"description": "Stable semantic handle for an interacted element (v2).\n\n`name` and `nearby_label` are **untrusted page text**.",
"type": "object",
"required": [
"tag"
@@ -397,8 +767,83 @@
}
}
},
"Trace": {
"description": "Persisted user-action trace exported by `tool.record_stop` / `await`.",
"TargetDescriptorV3": {
"description": "Stable semantic handle for an interacted element within a page observation.",
"type": "object",
"properties": {
"ctx": {
"type": [
"string",
"null"
]
},
"name": {
"type": [
"string",
"null"
]
},
"ref": {
"type": [
"string",
"null"
]
},
"role": {
"type": [
"string",
"null"
]
},
"unmatched": {
"type": "boolean"
}
}
},
"TraceEntry": {
"type": "object",
"required": [
"start_url"
],
"properties": {
"start_url": {
"type": "string"
}
}
},
"TraceStateV3": {
"description": "Page observation dictionary entry — referenced by steps via `state` / `result.state`.",
"type": "object",
"required": [
"body",
"id",
"url"
],
"properties": {
"body": {
"description": "Full page observation (front matter + VOM body + annotations).",
"type": "string"
},
"id": {
"type": "string"
},
"title": {
"type": [
"string",
"null"
]
},
"truncated": {
"type": "boolean"
},
"url": {
"type": "string"
}
}
},
"TraceV2": {
"title": "TraceV2",
"description": "Persisted user-action trace exported by legacy `tool.record_stop` / `await`.\n\nUnknown extension fields are ignored so older traces remain readable. `states[]` and a numeric `version` are reserved for Trace v3 / `RecordedTrace` classification.",
"type": "object",
"required": [
"entry",
@@ -413,11 +858,10 @@
"pages": {
"type": "array",
"items": {
"$ref": "#/definitions/PageRef"
"$ref": "#/definitions/PageRefV2"
}
},
"purpose": {
"description": "Optional user-provided goal from `--purpose` (metadata only).",
"type": [
"string",
"null"
@@ -433,25 +877,76 @@
"null"
]
},
"states": false,
"steps": {
"type": "array",
"items": {
"$ref": "#/definitions/Step"
"$ref": "#/definitions/StepV2"
}
},
"version": {
"description": "Numeric version selects Trace v3. Legacy v2 envelopes omit this field.",
"not": {
"type": "integer"
}
}
}
},
"TraceEntry": {
"description": "Recording entry point — first URL the flow starts from.",
"TraceV3": {
"description": "Wire trace returned by `tool.record_stop` / `await`.",
"type": "object",
"required": [
"start_url"
"entry",
"recorded_at",
"recorder",
"states",
"steps",
"stopped_by",
"version"
],
"properties": {
"start_url": {
"entry": {
"$ref": "#/definitions/TraceEntry"
},
"purpose": {
"type": [
"string",
"null"
]
},
"recorded_at": {
"type": "string"
},
"recorder": {
"$ref": "#/definitions/RecorderInfo"
},
"started_at": {
"type": [
"string",
"null"
]
},
"states": {
"type": "array",
"items": {
"$ref": "#/definitions/TraceStateV3"
}
},
"steps": {
"type": "array",
"items": {
"$ref": "#/definitions/StepV3"
}
},
"stopped_by": {
"$ref": "#/definitions/StopReason"
},
"version": {
"type": "integer",
"const": 3
}
}
},
"additionalProperties": false
}
}
}
@@ -7,10 +7,18 @@
],
"properties": {
"trace": {
"$ref": "#/definitions/Trace"
"$ref": "#/definitions/RecordedTrace"
}
},
"definitions": {
"FillCommit": {
"type": "string",
"enum": [
"enter",
"suggestion",
"blur"
]
},
"KeyModifier": {
"description": "Keyboard modifier flags. Multiple flags may be combined; the extension folds them into CDP's bitfield (`alt=1, ctrl=2, meta=4, shift=8`).",
"type": "string",
@@ -21,7 +29,19 @@
"shift"
]
},
"PageRef": {
"NavigationCause": {
"type": "string",
"enum": [
"user_typed",
"link",
"form_submit",
"reload",
"history",
"script",
"browser"
]
},
"PageRefV2": {
"description": "Page context dictionary entry — referenced by steps via `page` id.",
"type": "object",
"required": [
@@ -43,7 +63,42 @@
}
}
},
"SelectedOption": {
"RecordedTrace": {
"if": {
"required": [
"version"
],
"properties": {
"version": {
"type": "integer"
}
}
},
"then": {
"$ref": "#/definitions/TraceV3"
},
"else": {
"$ref": "#/definitions/TraceV2"
}
},
"RecorderInfo": {
"type": "object",
"required": [
"bsk",
"vom"
],
"properties": {
"bsk": {
"type": "string"
},
"vom": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
}
}
},
"SelectedOptionV2": {
"description": "One selected option (`select` op).",
"type": "object",
"required": [
@@ -61,11 +116,53 @@
}
}
},
"Step": {
"description": "One recorded user action — discriminated union by `op`.",
"SelectedOptionV3": {
"description": "One selected option (`select` op).",
"type": "object",
"required": [
"value"
],
"properties": {
"label": {
"type": [
"string",
"null"
]
},
"value": {
"type": "string"
}
}
},
"StepEffectV2": {
"description": "Observed navigation after a step (objective fact only).",
"type": "object",
"required": [
"navigated_to"
],
"properties": {
"navigated_to": {
"description": "Reference into `pages[]` for the destination page.",
"type": "string"
}
}
},
"StepResultV3": {
"type": "object",
"required": [
"state"
],
"properties": {
"state": {
"type": "string"
}
}
},
"StepV2": {
"description": "One recorded user action — discriminated union by `op` (v2).",
"oneOf": [
{
"description": "Fields shared by every step variant (flattened in JSON).",
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
@@ -77,7 +174,7 @@
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
@@ -105,7 +202,7 @@
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
@@ -117,7 +214,7 @@
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
@@ -140,12 +237,12 @@
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
"$ref": "#/definitions/TargetDescriptorV2"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
@@ -157,7 +254,7 @@
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
@@ -180,12 +277,12 @@
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
"$ref": "#/definitions/TargetDescriptorV2"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
@@ -198,7 +295,7 @@
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
@@ -227,7 +324,7 @@
]
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
"$ref": "#/definitions/TargetDescriptorV2"
},
"value": {
"type": "string"
@@ -235,7 +332,7 @@
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
@@ -248,7 +345,7 @@
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
@@ -273,16 +370,16 @@
"selection": {
"type": "array",
"items": {
"$ref": "#/definitions/SelectedOption"
"$ref": "#/definitions/SelectedOptionV2"
}
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
"$ref": "#/definitions/TargetDescriptorV2"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
@@ -294,7 +391,7 @@
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
@@ -331,7 +428,7 @@
"target": {
"anyOf": [
{
"$ref": "#/definitions/TargetDescriptor"
"$ref": "#/definitions/TargetDescriptorV2"
},
{
"type": "null"
@@ -342,21 +439,294 @@
}
]
},
"StepEffect": {
"description": "Observed navigation after a step (objective fact only).",
"type": "object",
"required": [
"navigated_to"
],
"properties": {
"navigated_to": {
"description": "Reference into `pages[]` for the destination page.",
"type": "string"
"StepV3": {
"description": "One recorded user action — discriminated union by `op`.",
"oneOf": [
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"cause",
"id",
"op",
"result",
"state",
"to"
],
"properties": {
"cause": {
"$ref": "#/definitions/NavigationCause"
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"navigate"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"to": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state",
"target"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"click"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state",
"target"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"hover"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"commit",
"id",
"op",
"result",
"state",
"target",
"value"
],
"properties": {
"commit": {
"$ref": "#/definitions/FillCommit"
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"fill"
]
},
"redacted": {
"type": "boolean"
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
},
"value": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state",
"target"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"select"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"selection": {
"type": "array",
"items": {
"$ref": "#/definitions/SelectedOptionV3"
}
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"key",
"op",
"result",
"state"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"key": {
"type": "string"
},
"modifiers": {
"type": [
"array",
"null"
],
"items": {
"$ref": "#/definitions/KeyModifier"
}
},
"op": {
"type": "string",
"enum": [
"press"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"anyOf": [
{
"$ref": "#/definitions/TargetDescriptorV3"
},
{
"type": "null"
}
]
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"scroll"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
}
}
}
}
]
},
"TargetDescriptor": {
"description": "Stable semantic handle for an interacted element.\n\n`name` and `nearby_label` are **untrusted page text**.",
"StopReason": {
"type": "string",
"enum": [
"user_finish",
"cli_stop"
]
},
"TargetDescriptorV2": {
"description": "Stable semantic handle for an interacted element (v2).\n\n`name` and `nearby_label` are **untrusted page text**.",
"type": "object",
"required": [
"tag"
@@ -397,8 +767,83 @@
}
}
},
"Trace": {
"description": "Persisted user-action trace exported by `tool.record_stop` / `await`.",
"TargetDescriptorV3": {
"description": "Stable semantic handle for an interacted element within a page observation.",
"type": "object",
"properties": {
"ctx": {
"type": [
"string",
"null"
]
},
"name": {
"type": [
"string",
"null"
]
},
"ref": {
"type": [
"string",
"null"
]
},
"role": {
"type": [
"string",
"null"
]
},
"unmatched": {
"type": "boolean"
}
}
},
"TraceEntry": {
"type": "object",
"required": [
"start_url"
],
"properties": {
"start_url": {
"type": "string"
}
}
},
"TraceStateV3": {
"description": "Page observation dictionary entry — referenced by steps via `state` / `result.state`.",
"type": "object",
"required": [
"body",
"id",
"url"
],
"properties": {
"body": {
"description": "Full page observation (front matter + VOM body + annotations).",
"type": "string"
},
"id": {
"type": "string"
},
"title": {
"type": [
"string",
"null"
]
},
"truncated": {
"type": "boolean"
},
"url": {
"type": "string"
}
}
},
"TraceV2": {
"title": "TraceV2",
"description": "Persisted user-action trace exported by legacy `tool.record_stop` / `await`.\n\nUnknown extension fields are ignored so older traces remain readable. `states[]` and a numeric `version` are reserved for Trace v3 / `RecordedTrace` classification.",
"type": "object",
"required": [
"entry",
@@ -413,11 +858,10 @@
"pages": {
"type": "array",
"items": {
"$ref": "#/definitions/PageRef"
"$ref": "#/definitions/PageRefV2"
}
},
"purpose": {
"description": "Optional user-provided goal from `--purpose` (metadata only).",
"type": [
"string",
"null"
@@ -433,25 +877,76 @@
"null"
]
},
"states": false,
"steps": {
"type": "array",
"items": {
"$ref": "#/definitions/Step"
"$ref": "#/definitions/StepV2"
}
},
"version": {
"description": "Numeric version selects Trace v3. Legacy v2 envelopes omit this field.",
"not": {
"type": "integer"
}
}
}
},
"TraceEntry": {
"description": "Recording entry point — first URL the flow starts from.",
"TraceV3": {
"description": "Wire trace returned by `tool.record_stop` / `await`.",
"type": "object",
"required": [
"start_url"
"entry",
"recorded_at",
"recorder",
"states",
"steps",
"stopped_by",
"version"
],
"properties": {
"start_url": {
"entry": {
"$ref": "#/definitions/TraceEntry"
},
"purpose": {
"type": [
"string",
"null"
]
},
"recorded_at": {
"type": "string"
},
"recorder": {
"$ref": "#/definitions/RecorderInfo"
},
"started_at": {
"type": [
"string",
"null"
]
},
"states": {
"type": "array",
"items": {
"$ref": "#/definitions/TraceStateV3"
}
},
"steps": {
"type": "array",
"items": {
"$ref": "#/definitions/StepV3"
}
},
"stopped_by": {
"$ref": "#/definitions/StopReason"
},
"version": {
"type": "integer",
"const": 3
}
}
},
"additionalProperties": false
}
}
}
+571 -76
View File
@@ -1,49 +1,31 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Trace",
"description": "Persisted user-action trace exported by `tool.record_stop` / `await`.",
"type": "object",
"required": [
"entry",
"pages",
"recorded_at",
"steps"
],
"properties": {
"entry": {
"$ref": "#/definitions/TraceEntry"
},
"pages": {
"type": "array",
"items": {
"$ref": "#/definitions/PageRef"
}
},
"purpose": {
"description": "Optional user-provided goal from `--purpose` (metadata only).",
"type": [
"string",
"null"
]
},
"recorded_at": {
"description": "RFC 3339 timestamp when recording stopped.",
"type": "string"
},
"started_at": {
"type": [
"string",
"null"
]
},
"steps": {
"type": "array",
"items": {
"$ref": "#/definitions/Step"
"title": "RecordedTrace",
"if": {
"required": [
"version"
],
"properties": {
"version": {
"type": "integer"
}
}
},
"then": {
"$ref": "#/definitions/TraceV3"
},
"else": {
"$ref": "#/definitions/TraceV2"
},
"definitions": {
"FillCommit": {
"type": "string",
"enum": [
"enter",
"suggestion",
"blur"
]
},
"KeyModifier": {
"description": "Keyboard modifier flags. Multiple flags may be combined; the extension folds them into CDP's bitfield (`alt=1, ctrl=2, meta=4, shift=8`).",
"type": "string",
@@ -54,7 +36,19 @@
"shift"
]
},
"PageRef": {
"NavigationCause": {
"type": "string",
"enum": [
"user_typed",
"link",
"form_submit",
"reload",
"history",
"script",
"browser"
]
},
"PageRefV2": {
"description": "Page context dictionary entry — referenced by steps via `page` id.",
"type": "object",
"required": [
@@ -76,7 +70,24 @@
}
}
},
"SelectedOption": {
"RecorderInfo": {
"type": "object",
"required": [
"bsk",
"vom"
],
"properties": {
"bsk": {
"type": "string"
},
"vom": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
}
}
},
"SelectedOptionV2": {
"description": "One selected option (`select` op).",
"type": "object",
"required": [
@@ -94,11 +105,53 @@
}
}
},
"Step": {
"description": "One recorded user action — discriminated union by `op`.",
"SelectedOptionV3": {
"description": "One selected option (`select` op).",
"type": "object",
"required": [
"value"
],
"properties": {
"label": {
"type": [
"string",
"null"
]
},
"value": {
"type": "string"
}
}
},
"StepEffectV2": {
"description": "Observed navigation after a step (objective fact only).",
"type": "object",
"required": [
"navigated_to"
],
"properties": {
"navigated_to": {
"description": "Reference into `pages[]` for the destination page.",
"type": "string"
}
}
},
"StepResultV3": {
"type": "object",
"required": [
"state"
],
"properties": {
"state": {
"type": "string"
}
}
},
"StepV2": {
"description": "One recorded user action — discriminated union by `op` (v2).",
"oneOf": [
{
"description": "Fields shared by every step variant (flattened in JSON).",
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
@@ -110,7 +163,7 @@
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
@@ -138,7 +191,7 @@
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
@@ -150,7 +203,7 @@
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
@@ -173,12 +226,12 @@
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
"$ref": "#/definitions/TargetDescriptorV2"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
@@ -190,7 +243,7 @@
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
@@ -213,12 +266,12 @@
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
"$ref": "#/definitions/TargetDescriptorV2"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
@@ -231,7 +284,7 @@
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
@@ -260,7 +313,7 @@
]
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
"$ref": "#/definitions/TargetDescriptorV2"
},
"value": {
"type": "string"
@@ -268,7 +321,7 @@
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
@@ -281,7 +334,7 @@
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
@@ -306,16 +359,16 @@
"selection": {
"type": "array",
"items": {
"$ref": "#/definitions/SelectedOption"
"$ref": "#/definitions/SelectedOptionV2"
}
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
"$ref": "#/definitions/TargetDescriptorV2"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
@@ -327,7 +380,7 @@
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
@@ -364,7 +417,7 @@
"target": {
"anyOf": [
{
"$ref": "#/definitions/TargetDescriptor"
"$ref": "#/definitions/TargetDescriptorV2"
},
{
"type": "null"
@@ -375,21 +428,294 @@
}
]
},
"StepEffect": {
"description": "Observed navigation after a step (objective fact only).",
"type": "object",
"required": [
"navigated_to"
],
"properties": {
"navigated_to": {
"description": "Reference into `pages[]` for the destination page.",
"type": "string"
"StepV3": {
"description": "One recorded user action — discriminated union by `op`.",
"oneOf": [
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"cause",
"id",
"op",
"result",
"state",
"to"
],
"properties": {
"cause": {
"$ref": "#/definitions/NavigationCause"
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"navigate"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"to": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state",
"target"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"click"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state",
"target"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"hover"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"commit",
"id",
"op",
"result",
"state",
"target",
"value"
],
"properties": {
"commit": {
"$ref": "#/definitions/FillCommit"
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"fill"
]
},
"redacted": {
"type": "boolean"
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
},
"value": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state",
"target"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"select"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"selection": {
"type": "array",
"items": {
"$ref": "#/definitions/SelectedOptionV3"
}
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"key",
"op",
"result",
"state"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"key": {
"type": "string"
},
"modifiers": {
"type": [
"array",
"null"
],
"items": {
"$ref": "#/definitions/KeyModifier"
}
},
"op": {
"type": "string",
"enum": [
"press"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"anyOf": [
{
"$ref": "#/definitions/TargetDescriptorV3"
},
{
"type": "null"
}
]
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"scroll"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
}
}
}
}
]
},
"TargetDescriptor": {
"description": "Stable semantic handle for an interacted element.\n\n`name` and `nearby_label` are **untrusted page text**.",
"StopReason": {
"type": "string",
"enum": [
"user_finish",
"cli_stop"
]
},
"TargetDescriptorV2": {
"description": "Stable semantic handle for an interacted element (v2).\n\n`name` and `nearby_label` are **untrusted page text**.",
"type": "object",
"required": [
"tag"
@@ -430,8 +756,40 @@
}
}
},
"TargetDescriptorV3": {
"description": "Stable semantic handle for an interacted element within a page observation.",
"type": "object",
"properties": {
"ctx": {
"type": [
"string",
"null"
]
},
"name": {
"type": [
"string",
"null"
]
},
"ref": {
"type": [
"string",
"null"
]
},
"role": {
"type": [
"string",
"null"
]
},
"unmatched": {
"type": "boolean"
}
}
},
"TraceEntry": {
"description": "Recording entry point — first URL the flow starts from.",
"type": "object",
"required": [
"start_url"
@@ -441,6 +799,143 @@
"type": "string"
}
}
},
"TraceStateV3": {
"description": "Page observation dictionary entry — referenced by steps via `state` / `result.state`.",
"type": "object",
"required": [
"body",
"id",
"url"
],
"properties": {
"body": {
"description": "Full page observation (front matter + VOM body + annotations).",
"type": "string"
},
"id": {
"type": "string"
},
"title": {
"type": [
"string",
"null"
]
},
"truncated": {
"type": "boolean"
},
"url": {
"type": "string"
}
}
},
"TraceV2": {
"title": "TraceV2",
"description": "Persisted user-action trace exported by legacy `tool.record_stop` / `await`.\n\nUnknown extension fields are ignored so older traces remain readable. `states[]` and a numeric `version` are reserved for Trace v3 / `RecordedTrace` classification.",
"type": "object",
"required": [
"entry",
"pages",
"recorded_at",
"steps"
],
"properties": {
"entry": {
"$ref": "#/definitions/TraceEntry"
},
"pages": {
"type": "array",
"items": {
"$ref": "#/definitions/PageRefV2"
}
},
"purpose": {
"type": [
"string",
"null"
]
},
"recorded_at": {
"description": "RFC 3339 timestamp when recording stopped.",
"type": "string"
},
"started_at": {
"type": [
"string",
"null"
]
},
"states": false,
"steps": {
"type": "array",
"items": {
"$ref": "#/definitions/StepV2"
}
},
"version": {
"description": "Numeric version selects Trace v3. Legacy v2 envelopes omit this field.",
"not": {
"type": "integer"
}
}
}
},
"TraceV3": {
"description": "Wire trace returned by `tool.record_stop` / `await`.",
"type": "object",
"required": [
"entry",
"recorded_at",
"recorder",
"states",
"steps",
"stopped_by",
"version"
],
"properties": {
"entry": {
"$ref": "#/definitions/TraceEntry"
},
"purpose": {
"type": [
"string",
"null"
]
},
"recorded_at": {
"type": "string"
},
"recorder": {
"$ref": "#/definitions/RecorderInfo"
},
"started_at": {
"type": [
"string",
"null"
]
},
"states": {
"type": "array",
"items": {
"$ref": "#/definitions/TraceStateV3"
}
},
"steps": {
"type": "array",
"items": {
"$ref": "#/definitions/StepV3"
}
},
"stopped_by": {
"$ref": "#/definitions/StopReason"
},
"version": {
"type": "integer",
"const": 3
}
},
"additionalProperties": false
}
}
}
+664 -284
View File
@@ -1,286 +1,32 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Step",
"description": "One recorded user action — discriminated union by `op`.",
"oneOf": [
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"to"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"navigate"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"to": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"click"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"hover"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target",
"value"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"fill"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"redacted": {
"type": [
"boolean",
"null"
]
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
},
"value": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"selection",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"select"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"selection": {
"type": "array",
"items": {
"$ref": "#/definitions/SelectedOption"
}
},
"target": {
"$ref": "#/definitions/TargetDescriptor"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"key",
"op",
"page"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffect"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"key": {
"type": "string"
},
"modifiers": {
"type": [
"array",
"null"
],
"items": {
"$ref": "#/definitions/KeyModifier"
}
},
"op": {
"type": "string",
"enum": [
"press"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"anyOf": [
{
"$ref": "#/definitions/TargetDescriptor"
},
{
"type": "null"
}
]
}
}
}
],
"title": "RecordedStep",
"not": {
"required": [
"page",
"state"
]
},
"if": {
"required": [
"state"
]
},
"then": {
"$ref": "#/definitions/StepV3"
},
"else": {
"$ref": "#/definitions/StepV2"
},
"definitions": {
"FillCommit": {
"type": "string",
"enum": [
"enter",
"suggestion",
"blur"
]
},
"KeyModifier": {
"description": "Keyboard modifier flags. Multiple flags may be combined; the extension folds them into CDP's bitfield (`alt=1, ctrl=2, meta=4, shift=8`).",
"type": "string",
@@ -291,7 +37,19 @@
"shift"
]
},
"SelectedOption": {
"NavigationCause": {
"type": "string",
"enum": [
"user_typed",
"link",
"form_submit",
"reload",
"history",
"script",
"browser"
]
},
"SelectedOptionV2": {
"description": "One selected option (`select` op).",
"type": "object",
"required": [
@@ -309,7 +67,25 @@
}
}
},
"StepEffect": {
"SelectedOptionV3": {
"description": "One selected option (`select` op).",
"type": "object",
"required": [
"value"
],
"properties": {
"label": {
"type": [
"string",
"null"
]
},
"value": {
"type": "string"
}
}
},
"StepEffectV2": {
"description": "Observed navigation after a step (objective fact only).",
"type": "object",
"required": [
@@ -322,8 +98,579 @@
}
}
},
"TargetDescriptor": {
"description": "Stable semantic handle for an interacted element.\n\n`name` and `nearby_label` are **untrusted page text**.",
"StepResultV3": {
"type": "object",
"required": [
"state"
],
"properties": {
"state": {
"type": "string"
}
}
},
"StepV2": {
"description": "One recorded user action — discriminated union by `op` (v2).",
"oneOf": [
{
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"to"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"navigate"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"to": {
"type": "string"
}
}
},
{
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"click"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV2"
}
}
},
{
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"hover"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV2"
}
}
},
{
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target",
"value"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"fill"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"redacted": {
"type": [
"boolean",
"null"
]
},
"target": {
"$ref": "#/definitions/TargetDescriptorV2"
},
"value": {
"type": "string"
}
}
},
{
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"selection",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"select"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"selection": {
"type": "array",
"items": {
"$ref": "#/definitions/SelectedOptionV2"
}
},
"target": {
"$ref": "#/definitions/TargetDescriptorV2"
}
}
},
{
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"key",
"op",
"page"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"key": {
"type": "string"
},
"modifiers": {
"type": [
"array",
"null"
],
"items": {
"$ref": "#/definitions/KeyModifier"
}
},
"op": {
"type": "string",
"enum": [
"press"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"anyOf": [
{
"$ref": "#/definitions/TargetDescriptorV2"
},
{
"type": "null"
}
]
}
}
}
]
},
"StepV3": {
"description": "One recorded user action — discriminated union by `op`.",
"oneOf": [
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"cause",
"id",
"op",
"result",
"state",
"to"
],
"properties": {
"cause": {
"$ref": "#/definitions/NavigationCause"
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"navigate"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"to": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state",
"target"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"click"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state",
"target"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"hover"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"commit",
"id",
"op",
"result",
"state",
"target",
"value"
],
"properties": {
"commit": {
"$ref": "#/definitions/FillCommit"
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"fill"
]
},
"redacted": {
"type": "boolean"
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
},
"value": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state",
"target"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"select"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"selection": {
"type": "array",
"items": {
"$ref": "#/definitions/SelectedOptionV3"
}
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"key",
"op",
"result",
"state"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"key": {
"type": "string"
},
"modifiers": {
"type": [
"array",
"null"
],
"items": {
"$ref": "#/definitions/KeyModifier"
}
},
"op": {
"type": "string",
"enum": [
"press"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"anyOf": [
{
"$ref": "#/definitions/TargetDescriptorV3"
},
{
"type": "null"
}
]
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"scroll"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
}
}
}
]
},
"TargetDescriptorV2": {
"description": "Stable semantic handle for an interacted element (v2).\n\n`name` and `nearby_label` are **untrusted page text**.",
"type": "object",
"required": [
"tag"
@@ -363,6 +710,39 @@
"type": "string"
}
}
},
"TargetDescriptorV3": {
"description": "Stable semantic handle for an interacted element within a page observation.",
"type": "object",
"properties": {
"ctx": {
"type": [
"string",
"null"
]
},
"name": {
"type": [
"string",
"null"
]
},
"ref": {
"type": [
"string",
"null"
]
},
"role": {
"type": [
"string",
"null"
]
},
"unmatched": {
"type": "boolean"
}
}
}
}
}
@@ -0,0 +1,368 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "StepV2",
"description": "One recorded user action — discriminated union by `op` (v2).",
"oneOf": [
{
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"to"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"navigate"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"to": {
"type": "string"
}
}
},
{
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"click"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV2"
}
}
},
{
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"hover"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV2"
}
}
},
{
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target",
"value"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"fill"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"redacted": {
"type": [
"boolean",
"null"
]
},
"target": {
"$ref": "#/definitions/TargetDescriptorV2"
},
"value": {
"type": "string"
}
}
},
{
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"selection",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"select"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"selection": {
"type": "array",
"items": {
"$ref": "#/definitions/SelectedOptionV2"
}
},
"target": {
"$ref": "#/definitions/TargetDescriptorV2"
}
}
},
{
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"key",
"op",
"page"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"key": {
"type": "string"
},
"modifiers": {
"type": [
"array",
"null"
],
"items": {
"$ref": "#/definitions/KeyModifier"
}
},
"op": {
"type": "string",
"enum": [
"press"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"anyOf": [
{
"$ref": "#/definitions/TargetDescriptorV2"
},
{
"type": "null"
}
]
}
}
}
],
"definitions": {
"KeyModifier": {
"description": "Keyboard modifier flags. Multiple flags may be combined; the extension folds them into CDP's bitfield (`alt=1, ctrl=2, meta=4, shift=8`).",
"type": "string",
"enum": [
"alt",
"ctrl",
"meta",
"shift"
]
},
"SelectedOptionV2": {
"description": "One selected option (`select` op).",
"type": "object",
"required": [
"value"
],
"properties": {
"label": {
"type": [
"string",
"null"
]
},
"value": {
"type": "string"
}
}
},
"StepEffectV2": {
"description": "Observed navigation after a step (objective fact only).",
"type": "object",
"required": [
"navigated_to"
],
"properties": {
"navigated_to": {
"description": "Reference into `pages[]` for the destination page.",
"type": "string"
}
}
},
"TargetDescriptorV2": {
"description": "Stable semantic handle for an interacted element (v2).\n\n`name` and `nearby_label` are **untrusted page text**.",
"type": "object",
"required": [
"tag"
],
"properties": {
"name": {
"type": [
"string",
"null"
]
},
"name_attr": {
"type": [
"string",
"null"
]
},
"nearby_label": {
"type": [
"string",
"null"
]
},
"placeholder": {
"type": [
"string",
"null"
]
},
"role": {
"type": [
"string",
"null"
]
},
"tag": {
"type": "string"
}
}
}
}
}
@@ -0,0 +1,375 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "StepV3",
"description": "One recorded user action — discriminated union by `op`.",
"oneOf": [
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"cause",
"id",
"op",
"result",
"state",
"to"
],
"properties": {
"cause": {
"$ref": "#/definitions/NavigationCause"
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"navigate"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"to": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state",
"target"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"click"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state",
"target"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"hover"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"commit",
"id",
"op",
"result",
"state",
"target",
"value"
],
"properties": {
"commit": {
"$ref": "#/definitions/FillCommit"
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"fill"
]
},
"redacted": {
"type": "boolean"
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
},
"value": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state",
"target"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"select"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"selection": {
"type": "array",
"items": {
"$ref": "#/definitions/SelectedOptionV3"
}
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"key",
"op",
"result",
"state"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"key": {
"type": "string"
},
"modifiers": {
"type": [
"array",
"null"
],
"items": {
"$ref": "#/definitions/KeyModifier"
}
},
"op": {
"type": "string",
"enum": [
"press"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"anyOf": [
{
"$ref": "#/definitions/TargetDescriptorV3"
},
{
"type": "null"
}
]
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"scroll"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
}
}
}
],
"definitions": {
"FillCommit": {
"type": "string",
"enum": [
"enter",
"suggestion",
"blur"
]
},
"KeyModifier": {
"description": "Keyboard modifier flags. Multiple flags may be combined; the extension folds them into CDP's bitfield (`alt=1, ctrl=2, meta=4, shift=8`).",
"type": "string",
"enum": [
"alt",
"ctrl",
"meta",
"shift"
]
},
"NavigationCause": {
"type": "string",
"enum": [
"user_typed",
"link",
"form_submit",
"reload",
"history",
"script",
"browser"
]
},
"SelectedOptionV3": {
"description": "One selected option (`select` op).",
"type": "object",
"required": [
"value"
],
"properties": {
"label": {
"type": [
"string",
"null"
]
},
"value": {
"type": "string"
}
}
},
"StepResultV3": {
"type": "object",
"required": [
"state"
],
"properties": {
"state": {
"type": "string"
}
}
},
"TargetDescriptorV3": {
"description": "Stable semantic handle for an interacted element within a page observation.",
"type": "object",
"properties": {
"ctx": {
"type": [
"string",
"null"
]
},
"name": {
"type": [
"string",
"null"
]
},
"ref": {
"type": [
"string",
"null"
]
},
"role": {
"type": [
"string",
"null"
]
},
"unmatched": {
"type": "boolean"
}
}
}
}
}
+451
View File
@@ -0,0 +1,451 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "TraceV2",
"description": "Persisted user-action trace exported by legacy `tool.record_stop` / `await`.\n\nUnknown extension fields are ignored so older traces remain readable. `states[]` and a numeric `version` are reserved for Trace v3 / `RecordedTrace` classification.",
"type": "object",
"required": [
"entry",
"pages",
"recorded_at",
"steps"
],
"properties": {
"entry": {
"$ref": "#/definitions/TraceEntry"
},
"pages": {
"type": "array",
"items": {
"$ref": "#/definitions/PageRefV2"
}
},
"purpose": {
"type": [
"string",
"null"
]
},
"recorded_at": {
"description": "RFC 3339 timestamp when recording stopped.",
"type": "string"
},
"started_at": {
"type": [
"string",
"null"
]
},
"states": false,
"steps": {
"type": "array",
"items": {
"$ref": "#/definitions/StepV2"
}
},
"version": {
"description": "Numeric version selects Trace v3. Legacy v2 envelopes omit this field.",
"not": {
"type": "integer"
}
}
},
"definitions": {
"KeyModifier": {
"description": "Keyboard modifier flags. Multiple flags may be combined; the extension folds them into CDP's bitfield (`alt=1, ctrl=2, meta=4, shift=8`).",
"type": "string",
"enum": [
"alt",
"ctrl",
"meta",
"shift"
]
},
"PageRefV2": {
"description": "Page context dictionary entry — referenced by steps via `page` id.",
"type": "object",
"required": [
"id",
"url"
],
"properties": {
"id": {
"type": "string"
},
"title": {
"type": [
"string",
"null"
]
},
"url": {
"type": "string"
}
}
},
"SelectedOptionV2": {
"description": "One selected option (`select` op).",
"type": "object",
"required": [
"value"
],
"properties": {
"label": {
"type": [
"string",
"null"
]
},
"value": {
"type": "string"
}
}
},
"StepEffectV2": {
"description": "Observed navigation after a step (objective fact only).",
"type": "object",
"required": [
"navigated_to"
],
"properties": {
"navigated_to": {
"description": "Reference into `pages[]` for the destination page.",
"type": "string"
}
}
},
"StepV2": {
"description": "One recorded user action — discriminated union by `op` (v2).",
"oneOf": [
{
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"to"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"navigate"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"to": {
"type": "string"
}
}
},
{
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"click"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV2"
}
}
},
{
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"hover"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV2"
}
}
},
{
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"target",
"value"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"fill"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"redacted": {
"type": [
"boolean",
"null"
]
},
"target": {
"$ref": "#/definitions/TargetDescriptorV2"
},
"value": {
"type": "string"
}
}
},
{
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"page",
"selection",
"target"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"select"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"selection": {
"type": "array",
"items": {
"$ref": "#/definitions/SelectedOptionV2"
}
},
"target": {
"$ref": "#/definitions/TargetDescriptorV2"
}
}
},
{
"description": "Fields shared by every v2 step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"key",
"op",
"page"
],
"properties": {
"effect": {
"anyOf": [
{
"$ref": "#/definitions/StepEffectV2"
},
{
"type": "null"
}
]
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"key": {
"type": "string"
},
"modifiers": {
"type": [
"array",
"null"
],
"items": {
"$ref": "#/definitions/KeyModifier"
}
},
"op": {
"type": "string",
"enum": [
"press"
]
},
"page": {
"description": "Reference into `pages[]`.",
"type": "string"
},
"target": {
"anyOf": [
{
"$ref": "#/definitions/TargetDescriptorV2"
},
{
"type": "null"
}
]
}
}
}
]
},
"TargetDescriptorV2": {
"description": "Stable semantic handle for an interacted element (v2).\n\n`name` and `nearby_label` are **untrusted page text**.",
"type": "object",
"required": [
"tag"
],
"properties": {
"name": {
"type": [
"string",
"null"
]
},
"name_attr": {
"type": [
"string",
"null"
]
},
"nearby_label": {
"type": [
"string",
"null"
]
},
"placeholder": {
"type": [
"string",
"null"
]
},
"role": {
"type": [
"string",
"null"
]
},
"tag": {
"type": "string"
}
}
},
"TraceEntry": {
"type": "object",
"required": [
"start_url"
],
"properties": {
"start_url": {
"type": "string"
}
}
}
}
}
+496
View File
@@ -0,0 +1,496 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "TraceV3",
"description": "Wire trace returned by `tool.record_stop` / `await`.",
"type": "object",
"required": [
"entry",
"recorded_at",
"recorder",
"states",
"steps",
"stopped_by",
"version"
],
"properties": {
"entry": {
"$ref": "#/definitions/TraceEntry"
},
"purpose": {
"type": [
"string",
"null"
]
},
"recorded_at": {
"type": "string"
},
"recorder": {
"$ref": "#/definitions/RecorderInfo"
},
"started_at": {
"type": [
"string",
"null"
]
},
"states": {
"type": "array",
"items": {
"$ref": "#/definitions/TraceStateV3"
}
},
"steps": {
"type": "array",
"items": {
"$ref": "#/definitions/StepV3"
}
},
"stopped_by": {
"$ref": "#/definitions/StopReason"
},
"version": {
"type": "integer",
"const": 3
}
},
"additionalProperties": false,
"definitions": {
"FillCommit": {
"type": "string",
"enum": [
"enter",
"suggestion",
"blur"
]
},
"KeyModifier": {
"description": "Keyboard modifier flags. Multiple flags may be combined; the extension folds them into CDP's bitfield (`alt=1, ctrl=2, meta=4, shift=8`).",
"type": "string",
"enum": [
"alt",
"ctrl",
"meta",
"shift"
]
},
"NavigationCause": {
"type": "string",
"enum": [
"user_typed",
"link",
"form_submit",
"reload",
"history",
"script",
"browser"
]
},
"RecorderInfo": {
"type": "object",
"required": [
"bsk",
"vom"
],
"properties": {
"bsk": {
"type": "string"
},
"vom": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
}
}
},
"SelectedOptionV3": {
"description": "One selected option (`select` op).",
"type": "object",
"required": [
"value"
],
"properties": {
"label": {
"type": [
"string",
"null"
]
},
"value": {
"type": "string"
}
}
},
"StepResultV3": {
"type": "object",
"required": [
"state"
],
"properties": {
"state": {
"type": "string"
}
}
},
"StepV3": {
"description": "One recorded user action — discriminated union by `op`.",
"oneOf": [
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"cause",
"id",
"op",
"result",
"state",
"to"
],
"properties": {
"cause": {
"$ref": "#/definitions/NavigationCause"
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"navigate"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"to": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state",
"target"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"click"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state",
"target"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"hover"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"commit",
"id",
"op",
"result",
"state",
"target",
"value"
],
"properties": {
"commit": {
"$ref": "#/definitions/FillCommit"
},
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"fill"
]
},
"redacted": {
"type": "boolean"
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
},
"value": {
"type": "string"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state",
"target"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"select"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"selection": {
"type": "array",
"items": {
"$ref": "#/definitions/SelectedOptionV3"
}
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"$ref": "#/definitions/TargetDescriptorV3"
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"key",
"op",
"result",
"state"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"key": {
"type": "string"
},
"modifiers": {
"type": [
"array",
"null"
],
"items": {
"$ref": "#/definitions/KeyModifier"
}
},
"op": {
"type": "string",
"enum": [
"press"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
},
"target": {
"anyOf": [
{
"$ref": "#/definitions/TargetDescriptorV3"
},
{
"type": "null"
}
]
}
}
},
{
"description": "Fields shared by every step variant (flattened in JSON).",
"type": "object",
"required": [
"id",
"op",
"result",
"state"
],
"properties": {
"id": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"op": {
"type": "string",
"enum": [
"scroll"
]
},
"result": {
"$ref": "#/definitions/StepResultV3"
},
"state": {
"description": "Observation id immediately before this action.",
"type": "string"
}
}
}
]
},
"StopReason": {
"type": "string",
"enum": [
"user_finish",
"cli_stop"
]
},
"TargetDescriptorV3": {
"description": "Stable semantic handle for an interacted element within a page observation.",
"type": "object",
"properties": {
"ctx": {
"type": [
"string",
"null"
]
},
"name": {
"type": [
"string",
"null"
]
},
"ref": {
"type": [
"string",
"null"
]
},
"role": {
"type": [
"string",
"null"
]
},
"unmatched": {
"type": "boolean"
}
}
},
"TraceEntry": {
"type": "object",
"required": [
"start_url"
],
"properties": {
"start_url": {
"type": "string"
}
}
},
"TraceStateV3": {
"description": "Page observation dictionary entry — referenced by steps via `state` / `result.state`.",
"type": "object",
"required": [
"body",
"id",
"url"
],
"properties": {
"body": {
"description": "Full page observation (front matter + VOM body + annotations).",
"type": "string"
},
"id": {
"type": "string"
},
"title": {
"type": [
"string",
"null"
]
},
"truncated": {
"type": "boolean"
},
"url": {
"type": "string"
}
}
}
}
}
+6 -2
View File
@@ -113,8 +113,12 @@ fn main() {
dump!(RequestHelpParams, "tool_request_help_params");
dump!(RequestHelpResult, "tool_request_help_result");
dump!(Trace, "trace");
dump!(Step, "trace_step");
dump!(TraceV2, "trace_v2");
dump!(TraceV3, "trace_v3");
dump!(RecordedTrace, "trace");
dump!(StepV2, "trace_step_v2");
dump!(StepV3, "trace_step_v3");
dump!(RecordedStep, "trace_step");
dump!(RecordStartParams, "tool_record_start_params");
dump!(RecordStartResult, "tool_record_start_result");
dump!(RecordStopParams, "tool_record_stop_params");
+3
View File
@@ -9,6 +9,9 @@ pub mod navigation;
pub mod network;
pub mod observation;
pub mod record;
mod record_common;
mod record_v2;
mod record_v3;
pub mod script;
pub mod session;
pub mod tabs;
+380 -388
View File
@@ -1,147 +1,23 @@
//! Semantic user-action recording (`tool.record_start` / `stop` / `await`).
//!
//! Trace v2 is a **record-only** log of user actions: what was clicked, filled,
//! selected, and where navigation occurred. No LLM runs during recording, so
//! variable-vs-constant classification is **not** stored — executing agents
//! infer that at run time from raw values and control names.
//! Wire traces are either Trace v2 (`pages[]`) or Trace v3 (`version: 3`,
//! `states[]`). Version-specific models live in `record_v2` / `record_v3`.
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
use super::interaction::KeyModifier;
pub use super::record_common::TraceEntry;
pub use super::record_v2::{
PageRefV2, SelectedOptionV2, StepCommonV2, StepEffectV2, StepV2, TargetDescriptorV2, TraceV2,
};
pub use super::record_v3::{
FillCommit, NavigationCause, RecorderInfo, SelectedOptionV3, StepCommonV3, StepResultV3,
StepV3, StopReason, TRACE_VERSION_V3, TargetDescriptorV3, TraceStateV3, TraceV3,
VOM_FORMAT_VERSION,
};
// ---------------------------------------------------------------------------
// Target
// ---------------------------------------------------------------------------
/// Stable semantic handle for an interacted element.
///
/// `name` and `nearby_label` are **untrusted page text**.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct TargetDescriptor {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub role: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub name: Option<String>,
pub tag: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub name_attr: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub placeholder: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub nearby_label: Option<String>,
}
// ---------------------------------------------------------------------------
// Trace envelope
// ---------------------------------------------------------------------------
/// Recording entry point — first URL the flow starts from.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct TraceEntry {
pub start_url: String,
}
/// Page context dictionary entry — referenced by steps via `page` id.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct PageRef {
pub id: String,
pub url: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub title: Option<String>,
}
/// One selected option (`select` op).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct SelectedOption {
pub value: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub label: Option<String>,
}
/// Observed navigation after a step (objective fact only).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct StepEffect {
/// Reference into `pages[]` for the destination page.
pub navigated_to: String,
}
/// Fields shared by every step variant (flattened in JSON).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct StepCommon {
pub id: u32,
/// Reference into `pages[]`.
pub page: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub effect: Option<StepEffect>,
}
// ---------------------------------------------------------------------------
// Step op-specific payloads
// ---------------------------------------------------------------------------
/// One recorded user action — discriminated union by `op`.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
#[serde(tag = "op", rename_all = "snake_case")]
pub enum Step {
Navigate {
#[serde(flatten)]
common: StepCommon,
to: String,
},
Click {
#[serde(flatten)]
common: StepCommon,
target: TargetDescriptor,
},
Hover {
#[serde(flatten)]
common: StepCommon,
target: TargetDescriptor,
},
Fill {
#[serde(flatten)]
common: StepCommon,
target: TargetDescriptor,
value: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
redacted: Option<bool>,
},
Select {
#[serde(flatten)]
common: StepCommon,
target: TargetDescriptor,
selection: Vec<SelectedOption>,
},
Press {
#[serde(flatten)]
common: StepCommon,
key: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
modifiers: Option<Vec<KeyModifier>>,
#[serde(default, skip_serializing_if = "Option::is_none")]
target: Option<TargetDescriptor>,
},
}
// ---------------------------------------------------------------------------
// Trace root
// ---------------------------------------------------------------------------
/// Persisted user-action trace exported by `tool.record_stop` / `await`.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct Trace {
/// RFC 3339 timestamp when recording stopped.
pub recorded_at: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub started_at: Option<String>,
/// Optional user-provided goal from `--purpose` (metadata only).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub purpose: Option<String>,
pub entry: TraceEntry,
pub pages: Vec<PageRef>,
pub steps: Vec<Step>,
}
/// Logical v2 identifier. Not a wire field — v2 envelopes omit `version`.
pub const TRACE_VERSION_V2: u32 = 2;
// ---------------------------------------------------------------------------
// RPC params / results
@@ -169,9 +45,209 @@ pub struct RecordStopParams {
pub session_id: String,
}
/// Wire trace payload — v2 (legacy `pages[]`) or v3 (`version: 3`, `states[]`).
///
/// Classification matches [`RecordedTrace::classify_value`]: a numeric `version`
/// selects v3 (and forbids `pages[]`); otherwise the envelope is v2 and must
/// not include `states[]`.
#[derive(Debug, Clone, PartialEq)]
pub enum RecordedTrace {
V2(TraceV2),
V3(TraceV3),
}
impl RecordedTrace {
pub fn classify_value(v: &serde_json::Value) -> Result<Self, String> {
if let Some(ver) = v.get("version").and_then(|x| x.as_u64()) {
if ver != u64::from(TRACE_VERSION_V3) {
return Err(format!("unsupported trace version {ver}"));
}
if v.get("pages").is_some() {
return Err("trace v3 must not include legacy pages[]".into());
}
return serde_json::from_value(v.clone())
.map(RecordedTrace::V3)
.map_err(|e| e.to_string());
}
if v.get("states").is_some() {
return Err("trace v2 must not include states[]; set version: 3 for Trace v3".into());
}
if v.get("pages").is_some() {
return serde_json::from_value(v.clone())
.map(RecordedTrace::V2)
.map_err(|e| e.to_string());
}
Err("ambiguous or unparseable trace".into())
}
}
impl Serialize for RecordedTrace {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
match self {
RecordedTrace::V2(t) => t.serialize(serializer),
RecordedTrace::V3(t) => t.serialize(serializer),
}
}
}
impl<'de> Deserialize<'de> for RecordedTrace {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: serde::Deserializer<'de>,
{
let value = serde_json::Value::deserialize(deserializer)?;
Self::classify_value(&value).map_err(serde::de::Error::custom)
}
}
impl JsonSchema for RecordedTrace {
fn schema_name() -> String {
"RecordedTrace".into()
}
fn json_schema(generator: &mut schemars::SchemaGenerator) -> schemars::schema::Schema {
// Mirror classify_value: a numeric `version` selects v3; otherwise v2.
// TraceV2 stays open for unknown extension fields but forbids `states[]`
// and a numeric `version`. TraceV3 already denies `pages[]`.
let mut version_props = schemars::Map::new();
version_props.insert(
"version".into(),
schemars::schema::SchemaObject {
instance_type: Some(schemars::schema::InstanceType::Integer.into()),
..Default::default()
}
.into(),
);
let mut required = schemars::Set::new();
required.insert("version".into());
schemars::schema::SchemaObject {
subschemas: Some(Box::new(schemars::schema::SubschemaValidation {
if_schema: Some(Box::new(
schemars::schema::SchemaObject {
object: Some(Box::new(schemars::schema::ObjectValidation {
properties: version_props,
required,
..Default::default()
})),
..Default::default()
}
.into(),
)),
then_schema: Some(Box::new(generator.subschema_for::<TraceV3>())),
else_schema: Some(Box::new(generator.subschema_for::<TraceV2>())),
..Default::default()
})),
..Default::default()
}
.into()
}
}
/// One recorded step — v2 (`page`) or v3 (`state` / `result.state`).
///
/// Classification matches [`RecordedStep::classify_value`]: `state` selects v3
/// (and forbids `page`); otherwise the step is v2 and must include `page`.
#[derive(Debug, Clone, PartialEq)]
pub enum RecordedStep {
V2(StepV2),
V3(StepV3),
}
impl RecordedStep {
pub fn classify_value(v: &serde_json::Value) -> Result<Self, String> {
if v.get("state").is_some() {
if v.get("page").is_some() {
return Err("step v3 must not include legacy page".into());
}
return serde_json::from_value(v.clone())
.map(RecordedStep::V3)
.map_err(|e| e.to_string());
}
if v.get("page").is_some() {
return serde_json::from_value(v.clone())
.map(RecordedStep::V2)
.map_err(|e| e.to_string());
}
Err("ambiguous or unparseable step".into())
}
}
impl Serialize for RecordedStep {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
match self {
RecordedStep::V2(s) => s.serialize(serializer),
RecordedStep::V3(s) => s.serialize(serializer),
}
}
}
impl<'de> Deserialize<'de> for RecordedStep {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: serde::Deserializer<'de>,
{
let value = serde_json::Value::deserialize(deserializer)?;
Self::classify_value(&value).map_err(serde::de::Error::custom)
}
}
impl JsonSchema for RecordedStep {
fn schema_name() -> String {
"RecordedStep".into()
}
fn json_schema(generator: &mut schemars::SchemaGenerator) -> schemars::schema::Schema {
// Mirror classify_value: `state` selects v3; otherwise v2.
// Mixed page + state is rejected at this layer because StepV2/StepV3
// still allow unknown extension fields.
let mut required = schemars::Set::new();
required.insert("state".into());
let mut mixed_keys = schemars::Set::new();
mixed_keys.insert("page".into());
mixed_keys.insert("state".into());
schemars::schema::SchemaObject {
subschemas: Some(Box::new(schemars::schema::SubschemaValidation {
if_schema: Some(Box::new(
schemars::schema::SchemaObject {
object: Some(Box::new(schemars::schema::ObjectValidation {
required,
..Default::default()
})),
..Default::default()
}
.into(),
)),
then_schema: Some(Box::new(generator.subschema_for::<StepV3>())),
else_schema: Some(Box::new(generator.subschema_for::<StepV2>())),
not: Some(Box::new(
schemars::schema::SchemaObject {
object: Some(Box::new(schemars::schema::ObjectValidation {
required: mixed_keys,
..Default::default()
})),
..Default::default()
}
.into(),
)),
..Default::default()
})),
..Default::default()
}
.into()
}
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct RecordStopResult {
pub trace: Trace,
pub trace: RecordedTrace,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
@@ -183,281 +259,197 @@ pub struct RecordAwaitParams {
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct RecordAwaitResult {
pub trace: Trace,
pub trace: RecordedTrace,
}
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
fn sample_common(id: u32) -> StepCommon {
StepCommon {
id,
page: "p1".into(),
effect: None,
#[test]
fn recorded_trace_classifies_v2_and_v3() {
let v2 = json!({
"recorded_at": "2026-07-21T08:00:00Z",
"entry": { "start_url": "https://example.com/" },
"pages": [{ "id": "p1", "url": "https://example.com/" }],
"steps": []
});
match RecordedTrace::classify_value(&v2).unwrap() {
RecordedTrace::V2(_) => {}
other => panic!("expected v2, got {other:?}"),
}
}
fn sample_target() -> TargetDescriptor {
TargetDescriptor {
role: Some("button".into()),
name: Some("发布".into()),
tag: "button".into(),
name_attr: None,
placeholder: None,
nearby_label: None,
}
}
fn sample_trace() -> Trace {
Trace {
recorded_at: "2026-07-17T09:01:10Z".into(),
started_at: Some("2026-07-17T09:00:00Z".into()),
purpose: Some("发布一篇文章".into()),
entry: TraceEntry {
start_url: "https://x.com/editor".into(),
},
pages: vec![
PageRef {
id: "p1".into(),
url: "https://x.com/editor".into(),
title: Some("写文章".into()),
},
PageRef {
id: "p2".into(),
url: "https://x.com/p/99".into(),
title: None,
},
],
steps: vec![
Step::Fill {
common: sample_common(1),
target: TargetDescriptor {
role: Some("textbox".into()),
name: Some("标题".into()),
tag: "input".into(),
name_attr: None,
placeholder: None,
nearby_label: None,
},
value: "我的第一篇文章".into(),
redacted: None,
},
Step::Select {
common: sample_common(3),
target: TargetDescriptor {
role: Some("combobox".into()),
name: Some("分类".into()),
tag: "select".into(),
name_attr: None,
placeholder: None,
nearby_label: None,
},
selection: vec![SelectedOption {
value: "tech".into(),
label: Some("技术分享".into()),
}],
},
Step::Click {
common: StepCommon {
effect: Some(StepEffect {
navigated_to: "p2".into(),
}),
..sample_common(4)
},
target: sample_target(),
},
],
let v3 = json!({
"version": 3,
"recorded_at": "2026-07-21T08:00:00Z",
"stopped_by": "user_finish",
"entry": { "start_url": "https://example.com/" },
"recorder": { "bsk": "0.1.10", "vom": 1 },
"states": [],
"steps": []
});
match RecordedTrace::classify_value(&v3).unwrap() {
RecordedTrace::V3(t) => assert_eq!(t.version, 3),
other => panic!("expected v3, got {other:?}"),
}
}
#[test]
fn step_click_round_trips() {
let step = Step::Click {
common: sample_common(1),
target: sample_target(),
};
let v = serde_json::to_value(&step).unwrap();
assert_eq!(v.get("op").and_then(|v| v.as_str()), Some("click"));
assert_eq!(v.get("page").and_then(|v| v.as_str()), Some("p1"));
assert!(v.get("key").is_none());
let round: Step = serde_json::from_value(v).unwrap();
assert_eq!(round, step);
fn recorded_trace_rejects_mixed_and_unsupported_versions() {
let v2_with_states = json!({
"recorded_at": "2026-07-21T08:00:00Z",
"entry": { "start_url": "https://example.com/" },
"pages": [{ "id": "p1", "url": "https://example.com/" }],
"states": [],
"steps": []
});
assert!(RecordedTrace::classify_value(&v2_with_states).is_err());
let v3_with_pages = json!({
"version": 3,
"recorded_at": "2026-07-21T08:00:00Z",
"stopped_by": "user_finish",
"entry": { "start_url": "https://example.com/" },
"recorder": { "bsk": "0.1.10", "vom": 1 },
"pages": [{ "id": "p1", "url": "https://example.com/" }],
"states": [],
"steps": []
});
assert!(RecordedTrace::classify_value(&v3_with_pages).is_err());
let unsupported = json!({
"version": 2,
"recorded_at": "2026-07-21T08:00:00Z",
"stopped_by": "user_finish",
"entry": { "start_url": "https://example.com/" },
"recorder": { "bsk": "0.1.10", "vom": 1 },
"states": [],
"steps": []
});
assert!(RecordedTrace::classify_value(&unsupported).is_err());
assert!(serde_json::from_value::<TraceV3>(unsupported).is_err());
}
#[test]
fn step_fill_with_raw_value_round_trips() {
let step = Step::Fill {
common: sample_common(2),
target: TargetDescriptor {
role: Some("textbox".into()),
name: Some("服务名称".into()),
tag: "input".into(),
name_attr: Some("serviceName".into()),
placeholder: None,
nearby_label: None,
},
value: "my-svc".into(),
redacted: None,
};
let v = serde_json::to_value(&step).unwrap();
assert_eq!(v["op"], "fill");
assert_eq!(v["value"], "my-svc");
let round: Step = serde_json::from_value(v).unwrap();
assert_eq!(round, step);
}
#[test]
fn step_fill_password_is_redacted() {
let step = Step::Fill {
common: sample_common(1),
target: TargetDescriptor {
role: Some("textbox".into()),
name: Some("密码".into()),
tag: "input".into(),
name_attr: None,
placeholder: None,
nearby_label: None,
},
value: "***".into(),
redacted: Some(true),
};
let v = serde_json::to_value(&step).unwrap();
assert_eq!(v["value"], "***");
assert_eq!(v["redacted"], true);
}
#[test]
fn step_select_uses_object_array() {
let step = Step::Select {
common: sample_common(3),
target: TargetDescriptor {
role: Some("combobox".into()),
name: Some("分类".into()),
tag: "select".into(),
name_attr: None,
placeholder: None,
nearby_label: None,
},
selection: vec![SelectedOption {
value: "tech".into(),
label: Some("技术分享".into()),
}],
};
let v = serde_json::to_value(&step).unwrap();
assert_eq!(v["selection"][0]["value"], "tech");
assert_eq!(v["selection"][0]["label"], "技术分享");
let round: Step = serde_json::from_value(v).unwrap();
assert_eq!(round, step);
}
#[test]
fn step_navigate_round_trips() {
let step = Step::Navigate {
common: sample_common(1),
to: "https://example.com".into(),
};
let v = serde_json::to_value(&step).unwrap();
assert_eq!(v["op"], "navigate");
assert_eq!(v["to"], "https://example.com");
let round: Step = serde_json::from_value(v).unwrap();
assert_eq!(round, step);
}
#[test]
fn step_press_with_modifiers_round_trips() {
let step = Step::Press {
common: StepCommon {
effect: Some(StepEffect {
navigated_to: "p2".into(),
}),
..sample_common(2)
},
key: "Enter".into(),
modifiers: Some(vec![KeyModifier::Ctrl, KeyModifier::Shift]),
target: Some(sample_target()),
};
let v = serde_json::to_value(&step).unwrap();
assert_eq!(v["key"], "Enter");
assert_eq!(v["modifiers"], json!(["ctrl", "shift"]));
assert_eq!(v["effect"]["navigated_to"], "p2");
let round: Step = serde_json::from_value(v).unwrap();
assert_eq!(round, step);
}
#[test]
fn trace_record_only_round_trips() {
let trace = sample_trace();
let v = serde_json::to_value(&trace).unwrap();
assert!(v.get("version").is_none());
fn assert_classifies_by_integer_version(schema: &serde_json::Value) {
assert_eq!(
v.get("purpose").and_then(|v| v.as_str()),
Some("发布一篇文章")
schema["if"]["required"],
json!(["version"]),
"numeric version is the RecordedTrace classification key"
);
assert!(v.get("parameters").is_none());
assert!(v.get("site").is_none());
assert!(v.get("goal").is_none());
assert_eq!(
v.get("entry")
.and_then(|e| e.get("start_url"))
.and_then(|u| u.as_str()),
Some("https://x.com/editor")
);
let round: Trace = serde_json::from_value(v).unwrap();
assert_eq!(round, trace);
assert_eq!(schema["if"]["properties"]["version"]["type"], "integer");
assert_eq!(schema["then"]["$ref"], "#/definitions/TraceV3");
assert_eq!(schema["else"]["$ref"], "#/definitions/TraceV2");
}
#[test]
fn discriminated_union_click_ignores_extra_key_field() {
let bad = json!({
fn recorded_trace_schema_includes_v2_and_v3() {
let schema = serde_json::to_value(schemars::schema_for!(RecordStopResult)).unwrap();
assert_classifies_by_integer_version(&schema["definitions"]["RecordedTrace"]);
assert_eq!(
schema["definitions"]["TraceV3"]["properties"]["version"]["const"],
3
);
}
#[test]
fn recorded_trace_schema_follows_classify_value() {
let standalone = serde_json::to_value(schemars::schema_for!(RecordedTrace)).unwrap();
assert_classifies_by_integer_version(&standalone);
assert!(standalone["definitions"].get("TraceV2").is_some());
assert!(standalone["definitions"].get("TraceV3").is_some());
let stop_result = serde_json::to_value(schemars::schema_for!(RecordStopResult)).unwrap();
assert_classifies_by_integer_version(&stop_result["definitions"]["RecordedTrace"]);
}
#[test]
fn standalone_trace_schema_is_recorded_trace_union() {
let schema = serde_json::to_value(schemars::schema_for!(RecordedTrace)).unwrap();
assert_eq!(schema["title"], "RecordedTrace");
assert_classifies_by_integer_version(&schema);
assert!(schema["definitions"].get("TraceV2").is_some());
assert!(schema["definitions"].get("TraceV3").is_some());
}
#[test]
fn recorded_trace_accepts_v2_with_unknown_extension_fields() {
let v2 = json!({
"recorded_at": "2026-07-21T08:00:00Z",
"entry": { "start_url": "https://example.com/" },
"pages": [{ "id": "p1", "url": "https://example.com/" }],
"steps": [],
"meta": { "tool": "legacy-exporter" }
});
match RecordedTrace::classify_value(&v2).unwrap() {
RecordedTrace::V2(trace) => assert_eq!(trace.pages.len(), 1),
other => panic!("expected v2, got {other:?}"),
}
}
fn v2_click() -> serde_json::Value {
json!({
"op": "click",
"id": 1,
"page": "p1",
"target": { "tag": "button", "name": "OK" },
"key": "Enter"
});
let step: Step = serde_json::from_value(bad).unwrap();
assert!(matches!(step, Step::Click { .. }));
let v = serde_json::to_value(&step).unwrap();
assert!(v.get("key").is_none());
"target": { "tag": "button", "role": "button", "name": "发布" }
})
}
fn v3_click() -> serde_json::Value {
json!({
"op": "click",
"id": 1,
"state": "s1",
"result": { "state": "s2" },
"target": { "ref": "e1", "role": "button", "name": "发布" }
})
}
fn assert_step_classifies_by_state(schema: &serde_json::Value) {
assert_eq!(
schema["if"]["required"],
json!(["state"]),
"state is the RecordedStep classification key"
);
assert_eq!(schema["then"]["$ref"], "#/definitions/StepV3");
assert_eq!(schema["else"]["$ref"], "#/definitions/StepV2");
let mut forbidden = schema["not"]["required"]
.as_array()
.expect("RecordedStep schema must reject mixed page + state steps")
.iter()
.filter_map(|v| v.as_str())
.collect::<Vec<_>>();
forbidden.sort_unstable();
assert_eq!(forbidden, ["page", "state"]);
}
#[test]
fn extension_trace_deserializes() {
let v = json!({
"recorded_at": "2026-07-21T08:00:00Z",
"started_at": "2026-07-21T07:59:00Z",
"purpose": "demo",
"entry": { "start_url": "https://example.com/editor" },
"pages": [
{ "id": "p1", "url": "https://example.com/editor" },
{ "id": "p2", "url": "https://example.com/p/99" }
],
"steps": [
{
"op": "fill",
"id": 1,
"page": "p1",
"target": { "tag": "input", "role": "textbox", "name": "标题" },
"value": "hello"
},
{
"op": "click",
"id": 2,
"page": "p1",
"target": { "tag": "button", "role": "button", "name": "发布" },
"effect": { "navigated_to": "p2" }
}
]
});
let trace: Trace = serde_json::from_value(v).unwrap();
assert_eq!(trace.entry.start_url, "https://example.com/editor");
assert_eq!(trace.pages.len(), 2);
assert_eq!(trace.steps.len(), 2);
fn recorded_step_classifies_v2_and_v3() {
match RecordedStep::classify_value(&v2_click()).unwrap() {
RecordedStep::V2(StepV2::Click { .. }) => {}
other => panic!("expected v2 click, got {other:?}"),
}
match RecordedStep::classify_value(&v3_click()).unwrap() {
RecordedStep::V3(StepV3::Click { .. }) => {}
other => panic!("expected v3 click, got {other:?}"),
}
}
#[test]
fn recorded_step_rejects_mixed_page_and_state() {
let mut mixed = v2_click();
mixed["state"] = json!("s1");
mixed["result"] = json!({ "state": "s2" });
assert!(RecordedStep::classify_value(&mixed).is_err());
}
#[test]
fn recorded_step_schema_follows_classify_value() {
let schema = serde_json::to_value(schemars::schema_for!(RecordedStep)).unwrap();
assert_eq!(schema["title"], "RecordedStep");
assert_step_classifies_by_state(&schema);
assert!(schema["definitions"].get("StepV2").is_some());
assert!(schema["definitions"].get("StepV3").is_some());
}
}
@@ -0,0 +1,7 @@
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct TraceEntry {
pub start_url: String,
}
+330
View File
@@ -0,0 +1,330 @@
//! Trace v2 — record-only user-action log with `pages[]` and step `page` refs.
//!
//! Legacy wire format with no top-level `version` field.
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
use super::interaction::KeyModifier;
use super::record_common::TraceEntry;
/// Stable semantic handle for an interacted element (v2).
///
/// `name` and `nearby_label` are **untrusted page text**.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct TargetDescriptorV2 {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub role: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub name: Option<String>,
pub tag: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub name_attr: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub placeholder: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub nearby_label: Option<String>,
}
/// Page context dictionary entry — referenced by steps via `page` id.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct PageRefV2 {
pub id: String,
pub url: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub title: Option<String>,
}
/// One selected option (`select` op).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct SelectedOptionV2 {
pub value: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub label: Option<String>,
}
/// Observed navigation after a step (objective fact only).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct StepEffectV2 {
/// Reference into `pages[]` for the destination page.
pub navigated_to: String,
}
/// Fields shared by every v2 step variant (flattened in JSON).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct StepCommonV2 {
pub id: u32,
/// Reference into `pages[]`.
pub page: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub effect: Option<StepEffectV2>,
}
/// One recorded user action — discriminated union by `op` (v2).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
#[serde(tag = "op", rename_all = "snake_case")]
pub enum StepV2 {
Navigate {
#[serde(flatten)]
common: StepCommonV2,
to: String,
},
Click {
#[serde(flatten)]
common: StepCommonV2,
target: TargetDescriptorV2,
},
Hover {
#[serde(flatten)]
common: StepCommonV2,
target: TargetDescriptorV2,
},
Fill {
#[serde(flatten)]
common: StepCommonV2,
target: TargetDescriptorV2,
value: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
redacted: Option<bool>,
},
Select {
#[serde(flatten)]
common: StepCommonV2,
target: TargetDescriptorV2,
selection: Vec<SelectedOptionV2>,
},
Press {
#[serde(flatten)]
common: StepCommonV2,
key: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
modifiers: Option<Vec<KeyModifier>>,
#[serde(default, skip_serializing_if = "Option::is_none")]
target: Option<TargetDescriptorV2>,
},
}
/// Persisted user-action trace exported by legacy `tool.record_stop` / `await`.
///
/// Unknown extension fields are ignored so older traces remain readable.
/// `states[]` and a numeric `version` are reserved for Trace v3 / `RecordedTrace`
/// classification — serde still ignores them, but the JSON Schema rejects them.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct TraceV2 {
/// RFC 3339 timestamp when recording stopped.
pub recorded_at: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub started_at: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub purpose: Option<String>,
pub entry: TraceEntry,
pub pages: Vec<PageRefV2>,
pub steps: Vec<StepV2>,
}
fn constrain_trace_v2_schema(mut schema: schemars::schema::Schema) -> schemars::schema::Schema {
let schemars::schema::Schema::Object(obj) = &mut schema else {
return schema;
};
obj.metadata().title = Some("TraceV2".into());
obj.metadata().description = Some(
"Persisted user-action trace exported by legacy `tool.record_stop` / `await`.\n\n\
Unknown extension fields are ignored so older traces remain readable. \
`states[]` and a numeric `version` are reserved for Trace v3 / `RecordedTrace` classification."
.into(),
);
let object = obj.object.get_or_insert_with(Default::default);
object
.properties
.insert("states".into(), schemars::schema::Schema::Bool(false));
object.properties.insert(
"version".into(),
schemars::schema::SchemaObject {
metadata: Some(Box::new(schemars::schema::Metadata {
description: Some(
"Numeric version selects Trace v3. Legacy v2 envelopes omit this field.".into(),
),
..Default::default()
})),
subschemas: Some(Box::new(schemars::schema::SubschemaValidation {
not: Some(Box::new(
schemars::schema::SchemaObject {
instance_type: Some(schemars::schema::InstanceType::Integer.into()),
..Default::default()
}
.into(),
)),
..Default::default()
})),
..Default::default()
}
.into(),
);
schema
}
impl JsonSchema for TraceV2 {
fn schema_name() -> String {
"TraceV2".into()
}
fn json_schema(generator: &mut schemars::SchemaGenerator) -> schemars::schema::Schema {
#[derive(JsonSchema)]
#[allow(dead_code)]
struct TraceV2Shape {
/// RFC 3339 timestamp when recording stopped.
recorded_at: String,
started_at: Option<String>,
purpose: Option<String>,
entry: TraceEntry,
pages: Vec<PageRefV2>,
steps: Vec<StepV2>,
}
constrain_trace_v2_schema(TraceV2Shape::json_schema(generator))
}
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
fn sample_common(id: u32) -> StepCommonV2 {
StepCommonV2 {
id,
page: "p1".into(),
effect: None,
}
}
fn sample_target() -> TargetDescriptorV2 {
TargetDescriptorV2 {
role: Some("button".into()),
name: Some("发布".into()),
tag: "button".into(),
name_attr: None,
placeholder: None,
nearby_label: None,
}
}
#[test]
fn trace_v2_has_no_version_field() {
let trace = TraceV2 {
recorded_at: "2026-07-17T09:01:10Z".into(),
started_at: None,
purpose: None,
entry: TraceEntry {
start_url: "https://example.com/".into(),
},
pages: vec![PageRefV2 {
id: "p1".into(),
url: "https://example.com/".into(),
title: None,
}],
steps: vec![StepV2::Click {
common: sample_common(1),
target: sample_target(),
}],
};
let v = serde_json::to_value(&trace).unwrap();
assert!(v.get("version").is_none());
assert!(v.get("pages").is_some());
assert!(v.get("states").is_none());
}
#[test]
fn extension_v2_trace_deserializes() {
let v = json!({
"recorded_at": "2026-07-21T08:00:00Z",
"started_at": "2026-07-21T07:59:00Z",
"purpose": "demo",
"entry": { "start_url": "https://example.com/editor" },
"pages": [
{ "id": "p1", "url": "https://example.com/editor" },
{ "id": "p2", "url": "https://example.com/p/99" }
],
"steps": [
{
"op": "fill",
"id": 1,
"page": "p1",
"target": { "tag": "input", "role": "textbox", "name": "标题" },
"value": "hello"
},
{
"op": "click",
"id": 2,
"page": "p1",
"target": { "tag": "button", "role": "button", "name": "发布" },
"effect": { "navigated_to": "p2" }
}
]
});
let trace: TraceV2 = serde_json::from_value(v).unwrap();
assert_eq!(trace.pages.len(), 2);
assert_eq!(trace.steps.len(), 2);
}
#[test]
fn v2_hover_steps_round_trip() {
let value = json!({
"recorded_at": "2026-07-21T08:00:00Z",
"entry": { "start_url": "https://example.com/" },
"pages": [{ "id": "p1", "url": "https://example.com/" }],
"steps": [{
"op": "hover",
"id": 1,
"page": "p1",
"target": { "tag": "span", "role": "button", "name": "Account" }
}]
});
let trace: TraceV2 = serde_json::from_value(value).unwrap();
assert!(matches!(trace.steps.as_slice(), [StepV2::Hover { .. }]));
assert_eq!(
serde_json::to_value(&trace).unwrap()["steps"][0]["op"],
"hover"
);
}
#[test]
fn trace_v2_ignores_unknown_extension_fields() {
let value = json!({
"recorded_at": "2026-07-21T08:00:00Z",
"entry": { "start_url": "https://example.com/" },
"pages": [{ "id": "p1", "url": "https://example.com/" }],
"steps": [],
"meta": { "tool": "legacy-exporter" }
});
let trace: TraceV2 = serde_json::from_value(value).unwrap();
assert_eq!(trace.pages.len(), 1);
assert!(trace.steps.is_empty());
}
#[test]
fn trace_v2_schema_allows_additional_properties() {
let schema = serde_json::to_value(schemars::schema_for!(TraceV2)).unwrap();
assert_ne!(
schema.get("additionalProperties"),
Some(&serde_json::Value::Bool(false)),
"Trace v2 must keep accepting traces with unknown extension fields"
);
}
#[test]
fn trace_v2_schema_forbids_classification_keys() {
let schema = serde_json::to_value(schemars::schema_for!(TraceV2)).unwrap();
assert_eq!(
schema["properties"]["states"],
json!(false),
"Trace v2 schema must reject states[] (reserved for v3)"
);
assert_eq!(
schema["properties"]["version"]["not"]["type"], "integer",
"Trace v2 schema must reject a numeric version (that selects the v3 path)"
);
}
}
+428
View File
@@ -0,0 +1,428 @@
//! Trace v3 — state-action-state log with `states[]` and step `state` refs.
//!
//! Wire format with top-level `version: 3`.
use schemars::JsonSchema;
use serde::{Deserialize, Deserializer, Serialize};
use super::interaction::KeyModifier;
use super::record_common::TraceEntry;
pub const TRACE_VERSION_V3: u32 = 3;
pub const VOM_FORMAT_VERSION: u32 = 1;
fn deserialize_trace_v3_version<'de, D>(deserializer: D) -> Result<u32, D::Error>
where
D: Deserializer<'de>,
{
let version = u32::deserialize(deserializer)?;
if version != TRACE_VERSION_V3 {
return Err(serde::de::Error::custom(format!(
"unsupported trace version {version} (expected {TRACE_VERSION_V3})"
)));
}
Ok(version)
}
fn trace_v3_version_schema(_: &mut schemars::r#gen::SchemaGenerator) -> schemars::schema::Schema {
schemars::schema::SchemaObject {
instance_type: Some(schemars::schema::InstanceType::Integer.into()),
const_value: Some(serde_json::json!(TRACE_VERSION_V3)),
..Default::default()
}
.into()
}
/// Stable semantic handle for an interacted element within a page observation.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct TargetDescriptorV3 {
#[serde(default, skip_serializing_if = "Option::is_none", rename = "ref")]
pub element_ref: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub role: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub name: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub ctx: Option<String>,
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
pub unmatched: bool,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct RecorderInfo {
pub bsk: String,
pub vom: u32,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
#[serde(rename_all = "snake_case")]
pub enum StopReason {
UserFinish,
CliStop,
}
/// Page observation dictionary entry — referenced by steps via `state` / `result.state`.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct TraceStateV3 {
pub id: String,
pub url: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub title: Option<String>,
/// Full page observation (front matter + VOM body + annotations).
pub body: String,
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
pub truncated: bool,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct StepResultV3 {
pub state: String,
}
/// Fields shared by every step variant (flattened in JSON).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct StepCommonV3 {
pub id: u32,
/// Observation id immediately before this action.
pub state: String,
pub result: StepResultV3,
}
/// One selected option (`select` op).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct SelectedOptionV3 {
pub value: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub label: Option<String>,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
#[serde(rename_all = "snake_case")]
pub enum NavigationCause {
UserTyped,
Link,
FormSubmit,
Reload,
History,
Script,
Browser,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
#[serde(rename_all = "snake_case")]
pub enum FillCommit {
Enter,
Suggestion,
Blur,
}
/// One recorded user action — discriminated union by `op`.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
#[serde(tag = "op", rename_all = "snake_case")]
pub enum StepV3 {
Navigate {
#[serde(flatten)]
common: StepCommonV3,
to: String,
cause: NavigationCause,
},
Click {
#[serde(flatten)]
common: StepCommonV3,
target: TargetDescriptorV3,
},
Hover {
#[serde(flatten)]
common: StepCommonV3,
target: TargetDescriptorV3,
},
Fill {
#[serde(flatten)]
common: StepCommonV3,
target: TargetDescriptorV3,
value: String,
commit: FillCommit,
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
redacted: bool,
},
Select {
#[serde(flatten)]
common: StepCommonV3,
target: TargetDescriptorV3,
#[serde(default, skip_serializing_if = "Vec::is_empty")]
selection: Vec<SelectedOptionV3>,
},
Press {
#[serde(flatten)]
common: StepCommonV3,
key: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
modifiers: Option<Vec<KeyModifier>>,
#[serde(default, skip_serializing_if = "Option::is_none")]
target: Option<TargetDescriptorV3>,
},
Scroll {
#[serde(flatten)]
common: StepCommonV3,
},
}
/// Wire trace returned by `tool.record_stop` / `await`.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
#[serde(deny_unknown_fields)]
pub struct TraceV3 {
#[serde(deserialize_with = "deserialize_trace_v3_version")]
#[schemars(schema_with = "trace_v3_version_schema")]
pub version: u32,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub purpose: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub started_at: Option<String>,
pub recorded_at: String,
pub stopped_by: StopReason,
pub entry: TraceEntry,
pub recorder: RecorderInfo,
pub states: Vec<TraceStateV3>,
pub steps: Vec<StepV3>,
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
fn sample_common(id: u32, state: &str, result_state: &str) -> StepCommonV3 {
StepCommonV3 {
id,
state: state.into(),
result: StepResultV3 {
state: result_state.into(),
},
}
}
fn sample_target() -> TargetDescriptorV3 {
TargetDescriptorV3 {
element_ref: Some("e21".into()),
role: Some("button".into()),
name: Some("发布".into()),
ctx: Some("金桔柠檬 6 号".into()),
unmatched: false,
}
}
fn sample_trace() -> TraceV3 {
TraceV3 {
version: TRACE_VERSION_V3,
purpose: Some("把草稿商品发布上架".into()),
started_at: Some("2026-08-10T02:10:41.080Z".into()),
recorded_at: "2026-08-10T02:12:55.360Z".into(),
stopped_by: StopReason::UserFinish,
entry: TraceEntry {
start_url: "https://example.com/".into(),
},
recorder: RecorderInfo {
bsk: "0.1.10".into(),
vom: VOM_FORMAT_VERSION,
},
states: vec![
TraceStateV3 {
id: "s1".into(),
url: "https://example.com/".into(),
title: Some("Example Domain".into()),
body: "@vom 1\nRootWebArea \"Example Domain\"".into(),
truncated: false,
},
TraceStateV3 {
id: "s2".into(),
url: "https://shop.example.com/products?status=draft".into(),
title: Some("商品管理".into()),
body: "@vom 1\nRootWebArea \"商品管理\"".into(),
truncated: false,
},
],
steps: vec![
StepV3::Navigate {
common: sample_common(1, "s1", "s2"),
to: "https://shop.example.com/products?status=draft".into(),
cause: NavigationCause::UserTyped,
},
StepV3::Fill {
common: sample_common(2, "s2", "s2"),
target: TargetDescriptorV3 {
element_ref: Some("e12".into()),
role: Some("textbox".into()),
name: Some("搜索商品".into()),
ctx: None,
unmatched: false,
},
value: "金桔柠檬".into(),
commit: FillCommit::Enter,
redacted: false,
},
StepV3::Click {
common: sample_common(3, "s2", "s2"),
target: sample_target(),
},
],
}
}
#[test]
fn step_click_round_trips() {
let step = StepV3::Click {
common: sample_common(1, "s1", "s2"),
target: sample_target(),
};
let v = serde_json::to_value(&step).unwrap();
assert_eq!(v.get("op").and_then(|v| v.as_str()), Some("click"));
assert_eq!(v.get("state").and_then(|v| v.as_str()), Some("s1"));
assert_eq!(v["result"]["state"], "s2");
assert_eq!(v["target"]["ref"], "e21");
let round: StepV3 = serde_json::from_value(v).unwrap();
assert_eq!(round, step);
}
#[test]
fn step_fill_with_commit_round_trips() {
let step = StepV3::Fill {
common: sample_common(2, "s2", "s3"),
target: TargetDescriptorV3 {
element_ref: Some("e12".into()),
role: Some("textbox".into()),
name: Some("搜索商品".into()),
ctx: None,
unmatched: false,
},
value: "browser skill".into(),
commit: FillCommit::Enter,
redacted: false,
};
let v = serde_json::to_value(&step).unwrap();
assert_eq!(v["op"], "fill");
assert_eq!(v["commit"], "enter");
assert!(v.get("redacted").is_none());
let round: StepV3 = serde_json::from_value(v).unwrap();
assert_eq!(round, step);
}
#[test]
fn step_fill_password_is_redacted() {
let step = StepV3::Fill {
common: sample_common(1, "s1", "s1"),
target: TargetDescriptorV3 {
element_ref: Some("e3".into()),
role: Some("textbox".into()),
name: Some("密码".into()),
ctx: None,
unmatched: false,
},
value: "***".into(),
commit: FillCommit::Blur,
redacted: true,
};
let v = serde_json::to_value(&step).unwrap();
assert_eq!(v["value"], "***");
assert_eq!(v["redacted"], true);
}
#[test]
fn step_navigate_with_cause_round_trips() {
let step = StepV3::Navigate {
common: sample_common(1, "s1", "s2"),
to: "https://example.com".into(),
cause: NavigationCause::UserTyped,
};
let v = serde_json::to_value(&step).unwrap();
assert_eq!(v["op"], "navigate");
assert_eq!(v["cause"], "user_typed");
let round: StepV3 = serde_json::from_value(v).unwrap();
assert_eq!(round, step);
}
#[test]
fn trace_v3_round_trips() {
let trace = sample_trace();
let v = serde_json::to_value(&trace).unwrap();
assert_eq!(v.get("version").and_then(|v| v.as_u64()), Some(3));
assert!(v.get("pages").is_none());
assert_eq!(v["recorder"]["vom"], 1);
assert_eq!(v["states"].as_array().unwrap().len(), 2);
let round: TraceV3 = serde_json::from_value(v).unwrap();
assert_eq!(round, trace);
}
#[test]
fn default_fields_are_omitted() {
let step = StepV3::Click {
common: sample_common(1, "s1", "s1"),
target: TargetDescriptorV3 {
element_ref: Some("e1".into()),
role: Some("button".into()),
name: Some("OK".into()),
ctx: None,
unmatched: false,
},
};
let v = serde_json::to_value(&step).unwrap();
assert!(v.get("unmatched").is_none());
assert!(v["target"].get("ctx").is_none());
assert!(v["target"].get("unmatched").is_none());
}
#[test]
fn unmatched_target_serializes_flag() {
let step = StepV3::Click {
common: sample_common(1, "s1", "s2"),
target: TargetDescriptorV3 {
element_ref: None,
role: Some("button".into()),
name: Some("发布".into()),
ctx: None,
unmatched: true,
},
};
let v = serde_json::to_value(&step).unwrap();
assert_eq!(v["target"]["unmatched"], true);
assert!(v["target"].get("ref").is_none());
}
#[test]
fn extension_trace_deserializes() {
let v = json!({
"version": 3,
"recorded_at": "2026-07-21T08:00:00Z",
"started_at": "2026-07-21T07:59:00Z",
"purpose": "demo",
"stopped_by": "user_finish",
"entry": { "start_url": "https://example.com/editor" },
"recorder": { "bsk": "0.1.10", "vom": 1 },
"states": [
{ "id": "s1", "url": "https://example.com/editor", "body": "@vom 1" },
{ "id": "s2", "url": "https://example.com/p/99", "body": "@vom 1" }
],
"steps": [
{
"op": "fill",
"id": 1,
"state": "s1",
"result": { "state": "s1" },
"target": { "ref": "e1", "role": "textbox", "name": "标题" },
"value": "hello",
"commit": "blur"
},
{
"op": "click",
"id": 2,
"state": "s1",
"result": { "state": "s2" },
"target": { "ref": "e2", "role": "button", "name": "发布" }
}
]
});
let trace: TraceV3 = serde_json::from_value(v).unwrap();
assert_eq!(trace.version, 3);
assert_eq!(trace.states.len(), 2);
assert_eq!(trace.steps.len(), 2);
}
}
+8 -7
View File
@@ -6,7 +6,7 @@ A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (dsh) tool
[BrowserSkill](https://github.com/Tencent/BrowserSkill) (`bsk`) browser automation to the model.
Each tool maps to one `bsk <cmd> --json` invocation: the plugin spawns the bsk CLI, parses its
structured JSON output, and returns a canonical typed value. The bsk daemon, browser, and Chrome
structured JSON output, and returns a canonical typed value. The bsk daemon, browser, and browser
extension keep owning the actual browser control — this package is a thin, well-typed bridge.
## Tools
@@ -64,9 +64,10 @@ dsh plugin --profile <name> add @wxg-prc-cpg/browser-skill-dsh-plugin
dsh --profile <name>
```
Prerequisite: the `bsk` CLI must be installed and on `PATH`, and the BrowserSkill Chrome extension
must be connected — see the [BrowserSkill README](https://github.com/Tencent/BrowserSkill). When bsk
is missing, tool calls fail with install guidance instead of a bare spawn error.
Prerequisite: the `bsk` CLI must be installed and on `PATH`, and the BrowserSkill browser extension
(Chrome or Edge) must be connected — see the
[BrowserSkill README](https://github.com/Tencent/BrowserSkill). When bsk is missing, tool calls fail
with install guidance instead of a bare spawn error.
## Configuration
@@ -76,7 +77,7 @@ All fields are optional and validated through the plugin's Schemastery `Config`:
# cordis.patch.yml override example
- insert:
- id: browserskill
name: dsh-plugin-browserskill
name: "@wxg-prc-cpg/browser-skill-dsh-plugin"
config:
bskPath: bsk # path to the bsk binary (default: resolve from PATH)
defaultTimeoutMs: 120000
@@ -186,8 +187,8 @@ git tag dsh-plugin-v0.1.0
git push origin dsh-plugin-v0.1.0
```
Or run the workflow from the Actions tab (`workflow_dispatch`). The job reads
`NPM_TOKEN` from the GitHub Environment of the same name.
Or run the workflow from the Actions tab (`workflow_dispatch`). The job reads the
`NPM_TOKEN` secret from the `npm-publish` GitHub Environment.
## License
@@ -1,3 +1,5 @@
- insert:
- id: browserskill
name: dsh-plugin-browserskill
# Must be the published package name — the loader imports this specifier.
# Quoted because YAML reserves a leading '@' in plain scalars.
name: "@wxg-prc-cpg/browser-skill-dsh-plugin"
@@ -1,7 +1,7 @@
{
"name": "@wxg-prc-cpg/browser-skill-dsh-plugin",
"description": "DeepSeek Harness tool plugin that exposes BrowserSkill (bsk) browser automation to the model",
"version": "0.1.0",
"version": "0.1.1",
"license": "MIT",
"publishConfig": {
"access": "public",
@@ -12,7 +12,7 @@
* events, and never move the registry's current pointer.
*/
import { createHash } from "node:crypto";
import { createHash, randomUUID } from "node:crypto";
import { readFile, unlink } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
@@ -78,6 +78,7 @@ const FAILURE_BACKOFF_THRESHOLD = 3;
const FRAME_RING_SIZE = 2;
export class ObservationService {
private readonly scratchNamespace = randomUUID();
private readonly observations = new Map<string, SessionObservation>();
private readonly listeners = new Set<(event: ObservationEvent) => void>();
private readonly captureTimers = new Map<string, unknown>();
@@ -304,9 +305,9 @@ export class ObservationService {
if (current === undefined || current.dead === true) return;
if (this.captureInFlight.has(sessionId)) return;
this.captureInFlight.add(sessionId);
// One fixed scratch path per session (overwritten each frame) and always
// unlinked after the bytes are read — /tmp must not grow without bound.
const outPath = join(tmpdir(), `bsk-obs-${sessionId}.png`);
// Stable within this service, isolated from captures owned by other instances.
const sessionKey = createHash("sha256").update(sessionId).digest("hex").slice(0, 16);
const outPath = join(tmpdir(), `bsk-obs-${this.scratchNamespace}-${sessionKey}.png`);
let writtenPath = outPath;
try {
const result = await this.deps.queue.run(sessionId, () =>
@@ -5,7 +5,7 @@
import { existsSync, mkdtempSync, writeFileSync } from "node:fs";
import type { IncomingMessage, ServerResponse } from "node:http";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { basename, dirname } from "node:path";
import { describe, expect, it } from "vitest";
import {
actionForLabel,
@@ -145,7 +145,7 @@ function own(registry: SessionRegistry, sessionId: string): void {
}
/** Poll until the condition holds (capture completion is real async I/O). */
async function waitFor(cond: () => boolean, timeoutMs = 2000): Promise<void> {
async function waitFor(cond: () => boolean, timeoutMs = 10_000): Promise<void> {
const start = Date.now();
while (!cond()) {
if (Date.now() - start > timeoutMs) throw new Error("waitFor timeout");
@@ -525,22 +525,6 @@ describe("availability and dead sessions", () => {
});
});
describe("thumbnail read-back", () => {
it("serves captured frames through readThumbnail and 404s unknown ids", async () => {
const scheduler = fakeScheduler();
const runner = fakeRunner();
const { service } = setup({ runner, scheduler });
service.addSession("s1");
scheduler.runNext();
await waitFor(() => service.getState()[0]?.thumbnailAttachmentId !== undefined);
const frame = await service.readThumbnail("obs-s1-1");
expect(frame?.mediaType).toBe("image/png");
expect([...(frame?.data ?? [])]).toEqual([0x89, 0x50, 0x4e, 0x47]);
expect(await service.readThumbnail("nope")).toBeUndefined();
service.dispose();
});
});
describe("actionForLabel", () => {
it("maps tool labels onto observation verbs", () => {
expect(actionForLabel("navigate")).toBe("navigating");
@@ -564,8 +548,12 @@ describe("scratch files and thumbnail references", () => {
);
const firstOut = runner.calls.filter((c) => c.args[0] === "screenshot").at(-1)?.args;
const pathOf = (args: string[]) => args[args.indexOf("--out") + 1];
expect(pathOf(firstOut ?? [])).toBe(join(tmpdir(), "bsk-obs-s1.png"));
expect(existsSync(pathOf(firstOut ?? []))).toBe(false);
const firstPath = pathOf(firstOut ?? []);
expect(dirname(firstPath)).toBe(tmpdir());
expect(basename(firstPath)).toMatch(
/^bsk-obs-[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}-[0-9a-f]{16}\.png$/,
);
expect(existsSync(firstPath)).toBe(false);
// second frame: same path, deleted again
service.endAction("s1");
scheduler.runNext();
@@ -575,11 +563,33 @@ describe("scratch files and thumbnail references", () => {
scheduler.pending().length === 1,
);
const secondOut = runner.calls.filter((c) => c.args[0] === "screenshot").at(-1)?.args;
expect(pathOf(secondOut ?? [])).toBe(join(tmpdir(), "bsk-obs-s1.png"));
await waitFor(() => !existsSync(pathOf(secondOut ?? [])));
const secondPath = pathOf(secondOut ?? []);
expect(secondPath).toBe(firstPath);
await waitFor(() => !existsSync(secondPath));
service.dispose();
});
it("isolates scratch paths between service instances", async () => {
const first = setup({ scheduler: fakeScheduler(), runner: fakeRunner() });
const second = setup({ scheduler: fakeScheduler(), runner: fakeRunner() });
first.service.addSession("s1");
second.service.addSession("s1");
first.scheduler.runNext();
second.scheduler.runNext();
await waitFor(
() =>
first.service.getState()[0]?.thumbnailAttachmentId !== undefined &&
second.service.getState()[0]?.thumbnailAttachmentId !== undefined,
);
const pathOf = (runner: FakeRunner) => {
const args = runner.calls.find((call) => call.args[0] === "screenshot")?.args ?? [];
return args[args.indexOf("--out") + 1];
};
expect(pathOf(first.runner)).not.toBe(pathOf(second.runner));
first.service.dispose();
second.service.dispose();
});
it("keeps the last two distinct frames and evicts the third", async () => {
let n = 0;
const runner = fakeRunner({
@@ -2,6 +2,7 @@ import { readFile } from "node:fs/promises";
import { basename, dirname, resolve as resolvePath } from "node:path";
import { transform } from "lightningcss";
import { defineConfig, type UserConfig } from "tsdown";
import pkg from "./package.json" with { type: "json" };
/**
* Two build faces of the dual-face package:
@@ -33,7 +34,10 @@ const VENDORED_LIBRARY = /^@deepseek-ai\/(cosmokit|schemastery)(\/|$)/;
const CSS_VIRTUAL_PREFIX = "\0bsk-css:";
const CSS_VIRTUAL_SUFFIX = ".mjs";
const CLIENT_ID = "dsh-plugin-browserskill";
// The web shell resolves client bundles from its module table by package name
// (dsh's own bundles register the same way), so this must never drift from
// package.json.
const CLIENT_ID = pkg.name;
const client: UserConfig = {
name: `${CLIENT_ID}/client`,
Loaded 100 of 104 files, more files were not shown because too many files have changed in this diff. Show more