Compare commits

..
Author SHA1 Message Date
Zhang GH 945bf1523d Merge pull request #173 from Tencent/release/0.2.0
Release/0.2.0
2026-09-02 19:00:21 +08:00
drakezhangandCursor a928a9fb76 test: update popup version display assertions for CLI/Ext labels
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 18:32:33 +08:00
drakezhangandCursor 0b2d7e95cc style: format release script for biome
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 18:26:35 +08:00
drakezhangandCursor 556bea1645 chore(release): 0.2.0
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 18:17:56 +08:00
drakezhangandCursor 0302d5592b chore: add release script, changelog, and version display labels
- scripts/release.mjs: unified version bump + tag + push automation
- CHANGELOG.md: document all changes from 0.1.x through 0.2.0
- popup: show "CLI {ver} / Ext {ver}" instead of bare version numbers

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 18:05:32 +08:00
Hannah Liu eb34ec0dbf Merge pull request #124 from Tencent/feat/expand-upload-download
feat(bsk): support drop to upload
2026-09-02 17:56:55 +08:00
drakezhangandCursor 43067b6bfe style: remove extra blank lines in file-transfer test
Fix biome formatter error on CI.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 17:52:57 +08:00
drakezhangandCursor 6cc0d38489 docs(skill): compress CLI SKILL.md with file transfer guidance
Restore the lean style from PR #167 while integrating essential file
transfer documentation for upload/download. Reduces the skill from
362 lines / 26k chars to 180 lines / 9.2k chars.

A/B eval (24 browser runs, Composer 2.5 Fast, core suite) confirms
no pass-rate regression: candidate 12/12 vs baseline 10/12 (baseline
failures were a network blip and a behavioral variance, not skill
quality).

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 17:43:58 +08:00
drakezhangandCursor 0b71963344 merge: resolve conflicts with updated feat/upload-download base
Keep feat/expand-upload-download (drop mode superset) over the base
branch version for all 12 conflicting files. The drop-to-upload branch
already includes all upload/download functionality plus the new --mode
drop mechanism, overlay suppression, OOPIF coordinate handling, and
expanded error reasons.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 17:03:35 +08:00
Hannah Liu 59e2c3224e Merge pull request #123 from Tencent/feat/upload-download
feat(bsk): add secure, session-scoped file upload and download | 支持安全、会话级的文件上传与下载
2026-09-02 16:47:19 +08:00
drakezhangandCursor d70df753f6 merge: resolve SKILL.md conflicts with main
Keep the expanded command-reference tables from feat/upload-download
(including the new File Transfer section) over main's compact two-line
format.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 16:42:15 +08:00
Zhang GH 66ae555e40 Merge pull request #172 from Tencent/fix/pip-close-and-obs-mediatype
fix(dsh-plugin): PiP window close button & observation thumbnail media type detection
2026-09-02 16:25:33 +08:00
drakezhangandCursor fee7c0983c fix: sort import order to satisfy biome organizeImports
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 16:21:26 +08:00
drakezhangandCursor 51473c01ff fix(dsh-plugin): add PiP close button and sniff observation thumbnail media type
The Document PiP window had no close/return entry once popped out —
users could not dismiss it from inside the window. Add an onClosePip
callback and render a close button in the PiP header; clicking it
calls pipWindow.close() and the content returns to its carrier.

The observation thumbnail publisher hardcoded mediaType as "image/png"
regardless of the actual capture bytes. Reuse the existing
sniffImageMediaType helper (exported from image.ts) so the HTTP
thumbnail endpoint declares the correct Content-Type when Chromium
returns JPEG captures.

Closes #153 (Bug 2 fully; Bug 1 was fixed in #135, this covers the
remaining observation-thumbnail path).

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 16:14:24 +08:00
Zhang GH 49744233f7 Merge pull request #171 from Tencent/fix/borrow-confirmation-ux
fix(borrow): improve confirmation UX — proactive focus and longer tim…
2026-09-02 15:23:12 +08:00
drakezhangandCursor 95571f8b70 fix(borrow): improve confirmation UX — proactive focus and longer timeout
The borrow confirmation overlay was easily missed because the Agent
Window stole focus and the 5-second auto-deny was too aggressive for
users who had context-switched away.

- Focus the user window proactively when a borrow request is issued so
  the overlay is immediately visible even when the Agent Window was
  last focused.
- Raise CONFIRMATION_TIMEOUT_MS from 5 s to 60 s, giving users enough
  time to notice and respond — the OS notification with
  requireInteraction: true now has a matching window of usefulness.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 15:11:40 +08:00
Zhang GH 8344e311fa Merge pull request #169 from Tencent/perf/vom-hover-probe-opt-in
Perf/vom hover probe opt in
2026-09-02 14:27:29 +08:00
drakezhangandCursor 63338cd98e Merge branch 'main' into perf/vom-hover-probe-opt-in
main rewrote SKILL.md into a leaner core (#167) while this branch documented
the opt-in --probe-hover flag against the old structure. Keep main's rewritten
prose and re-apply the probe-hover guidance on top of it: the [has-submenu] /
[expanded] markers, when reaching for --probe-hover is warranted, and its slot
in the page-reading escalation ladder ahead of get-html/screenshot.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 12:18:06 +08:00
Zhang GH bd193ceb5d Merge pull request #167 from Tencent/refactor/lean-browser-skill
Refactor/lean browser skill
2026-09-02 12:09:42 +08:00
Zhang GH 5ed828736f Merge pull request #165 from Tencent/perf/vom-algorithmic-optimizations
perf(vom): optimize rendering algorithms
2026-09-02 12:09:22 +08:00
drakezhang ee620acf64 docs(skill): give probe-hover a decidable trigger condition
The previous wording asked the agent to notice when a page "looks like"
it hides content behind hover. That is not something an agent reading a
text tree can judge: hover-only content is invisible by definition, and
the marker that used to point at it came from the probing that is now off
by default.

Replaces it with a condition the agent can actually evaluate — an
expected control is missing and no [has-submenu] marker points at a
trigger — and places it on the escalation ladder ahead of get-html and
screenshot, which is where that dead end is currently reached.

Also documents [has-submenu]/[expanded] as the primary hover signal. It
comes from aria-haspopup in the accessibility tree, costs nothing, and is
unaffected by probing being off, so it covers menus that are marked up
correctly.
2026-09-02 10:59:18 +08:00
drakezhang 05a36075b8 perf(vom): make active hover probing opt-in on observe
Every tool.observe moved the real cursor across the page to look for
hover-only menus and tooltips. Measured against real Chrome, that cost a
median of 2.5s and up to 4.7s per observation — 73% to 99% of total
observe wall clock — and returned 3 hits across 12 pages. A positive
control confirms the probes work; the yield on real pages simply does not
justify putting them on the default path.

observe now probes only when asked, via `probe_hover` (`--probe-hover`).
Agents that know which element hides a menu should keep using
`bsk hover <ref>`, which is both cheaper and precise. A held hover latch
still suppresses probing, since probing would move the cursor off the
element the caller is deliberately holding.

Correctness fixes that apply whether or not probing is requested:

- Hover probing ran between the DOM snapshot and the accessibility tree
  fetch, so a hover that opened a menu left one half of an observation
  describing the page before the change and the other half after it. Both
  chains now run after all capture is complete.
- The overlay bypass toggled once per chain and swallowed restore
  failures, which could leave the agent overlay hidden for the rest of
  the tab's life. It is now reference counted across both chains, and a
  failed restore is reported instead of discarded.
- Observations report whether they hovered the page and whether that
  revealed content the static tree lacks, so callers can judge how far
  the live page may have drifted from the returned snapshot.

Probe budgets were checked at the top of the loop, so a candidate could
start with 1ms remaining and still run a full settle window, overshooting
by up to 634ms. Candidates are now admitted only if they fit. Raising
MAX_HOVER_PROBE_MS to 2600 is part of that fix rather than a relaxation:
2000 was never the real ceiling, and enforcing it literally would have
silently dropped a candidate the positive control depends on.

Tooltip misses are cached too, with a shorter TTL than hits, so repeat
observations of the same page no longer re-probe every unlabelled control
from scratch.
2026-09-02 10:59:18 +08:00
Ljy-0827 1bd13d5c3c fix(bsk): fix upload download racing 2026-09-02 10:56:02 +08:00
drakezhangandCursor 59d6533eff docs(skill): trim agent instructions to a leaner core
The CLI skill had grown to 20.8 KB of command tables and flag catalogs that
`bsk --help` already answers at runtime, so every agent paid for them on every
task. Rewrite both skills around the durable rules an agent cannot rediscover
-- session lifecycle, the observe/act/observe loop, the Agent Window boundary,
and the stop-on-success contract -- and delegate flag details to the CLI's own
help output. The CLI skill drops to 6.9 KB (-67%) and the DSH skill to 5.1 KB.

Keep the parts that help discovery has no answer for: the literal
`[hover first: ...]` observation marker, the `request-help` outcome values, the
complete command-name inventory, and the required flags agents guess wrong
(`fill --value`, `select --value`, `screenshot --out`, `emulate --device`).
Also bind `--no-focus` explicitly to `bsk session start`, since a bare mention
led agents to pass it to `bsk navigate`.

Validated with 36 agent runs of the `core` eval suite against a real browser,
comparing the old and new instructions with everything else held constant.
Pass rate did not regress (the lean version was better on `form-controls`) and
token cost per task came out even. The inventory and flag signatures cut
`--help` round-trips from 1.7 to 0.7 per task and removed the invented
`bsk back` / `bsk forward` calls the first draft produced.

Tighten the DSH size assertion to a range so the budget is bounded from both
sides and accidental truncation fails the test.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-01 18:39:19 +08:00
drakezhangandCursor 536d3a9844 chore: ignore local pnpm store and git worktrees
Both directories are local scratch space that kept showing up as untracked
noise in `git status`.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-01 18:39:06 +08:00
drakezhang 641d934337 perf(vom): optimize rendering algorithms 2026-09-01 14:55:36 +08:00
Ljy-0827 3b21f2f0a7 fix(bsk): download bug fix 2026-08-31 17:58:33 +08:00
Ljy-0827 405433ab65 fix(bsk): fix ci 2026-08-31 17:58:33 +08:00
Ljy-0827 58c7f38a05 feat(bsk): harden browser file transfer transactions 2026-08-31 17:58:33 +08:00
Ljy-0827 e0c88295c7 fix(bsk): fix ci 2026-08-31 17:58:33 +08:00
Ljy-0827 dc36e38cb9 fix(bsk): upload function safety 2026-08-31 17:58:33 +08:00
Ljy-0827 d5de9b0154 feat(bsk): fix upload function that cannot bypass layout 2026-08-31 17:58:33 +08:00
Ljy-0827 995061514c feat(bsk): fix different type of upload 2026-08-31 17:58:33 +08:00
Ljy-0827 406bb21b7e feat(bsk): support upload and download 2026-08-31 17:58:33 +08:00
Zhang GH f06a514bd2 Merge pull request #149 from Tencent/fix/protocol-upgrade-reminder
fix:将协议版本差异改为升级提醒
2026-08-31 12:25:29 +08:00
Zhang GH 1e53ee63ae Merge pull request #155 from Tencent/ci/edge-addons-publishing
ci(extension): publish updates to Microsoft Edge Add-ons
2026-08-31 11:24:22 +08:00
Zhang GH 120dc441bb Merge pull request #158 from Tencent/codex/fix-dsh-screenshot-test-name
test(dsh-plugin): use renamed screenshot tool
2026-08-29 16:08:01 +08:00
drakezhang 710b1219f4 test(dsh-plugin): use renamed screenshot tool 2026-08-29 16:05:06 +08:00
Zhang GH 177ecd3ebf Merge pull request #157 from Tencent/refactor/simplify-dsh-commands
Refactor/simplify dsh commands
2026-08-29 15:55:16 +08:00
Zhang GH 0e6b9c4542 Merge pull request #135 from cyhano/fix/screenshot-media-type
fix: use detected media type for screenshots instead of hard-coded image/png
2026-08-29 15:52:43 +08:00
drakezhang 506adbe708 Merge remote-tracking branch 'origin/main' into refactor/simplify-dsh-commands 2026-08-29 15:16:31 +08:00
Zhang GH 984d017b0f Merge pull request #156 from Tencent/fix/unify-dsh-plugin-package-id
fix(dsh-plugin): use the published package name as the Cordis id
2026-08-29 15:12:02 +08:00
drakezhang 5df93143c5 refactor(dsh-plugin): simplify browser commands 2026-08-29 14:38:48 +08:00
Zhang GH 7163f2f9b0 Merge pull request #151 from xiyanjun/docs/evaluate-json-ok
docs(skill): clarify bsk evaluate exit-code vs --json .ok semantics
close #133
2026-08-28 22:06:04 +08:00
drakezhang e9f0584af3 ci(extension): publish updates to Microsoft Edge Add-ons 2026-08-28 21:39:53 +08:00
drakezhangandCursor 2997f495db fix(dsh-plugin): use the published package name as the Cordis id
dsh 0.1.0-rc.8 imports and registers the client bundle by export const name, so a leftover dsh-plugin-browserskill id fails to load the 0.1.1 web UI.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-28 21:28:06 +08:00
xiyanjun f96bd72c3c docs(skill): clarify bsk evaluate exit-code vs --json .ok semantics
The exit code reflects RPC success only, not JS success. Document that programmatic callers must use --json and check the .ok field to detect JS errors. Addresses the discoverability gap raised in issue #133.
2026-08-28 18:02:38 +08:00
Zhang GH 329c68054b Merge pull request #150 from Tencent/codex/browser-eval-harness
test: 增加可扩展的浏览器能力回归测试框架
2026-08-28 16:41:00 +08:00
drakezhang 5cd43553ed docs(eval): clarify agent-neutral scope 2026-08-28 15:36:38 +08:00
drakezhang 9ad33801c7 test: add extensible browser evaluation harness 2026-08-28 12:16:39 +08:00
Zhang GH 7aaa5a2d01 Merge pull request #147 from Tencent/fix/dsh-plugin-session-lifecycle
fix(dsh-plugin): stabilize skill loading and browser sessions
2026-08-28 12:11:08 +08:00
drakezhang 210b21b637 style(dsh-plugin): format observation 2026-08-28 12:06:47 +08:00
Hannah Liu 48f64ec0df Merge pull request #122 from Tencent/feat/vom-functional-refactor
feat(vom): refactor semantic structure generation for grouped observations | 重构语义结构生成,支持分组式 VOM 观察
2026-08-28 11:24:34 +08:00
“haonan”andCursor c48a53269e style: rustfmt doctor protocol-version test asserts
cargo fmt --check failed on two long assert lines in PR CI.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-27 22:50:01 +08:00
“haonan”andCursor 3bbd6ab19f fix: treat protocol version differences as upgrade reminders
Agents were stopping on popup and status copy that looked like a hard incompatibility. Keep the connection usable, reframe exit 5 as an unknown method, and ask users to upgrade instead of aborting.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-27 22:22:30 +08:00
drakezhang 1a70d2200e fix(dsh-plugin): stabilize skill loading and browser sessions
Register bundled browser instructions at agent scope, prioritize foreground commands over thumbnail capture, and unify cancellation and session cleanup.
2026-08-27 19:59:20 +08:00
Zhang GH 554861d6b1 Merge pull request #131 from Tencent/feat/dsh-browser-tools-parity
feat(dsh-plugin): add Browser tool parity and dedicated Harness skill
2026-08-27 16:29:30 +08:00
Ljy-0827 0d7f976335 fix(vom): fix repeat names 2026-08-26 15:28:27 +08:00
Ljy-0827 d1d3eb6a94 fix(vom): bug fix 2026-08-26 15:28:27 +08:00
Ljy-0827 38fd7d3f4c fix(vom): safety policy and rebase bug fix 2026-08-26 15:28:27 +08:00
Ljy-0827 b5062ac19b feat(vom): fix name lost 2026-08-26 15:28:27 +08:00
Ljy-0827 41cb576256 feat(vom): function refactor 2026-08-26 15:28:27 +08:00
Hannah Liu f060a7c074 Merge pull request #142 from Tencent/chore/release-ext-0.1.7
chore(release): extension 0.1.7
2026-08-26 15:19:33 +08:00
Ljy-0827 60b62d5420 chore(release): extension 0.1.7 2026-08-26 15:12:44 +08:00
Hannah Liu 70b851a3e8 Merge pull request #141 from Tencent/chore/release-bsk-0.1.11
chore(release): bsk 0.1.11
2026-08-26 15:06:30 +08:00
Ljy-0827 da3534d066 chore(release): bsk 0.1.11 2026-08-26 14:59:09 +08:00
BB-fat a004291848 feat(dsh-plugin): better-sidebar 集成、会话关停与归档清理、悬浮窗修复 (#130)
* better-sidebar 集成:observation 视图迁入 better-sidebar 载体,按会话作用域展示 bsk 会话
* 会话关停与归档清理:支持会话 dispose 与归档自动清理
* 悬浮窗修复:悬浮 overlay 拖拽/缩放/PiP 弹出等问题修复
* 修复 Document PiP 空白窗口:载体卸载(关 tab、切会话、插件 HMR)时主动 close() PiP 窗口,避免残留空白窗口
2026-08-25 10:33:46 +08:00
Ljy-0827 b1b1ae1d48 feat(bsk): update skill to support drop 2026-08-24 16:55:16 +08:00
Ljy-0827 8822bba9a5 feat(bsk): support drop to upload 2026-08-24 16:55:16 +08:00
Ljy-0827 c74c119a82 fix(bsk): download bug fix 2026-08-24 16:26:04 +08:00
cyhanoandDeepSeek Harness 60fbf30e24 fix: use detected media type for screenshots instead of hard-coded image/png
browser_screenshot declared every capture as image/png regardless of the
actual bytes. On Chromium/Edge builds where captureVisibleTab returns JPEG
bytes, the stored attachment reference recorded mediaType "image/png" against
JPEG bytes, so dsh-attachment-local's read-time metadata verification fails
with ATTACHMENT_CORRUPT ("Stored attachment metadata does not match its
reference"), surfacing as code UNKNOWN and triggering endless LLM retries.

- image.ts: sniff the capture's magic bytes (PNG/JPEG/WebP/GIF) and declare
  that real media type to attachments.saveImage, instead of always "image/png".
- tools.ts: assemble the tool result from ref.mediaType (the store's verified
  type) rather than re-declaring "image/png" as const, and relax the output
  schema's const to the accepted media-type enum.
- tests: add a regression case asserting JPEG capture bytes surface as
  image/jpeg.

Co-Authored-By: DeepSeek Harness <noreply@deepseek.com>
2026-08-24 15:31:34 +08:00
Ljy-0827 b801bc1f62 fix(bsk): fix ci 2026-08-24 15:02:52 +08:00
Ljy-0827 bf59753aa3 feat(bsk): harden browser file transfer transactions 2026-08-24 15:02:52 +08:00
Ljy-0827 a86cbf724f fix(bsk): fix ci 2026-08-24 15:00:22 +08:00
Ljy-0827 76d447caa0 fix(bsk): upload function safety 2026-08-24 15:00:22 +08:00
Ljy-0827 14e5bbe4fc feat(bsk): fix upload function that cannot bypass layout 2026-08-24 15:00:22 +08:00
Ljy-0827 bef722107f feat(bsk): fix different type of upload 2026-08-24 14:59:53 +08:00
Ljy-0827 6800f8fdfe feat(bsk): support upload and download 2026-08-24 14:59:11 +08:00
Hannah Liu 4b6cdde168 Merge pull request #115 from Tencent/fix/recorder-iframe
fix(recording): support user action recording in iframes and OOPIFs
2026-08-24 13:18:05 +08:00
Ljy-0827 4aade4b94f fix(recorder): bug fix, oopif load too late 2026-08-24 10:42:46 +08:00
Ljy-0827 616af1099f fix(record): saftey policy and rebase bug fix 2026-08-24 10:42:46 +08:00
Ljy-0827 3656f0dd3c fix(recorder): ci fix 2026-08-24 10:42:46 +08:00
Ljy-0827 9742d83213 fix(recorder): bug fix 2026-08-24 10:42:46 +08:00
Ljy-0827 615e866608 fix(recorder): add iframe support 2026-08-24 10:42:46 +08:00
Ljy-0827 c08fed5de3 fix(record): saftey policy and rebase bug fix 2026-08-24 10:42:45 +08:00
“haonan” 887d1ed0b9 fix(record): drop 'hover' in tracev2 for compatible 2026-08-24 10:42:45 +08:00
Ljy-0827 1a3d3b7de8 fix(record): bug fix 2026-08-24 10:42:45 +08:00
“haonan”andCursor 72834980d2 fix(record): record switch_tab and keep Trace v3 states per tab
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-24 10:42:45 +08:00
“haonan” 38b0debba9 fix(test): share one BSK_HOME lock across lib tests 2026-08-24 10:40:37 +08:00
“haonan” 4677149e5e fix(record): recover failed Trace exports and hint --url when the default start page fails
Reject legacy JSON --output before recording starts, persist the completed Trace before writing the bundle so `bsk record stop --output <dir>` can retry after an export failure, and tell users to pass --url when https://example.com/ does not load.
2026-08-23 21:26:40 +08:00
Ljy-0827 8ea5477e11 fix(record): safety problem and bug fix 2026-08-23 21:26:40 +08:00
“haonan” 17af1af65a feat(record): export Trace v3 bundles with settled page states 2026-08-23 21:26:40 +08:00
Ljy-0827 6300eda0aa fix(record): rebase and dto policy 2026-08-23 21:26:40 +08:00
Ljy-0827 87cdcfe5ca fix(recording): using unified geometry 2026-08-23 21:26:40 +08:00
“haonan”andCursor 21476c3884 feat(record): capture Trace v3 states after pages settle
Wait for DOM quiet after actions, match targets to VOM refs, and reduce
drafts into deduped state-linked Trace v3 while keeping the v2 reducer.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-23 21:26:40 +08:00
“haonan” c2a225377b fix(protocol): align trace schemas with v2/v3 classify rules
RecordedTrace validation now follows classify_value — numeric version
selects v3, otherwise v2 — so mixed pages/states and v2 envelopes with
a version field fail Schema checks before Recall import. Keep standalone
trace_step as a v2|v3 union for tools that validate a single step.
2026-08-23 21:26:39 +08:00
“haonan” e64f4cac9f fix(protocol): keep generic trace schema compatible with v2 and v3
Split standalone v2/v3 schemas and drop TraceV2 deny_unknown_fields so legacy traces with extra keys still parse; mix and version checks stay in RecordedTrace.
2026-08-23 21:26:39 +08:00
Ljy-0827 368507b07d fix(protocol): compatible with dto 2026-08-23 21:26:39 +08:00
Ljy-0827 b750eed5d4 fix(protocol): integrate with vom 2026-08-23 21:26:39 +08:00
“haonan” 3d159c0254 test(protocol): align handshake fixtures with 1.0 floor and clarify v2/v3 naming 2026-08-23 21:26:39 +08:00
“haonan”andCursor 524516cb56 fix(protocol): align Trace v2/v3 classification and hover steps
Require Trace v3 version const 3, reject mixed pages/states payloads, and
align TypeScript selection optionality with the generated schema. Trace v2
on main already records hover steps, so the frozen v2 schema must accept
them instead of dropping them during reduction.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-23 21:26:39 +08:00
“haonan”andCursor f98198949a feat(protocol): negotiate Trace v3 with v2 compatibility
Define Trace v3 wire types and generated schemas while keeping protocol
1.1 peers compatible with 1.0. Default record start still produces Trace
v2, and only an explicit trace_version=3 request enables the new shape.
Refs #90

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-23 21:26:39 +08:00
Zhang GH d05279f991 Merge pull request #98 from NianJiuZst/codex/fix-daemon-idle-close-race
fix(daemon): make idle disconnect updates atomic
2026-08-23 21:25:24 +08:00
drakezhang a4b6a61d29 docs(dsh-plugin): clarify package description 2026-08-22 21:29:33 +08:00
drakezhang cad0b5cbc9 feat(dsh-plugin): add Browser tool parity 2026-08-22 21:15:53 +08:00
Zhang GH 2719e76438 Merge pull request #97 from NianJiuZst/codex/cancel-session-lifecycle-rpcs
fix(session): cancel lifecycle RPCs cooperatively
2026-08-22 14:51:38 +08:00
polarday dd9fc9edba Merge pull request #100 from Tencent/feat/record-vom-observation
feat(vom): expose record-safe semantic observations
2026-08-21 16:21:41 +08:00
Ljy-0827 005a6fd534 fix(vom): add record safe observations, unify definition and construction of safe DTO 2026-08-21 11:26:52 +08:00
“haonan” 20554508db fix(vom): omit raw AX/DOM trees from record-safe observations
Return only frame identity and node geometry for matching so a serialized capture result cannot leak form values.
2026-08-20 20:45:19 +08:00
NianJiuZst 8352c3c67a fix(daemon): refresh stale idle-task comment
After folding the IPC connection count and activity timestamp into a
single mutex-protected IpcActivityState, the idle task no longer
references the connection count directly; should_exit_for_idle already
returns false while a connection is open or within the idle window.
Update the comment to describe the post-refactor control flow so it
matches the code.
2026-08-20 13:53:23 +08:00
NianJiuZst 4eedd98c1d fix(daemon): make idle disconnect updates atomic 2026-08-20 13:53:23 +08:00
“haonan” 6aa931b60a chore: retrigger opensource scan 2026-08-19 19:56:07 +08:00
Ljy-0827 3df2738280 fix(vom): fix CI 2026-08-19 15:05:55 +08:00
Ljy-0827 f86e506d47 feat(vom): update contract 2026-08-19 15:05:55 +08:00
“haonan” 45a246c41b feat(vom): expose record-safe semantic observations
Share the VOM capture pipeline with recording, enrich rendered refs, and support value redaction without regressing cooperative cancellation.
2026-08-19 15:05:55 +08:00
NianJiuZst 7035e46024 fix(session): cancel lifecycle RPCs cooperatively 2026-08-17 15:13:53 +08:00
264 changed files with 35197 additions and 4169 deletions
+177
View File
@@ -325,3 +325,180 @@ jobs:
echo "Chrome Web Store submission created successfully:"
jq '{itemId, state, warningInfo}' "$PUBLISH_RESPONSE"
publish-edge-addons:
needs: [resolve, build-extension]
runs-on: ubuntu-latest
environment: edge-addons
permissions:
contents: read
steps:
- name: Download build artifact
uses: actions/download-artifact@v8
with:
name: extension-zip
path: dist
- name: Validate Microsoft Edge Add-ons configuration
shell: bash
env:
API_KEY: ${{ secrets.EDGE_ADDONS_API_KEY }}
CLIENT_ID: ${{ vars.EDGE_ADDONS_CLIENT_ID }}
PRODUCT_ID: ${{ vars.EDGE_ADDONS_PRODUCT_ID }}
run: |
set -euo pipefail
if [ -z "$API_KEY" ]; then
echo "::error::Environment secret EDGE_ADDONS_API_KEY is not configured"
exit 1
fi
if [ -z "$CLIENT_ID" ]; then
echo "::error::Environment variable EDGE_ADDONS_CLIENT_ID is not configured"
exit 1
fi
if [ -z "$PRODUCT_ID" ]; then
echo "::error::Environment variable EDGE_ADDONS_PRODUCT_ID is not configured"
exit 1
fi
if ! [[ "$PRODUCT_ID" =~ ^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$ ]]; then
echo "::error::EDGE_ADDONS_PRODUCT_ID must be the Partner Center product GUID, not the public Edge Add-ons extension ID"
exit 1
fi
ARCHIVE="$(find dist -maxdepth 1 -name '*.zip' -type f -print -quit)"
if [ -z "$ARCHIVE" ]; then
echo "::error::No extension zip found in the downloaded artifact"
exit 1
fi
echo "ARCHIVE=$ARCHIVE" >> "$GITHUB_ENV"
- name: Upload and publish Microsoft Edge Add-ons extension
shell: bash
env:
API_KEY: ${{ secrets.EDGE_ADDONS_API_KEY }}
CLIENT_ID: ${{ vars.EDGE_ADDONS_CLIENT_ID }}
PRODUCT_ID: ${{ vars.EDGE_ADDONS_PRODUCT_ID }}
RELEASE_VERSION: ${{ needs.resolve.outputs.version }}
run: |
set -euo pipefail
API_ROOT="https://api.addons.microsoftedge.microsoft.com/v1"
UPLOAD_HEADERS="$RUNNER_TEMP/edge-upload-headers.txt"
UPLOAD_RESPONSE="$RUNNER_TEMP/edge-upload-response.json"
STATUS_RESPONSE="$RUNNER_TEMP/edge-status-response.json"
PUBLISH_HEADERS="$RUNNER_TEMP/edge-publish-headers.txt"
PUBLISH_RESPONSE="$RUNNER_TEMP/edge-publish-response.json"
operation_id_from_headers() {
local headers="$1"
local location
location="$(awk '
tolower($1) == "location:" {
sub(/\r$/, "")
sub(/^[^:]*:[[:space:]]*/, "")
print
exit
}
' "$headers")"
location="${location%%\?*}"
printf '%s' "${location##*/}"
}
poll_operation() {
local operation_name="$1"
local status_url="$2"
local status="InProgress"
local http_status
for attempt in $(seq 1 30); do
sleep 5
http_status="$(curl --silent --show-error \
--output "$STATUS_RESPONSE" \
--write-out '%{http_code}' \
--header "Authorization: ApiKey $API_KEY" \
--header "X-ClientID: $CLIENT_ID" \
"$status_url")"
if [ "$http_status" -lt 200 ] || [ "$http_status" -ge 300 ]; then
echo "::error::Microsoft Edge Add-ons $operation_name status check failed with HTTP $http_status"
jq . "$STATUS_RESPONSE" || cat "$STATUS_RESPONSE"
exit 1
fi
status="$(jq -r '.status // empty' "$STATUS_RESPONSE")"
echo "$operation_name status check $attempt/30: $status"
case "$status" in
Succeeded)
return 0
;;
InProgress)
;;
Failed)
echo "::error::Microsoft Edge Add-ons $operation_name failed"
jq . "$STATUS_RESPONSE"
exit 1
;;
*)
echo "::error::Microsoft Edge Add-ons returned an unknown $operation_name status: $status"
jq . "$STATUS_RESPONSE"
exit 1
;;
esac
done
echo "::error::Microsoft Edge Add-ons $operation_name did not complete after 30 checks"
jq . "$STATUS_RESPONSE"
exit 1
}
HTTP_STATUS="$(curl --silent --show-error \
--output "$UPLOAD_RESPONSE" \
--dump-header "$UPLOAD_HEADERS" \
--write-out '%{http_code}' \
--request POST \
--header "Authorization: ApiKey $API_KEY" \
--header "X-ClientID: $CLIENT_ID" \
--header "Content-Type: application/zip" \
--data-binary "@$ARCHIVE" \
"$API_ROOT/products/$PRODUCT_ID/submissions/draft/package")"
if [ "$HTTP_STATUS" != "202" ]; then
echo "::error::Microsoft Edge Add-ons upload failed with HTTP $HTTP_STATUS"
jq . "$UPLOAD_RESPONSE" || cat "$UPLOAD_RESPONSE"
exit 1
fi
UPLOAD_OPERATION_ID="$(operation_id_from_headers "$UPLOAD_HEADERS")"
if [ -z "$UPLOAD_OPERATION_ID" ]; then
echo "::error::Microsoft Edge Add-ons upload response has no Location operation ID"
cat "$UPLOAD_HEADERS"
exit 1
fi
poll_operation \
"package upload" \
"$API_ROOT/products/$PRODUCT_ID/submissions/draft/package/operations/$UPLOAD_OPERATION_ID"
HTTP_STATUS="$(curl --silent --show-error \
--output "$PUBLISH_RESPONSE" \
--dump-header "$PUBLISH_HEADERS" \
--write-out '%{http_code}' \
--request POST \
--header "Authorization: ApiKey $API_KEY" \
--header "X-ClientID: $CLIENT_ID" \
--header "Content-Type: application/json" \
--data "{\"notes\":\"Automated BrowserSkill extension v${RELEASE_VERSION} release from GitHub Actions.\"}" \
"$API_ROOT/products/$PRODUCT_ID/submissions")"
if [ "$HTTP_STATUS" != "202" ]; then
echo "::error::Microsoft Edge Add-ons publish failed with HTTP $HTTP_STATUS"
jq . "$PUBLISH_RESPONSE" || cat "$PUBLISH_RESPONSE"
exit 1
fi
PUBLISH_OPERATION_ID="$(operation_id_from_headers "$PUBLISH_HEADERS")"
if [ -z "$PUBLISH_OPERATION_ID" ]; then
echo "::error::Microsoft Edge Add-ons publish response has no Location operation ID"
cat "$PUBLISH_HEADERS"
exit 1
fi
poll_operation \
"submission" \
"$API_ROOT/products/$PRODUCT_ID/submissions/operations/$PUBLISH_OPERATION_ID"
echo "Microsoft Edge Add-ons submission created successfully:"
jq '{status, message, errorCode, errors}' "$STATUS_RESPONSE"
+8
View File
@@ -7,10 +7,18 @@ dist/
.output/
.wxt/
.web-extension-id
.pnpm-store/
# Git worktrees
.worktrees/
# Logs
*.log
# Local browser evaluation artifacts
evals/browser/results/
evals/browser/agents.local.json
# macOS
.DS_Store
+92
View File
@@ -0,0 +1,92 @@
# Changelog
All notable changes to BrowserSkill will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/).
Starting from 0.2.0, CLI / Extension / DSH Plugin share the same version number.
## [Unreleased]
## [0.2.0] - 2026-09-02
### Added
- File transfer: upload and download support across CLI, Extension, and DSH Plugin
- File transfer: drag-and-drop upload (`drop-to-upload`)
- VOM semantic graph, name enrichment, and hover perception modules
- VOM hover probing (opt-in via `observe` parameter)
- DSH Plugin: browser tool parity with CLI commands
- Edge Add-ons automated publishing in CI
- Protocol upgrade reminder when CLI / Extension protocol versions differ
- Browser evaluation harness (`evals/browser/`)
- Unified release script (`scripts/release.mjs`)
### Changed
- **Version scheme**: all three components now share the same semver
- VOM rendering algorithm optimizations
- Leaner SKILL.md agent instructions
- DSH Plugin: simplified browser commands
- Borrow confirmation UX — proactive focus and longer timeout
- PiP window now has a close button
### Fixed
- Screenshot media type detection (was hard-coded to `image/png`)
- DSH Plugin session lifecycle stability
- DSH Plugin Cordis package ID mismatch
- Observation thumbnail media type sniffing
- Upload/download race conditions and layout bypass issues
- VOM repeated name and safety policy issues
---
*Previous releases used independent version numbers per component.*
## CLI 0.1.11 / Extension 0.1.7 / DSH Plugin 0.1.2 — 2026-08-26 ~ 2026-08-29
### Added
- DSH Plugin sidebar integration, session lifecycle, and archive cleanup
- Recorder iframe and OOPIF support
### Changed
- VOM functional refactor
### Fixed
- Recorder safety policy and bug fixes
## CLI 0.1.10 / Extension 0.1.6 — 2026-08-08
### Added
- VOM observation recording and settled-state detection
- Record overlay timer
### Fixed
- Browser keepalive disconnect handling
## CLI 0.1.9 / Extension 0.1.5 — 2026-07-29
### Added
- CLI auto-update mechanism
- Trace v3 protocol and recorder
### Fixed
- MV3 keepalive disconnect
## CLI 0.1.8 / Extension 0.1.4 — 2026-07-22
### Added
- More browser interaction actions
### Fixed
- Windows named-pipe hash-only path issue
## CLI 0.1.7 / Extension 0.1.3 — 2026-07-07
Initial public release pair.
## CLI 0.1.6 — 2026-06-30
### Fixed
- Minor CLI fixes
## CLI 0.1.5 / Extension 0.1.2 — 2026-06-22
First tagged releases.
Generated
+3 -2
View File
@@ -134,9 +134,10 @@ dependencies = [
[[package]]
name = "bsk"
version = "0.1.10"
version = "0.2.0"
dependencies = [
"anyhow",
"base64",
"bsk-protocol",
"clap",
"console",
@@ -168,7 +169,7 @@ dependencies = [
[[package]]
name = "bsk-protocol"
version = "0.1.10"
version = "0.2.0"
dependencies = [
"schemars",
"semver",
+3 -2
View File
@@ -7,7 +7,7 @@ members = [
default-members = ["crates/bsk-cli"]
[workspace.package]
version = "0.1.10"
version = "0.2.0"
edition = "2024"
rust-version = "1.85"
license = "MIT"
@@ -36,13 +36,14 @@ flate2 = "1.1.9"
tar = "0.4.46"
zip = { version = "7.2.0", default-features = false, features = ["deflate"] }
sha2 = "0.11.0"
base64 = "0.22"
tempfile = "3"
nix = { version = "0.29", features = ["signal", "process"] }
rand = "0.8"
uuid = { version = "1", features = ["v4", "serde"] }
# Workspace crates (version required for `cargo publish` of dependents)
bsk-protocol = { version = "0.1.9", path = "crates/bsk-protocol" }
bsk-protocol = { version = "0.2.0", path = "crates/bsk-protocol" }
[profile.release]
strip = "symbols"
+1
View File
@@ -208,6 +208,7 @@ The repository is a Cargo + pnpm workspace:
- `apps/extension` — browser extension
- `packages/ui` and `packages/i18n` — shared extension UI support
- `packages/dsh-plugin-browserskill` — DeepSeek Harness plugin (`@wxg-prc-cpg/browser-skill-dsh-plugin`)
- [`evals/browser`](evals/browser/README.md) — deterministic local pages and agent-neutral browser capability evaluation
## License
+1
View File
@@ -178,6 +178,7 @@ Agent 不直接与浏览器通信。它通过 `bsk` CLI 下发浏览器任务;
- `apps/extension` — 浏览器扩展
- `packages/ui` 和 `packages/i18n` — 扩展 UI 共享支持
- `packages/dsh-plugin-browserskill` — DeepSeek Harness 插件(`@wxg-prc-cpg/browser-skill-dsh-plugin`)
- [`evals/browser`](evals/browser/README.zh-CN.md) — 确定性本地页面与 Agent 无关的浏览器能力测试台
## 许可证
+6 -3
View File
@@ -1,6 +1,6 @@
# BrowserSkill — Privacy Policy
**Last updated:** May 25, 2026
**Last updated:** August 20, 2026
This Privacy Policy describes how the **BrowserSkill** browser extension (the "Extension") handles information when you install and use it. BrowserSkill is published as part of the open-source [BrowserSkill](https://github.com/Tencent/BrowserSkill) project. The source code is publicly auditable.
@@ -14,7 +14,7 @@ BrowserSkill is a local automation bridge that lets AI coding agents (such as Cu
## 2. Single Purpose
The Extension's single purpose is to expose browser automation primitives (navigation, DOM observation, screenshots, clicks, form filling, tab management) to a locally running BrowserSkill daemon over a WebSocket connection on `127.0.0.1`, so that an AI agent invoked by the user can interact with web pages on the user's behalf.
The Extension's single purpose is to expose browser automation primitives (navigation, DOM observation, screenshots, clicks, form filling, task-scoped file transfer, and tab management) to a locally running BrowserSkill daemon over a WebSocket connection on `127.0.0.1`, so that an AI agent invoked by the user can interact with web pages on the user's behalf.
## 3. Data the Extension Accesses
@@ -26,6 +26,7 @@ Depending on the commands the user (via their AI agent) sends to the local daemo
| **User input simulated by the agent** | Mouse clicks, keystrokes, and form values that the AI agent dispatches through the Chrome DevTools Protocol (CDP). | Required to perform automation actions the user has asked the agent to do. |
| **Tab and window metadata** | Tab IDs, URLs, titles, window IDs of the Agent Window and any tabs the user explicitly authorizes. | Required to target automation commands at the correct tab/window. |
| **Local extension storage** | A randomly generated 8-character instance ID and an optional user-supplied label. | Used so the local daemon can recognize this browser instance across reconnects. No personal data is stored. |
| **File transfers requested by the agent** | Local files explicitly supplied to `bsk upload`, and the file created by a single `bsk download` action. | Required to attach a task file to a web page or return a browser-generated download to the invoking local agent. |
| **OS notifications** | Permission to display a system notification when the agent requests to "borrow" one of the user's existing tabs. | Required to obtain explicit, per-tab user consent before the agent touches any pre-existing tab. |
## 4. Data the Extension Does **Not** Collect
@@ -34,7 +35,7 @@ BrowserSkill does **not**:
- Send any data to remote servers, the Extension's authors, or any third party.
- Call any LLM, AI, or cloud API. The Extension contains no API keys, model identifiers, or remote endpoints.
- Read or transmit cookies, browsing history, bookmarks, downloads, saved passwords, or autofill data.
- Read or transmit cookies, browsing history, bookmarks, saved passwords, or autofill data. It observes only the download initiated by an active `bsk download` call, not download history generally.
- Use webcam, microphone, geolocation, or any device sensor.
- Include analytics, telemetry, crash reporting, advertising SDKs, or fingerprinting code.
- Track users across websites or across sessions.
@@ -50,6 +51,7 @@ The Extension requests the following Chrome permissions. Each is used solely for
- **`alarms`** — Periodically wake the service worker to keep the local WebSocket connection alive.
- **`idle`** — Detect when the device returns from idle/locked so the Extension can promptly re-establish the local WebSocket connection after the machine wakes. No idle data is stored or transmitted.
- **`notifications`** — Show a system notification to obtain user approval before the agent borrows a user-owned tab.
- **`downloads`** — Correlate and route the one browser download initiated by an active `bsk download` command. If that claimed transaction fails, BrowserSkill cancels an in-progress file or removes its completed temporary browser file. It is not used to enumerate download history or alter unclaimed downloads.
- **`storage`** — Persist a random instance ID and optional label in `chrome.storage.local`.
- **Host permission `<all_urls>`** — Inject a small status overlay (showing "Agent Active") on pages controlled by the agent, and enable automation across whatever sites the user directs the agent to. The Extension does **not** read or transmit page content from sites the agent is not actively driving.
@@ -61,6 +63,7 @@ All Extension activity stays on the user's local device. The only network traffi
- The instance ID and optional label persist in `chrome.storage.local` until the user uninstalls the Extension or clears extension storage.
- Page content, screenshots, DOM snapshots, and other observed data are returned to the local daemon in response to commands and are **not retained by the Extension**. They live only as long as the agent's tool call.
- Upload and download bytes are staged by the local daemon in a private, session-scoped directory. Download staging is removed after it is copied to the requested destination. Upload staging is retained until the session ends so a later form submission can still read the attached file. Remaining staging is removed when the session ends or disconnects, or when the daemon next starts after a crash.
## 8. User Control
@@ -0,0 +1,81 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>BrowserSkill file-transfer fixture</title>
<style>
body { font: 16px system-ui; max-width: 760px; margin: 48px auto; }
.actions { display: flex; gap: 12px; align-items: center; margin: 24px 0; }
button, a { padding: 10px 16px; border: 1px solid #777; border-radius: 8px; }
a[aria-disabled="true"] { opacity: .45; pointer-events: none; }
#drop { padding: 36px; border: 3px dashed #777; border-radius: 12px; text-align: center; }
#drop.dragging { border-color: #2563eb; background: #eff6ff; }
img { display: block; max-width: 480px; max-height: 360px; margin-top: 20px; }
</style>
</head>
<body>
<h1>Upload then download the same image</h1>
<p id="status">No image selected</p>
<div class="actions">
<button id="upload" type="button">Upload through file input</button>
<input id="file" type="file" accept="image/png" hidden />
<a id="download" aria-disabled="true">Download uploaded image</a>
</div>
<div id="drop" role="button" tabindex="0">Drop an image here</div>
<img id="preview" alt="Uploaded image preview" hidden />
<script>
const input = document.querySelector('#file');
const upload = document.querySelector('#upload');
const drop = document.querySelector('#drop');
const download = document.querySelector('#download');
const preview = document.querySelector('#preview');
const status = document.querySelector('#status');
let objectUrl;
window.__bskTransferFixture = { events: [], result: null };
function accept(file, mechanism) {
if (!file) return;
if (objectUrl) URL.revokeObjectURL(objectUrl);
objectUrl = URL.createObjectURL(file);
preview.src = objectUrl;
preview.hidden = false;
download.href = objectUrl;
download.download = file.name;
download.removeAttribute('aria-disabled');
status.textContent = `Ready via ${mechanism}: ${file.name} (${file.size} bytes)`;
window.__bskTransferFixture.result = {
mechanism,
name: file.name,
size: file.size,
type: file.type,
hasPathProperty: 'path' in file,
};
}
upload.addEventListener('click', () => input.click());
input.addEventListener('change', () => accept(input.files[0], 'input'));
for (const type of ['dragenter', 'dragover']) {
drop.addEventListener(type, event => {
const types = [...event.dataTransfer.types];
window.__bskTransferFixture.events.push({ type, types });
if (!types.includes('Files')) return;
event.preventDefault();
drop.classList.add('dragging');
});
}
drop.addEventListener('dragleave', () => drop.classList.remove('dragging'));
drop.addEventListener('drop', event => {
window.__bskTransferFixture.events.push({
type: 'drop',
types: [...event.dataTransfer.types],
fileCount: event.dataTransfer.files.length,
});
event.preventDefault();
drop.classList.remove('dragging');
accept(event.dataTransfer.files[0], 'drop');
});
</script>
</body>
</html>
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@browser-skill/extension",
"version": "0.1.6",
"version": "0.2.0",
"private": true,
"type": "module",
"scripts": {
@@ -23,7 +23,7 @@ describe("BorrowConfirmationOverlay", () => {
id: "req-1",
isActiveTab: true,
tabTitle: "Example",
timeoutMs: 5000,
timeoutMs: 60_000,
onAllow,
onDeny,
},
@@ -32,7 +32,7 @@ describe("BorrowConfirmationOverlay", () => {
);
expect(screen.getByText("允许借用标签页?")).toBeTruthy();
for (let i = 0; i < 5; i++) {
for (let i = 0; i < 60; i++) {
await act(async () => {
await vi.advanceTimersByTimeAsync(1000);
});
@@ -66,7 +66,7 @@ describe("BorrowConfirmationOverlay", () => {
id: "req-fallback",
isActiveTab: true,
tabTitle: "Example",
timeoutMs: 5000,
timeoutMs: 60_000,
onAllow,
onDeny,
},
@@ -74,7 +74,7 @@ describe("BorrowConfirmationOverlay", () => {
/>,
);
for (let i = 0; i < 5; i++) {
for (let i = 0; i < 60; i++) {
await act(async () => {
await vi.advanceTimersByTimeAsync(1000);
});
@@ -1,10 +1,12 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { RECORD_STOP, type RecordStepPayload } from "@/lib/record-bridge";
import { handleRecordContentMessage, startRecordCapture } from "../record-capture";
import type { RecordStepPayload } from "@/lib/record-bridge";
import { startRecordCapture } from "../record-capture";
vi.stubGlobal("chrome", {
runtime: {
sendMessage: vi.fn(() => Promise.resolve()),
sendMessage: vi.fn((message: { sequence?: number }) =>
Promise.resolve({ ok: true, sequence: message.sequence }),
),
},
});
@@ -46,58 +48,6 @@ function click(el: Element): void {
el.dispatchEvent(new MouseEvent("click", { bubbles: true, button: 0, detail: 1 }));
}
describe("handleRecordContentMessage stop/cancel", () => {
it("ignores STOP when no recording is active", () => {
const dispose = vi.fn();
const onStop = vi.fn();
const setActiveRequestId = vi.fn();
const setCapture = vi.fn();
const sendResponse = vi.fn();
const needsAsync = handleRecordContentMessage(
{ type: RECORD_STOP, requestId: "rec-stale" },
{
activeRequestId: null,
capture: { dispose },
setActiveRequestId,
setCapture,
onStart: vi.fn(),
onStop,
},
sendResponse,
);
expect(needsAsync).toBe(false);
expect(dispose).not.toHaveBeenCalled();
expect(onStop).not.toHaveBeenCalled();
expect(setActiveRequestId).not.toHaveBeenCalled();
expect(setCapture).not.toHaveBeenCalled();
expect(sendResponse).not.toHaveBeenCalled();
});
it("ignores STOP for a mismatched requestId", () => {
const dispose = vi.fn();
const onStop = vi.fn();
const needsAsync = handleRecordContentMessage(
{ type: RECORD_STOP, requestId: "rec-other" },
{
activeRequestId: "rec-1",
capture: { dispose },
setActiveRequestId: vi.fn(),
setCapture: vi.fn(),
onStart: vi.fn(),
onStop,
},
vi.fn(),
);
expect(needsAsync).toBe(false);
expect(dispose).not.toHaveBeenCalled();
expect(onStop).not.toHaveBeenCalled();
});
});
describe("record-capture semantic", () => {
let steps: RecordStepPayload[];
@@ -174,6 +124,19 @@ describe("record-capture semantic", () => {
]);
});
it("does not emit page navigation from a child Document capture", () => {
const originalUrl = location.href;
const capture = startRecordCapture("rec-child", (step) => steps.push(step), {
captureNavigation: false,
});
history.pushState({}, "", "#inside-frame");
expect(steps).toEqual([]);
capture.dispose();
history.replaceState({}, "", originalUrl);
});
it("does not record clicks on anonymous layout divs", () => {
document.body.innerHTML = `
<div id="chrome">page chrome</div>
@@ -228,6 +191,7 @@ describe("record-capture semantic", () => {
expect(steps[0]).toMatchObject({
op: "hover",
target: { role: "button", name: "Open user navigation menu" },
geometry: { tag: "button", rect: { x: 900, y: 8, w: 32, h: 32 } },
});
expect(steps[1]).toMatchObject({
op: "click",
@@ -0,0 +1,102 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import type { RecordFramePortMessage } from "@/lib/recording/frame-bridge";
import { RECORD_FRAME_START } from "@/lib/recording/frame-bridge";
import { RECORD_DOCUMENT_ATTRIBUTE } from "@/shared/recording-document-identity";
import { RecordFrameAgent } from "../recording/frame-agent";
class PortListeners<T extends (...args: never[]) => unknown> {
readonly values = new Set<T>();
addListener = (listener: T) => this.values.add(listener);
removeListener = (listener: T) => this.values.delete(listener);
}
function portHarness() {
const onMessage = new PortListeners<(message: unknown) => void>();
const onDisconnect = new PortListeners<() => void>();
const outbound: RecordFramePortMessage[] = [];
const port = {
name: "bsk-record-frame",
onMessage,
onDisconnect,
postMessage(message: RecordFramePortMessage) {
outbound.push(message);
if (message.type === "ready") {
queueMicrotask(() => {
for (const listener of onMessage.values) {
listener({
type: "ready_ack",
requestId: message.requestId,
producerId: message.producerId,
});
}
});
}
},
disconnect: vi.fn(),
} as unknown as chrome.runtime.Port;
return {
port,
outbound,
receive(message: RecordFramePortMessage) {
for (const listener of onMessage.values) listener(message);
},
};
}
describe("RecordFrameAgent", () => {
afterEach(() => {
document.documentElement.removeAttribute(RECORD_DOCUMENT_ATTRIBUTE);
document.body.replaceChildren();
});
it("keeps a failed stop retryable and flushes the final dirty fill", async () => {
const harness = portHarness();
const sendMessage = vi
.fn<(message: { sequence: number }) => Promise<unknown>>()
.mockRejectedValueOnce(new Error("offline"))
.mockRejectedValueOnce(new Error("still offline"))
.mockImplementation(async (message) => ({ ok: true, sequence: message.sequence }));
vi.stubGlobal("chrome", {
runtime: {
connect: vi.fn(() => harness.port),
sendMessage,
},
});
document.body.innerHTML = `<label for="draft">Draft</label><input id="draft" />`;
const agent = new RecordFrameAgent();
await expect(
agent.start({ type: RECORD_FRAME_START, requestId: "rec-1", startedAtMs: 10 }),
).resolves.toEqual({ ok: true });
expect(document.documentElement.hasAttribute(RECORD_DOCUMENT_ATTRIBUTE)).toBe(true);
const input = document.querySelector("input")!;
input.dispatchEvent(new FocusEvent("focusin", { bubbles: true }));
input.value = "final value";
input.dispatchEvent(new Event("input", { bubbles: true }));
harness.receive({ type: "stop", requestId: "rec-1", commandId: "stop-1" });
await vi.waitFor(() =>
expect(harness.outbound).toContainEqual({
type: "stopped",
requestId: "rec-1",
commandId: "stop-1",
ok: false,
error: "failed to deliver one or more recorded steps",
}),
);
expect(document.documentElement.hasAttribute(RECORD_DOCUMENT_ATTRIBUTE)).toBe(true);
harness.receive({ type: "stop", requestId: "rec-1", commandId: "stop-2" });
await vi.waitFor(() =>
expect(harness.outbound).toContainEqual({
type: "stopped",
requestId: "rec-1",
commandId: "stop-2",
ok: true,
}),
);
expect(sendMessage).toHaveBeenCalledTimes(3);
expect(document.documentElement.hasAttribute(RECORD_DOCUMENT_ATTRIBUTE)).toBe(false);
});
});
@@ -0,0 +1,36 @@
import { describe, expect, it, vi } from "vitest";
import type { RecordStepMessage } from "@/lib/record-bridge";
import { RecordStepDelivery } from "../record-step-delivery";
describe("RecordStepDelivery", () => {
it("retries from the first unacknowledged sequence before sending later steps", async () => {
const sent: RecordStepMessage[] = [];
const send = vi.fn(async (message: RecordStepMessage) => {
sent.push(message);
if (sent.length === 1) throw new Error("service worker unavailable");
return { ok: true, sequence: message.sequence };
});
const delivery = new RecordStepDelivery("rec-ordered", send, "document-1");
delivery.enqueue({ op: "click", target: { tag: "button", name: "First" } });
delivery.enqueue({ op: "click", target: { tag: "button", name: "Second" } });
await expect(delivery.flush()).resolves.toBe(true);
expect(sent.map((message) => message.sequence)).toEqual([1, 1, 2]);
expect(sent.map((message) => message.step.target?.name)).toEqual(["First", "First", "Second"]);
});
it("keeps unacknowledged steps pending for a later flush", async () => {
const send = vi
.fn<(message: RecordStepMessage) => Promise<unknown>>()
.mockRejectedValueOnce(new Error("offline"))
.mockRejectedValueOnce(new Error("still offline"))
.mockImplementation(async (message) => ({ ok: true, sequence: message.sequence }));
const delivery = new RecordStepDelivery("rec-retry", send, "document-1");
delivery.enqueue({ op: "click", target: { tag: "button", name: "Save" } });
await expect(delivery.flush()).resolves.toBe(false);
await expect(delivery.flush()).resolves.toBe(true);
expect(send).toHaveBeenCalledTimes(3);
});
});
+76 -150
View File
@@ -1,9 +1,9 @@
import {
type CaptureTargetDescriptor,
describeEventTarget,
describeTarget,
resolveClickableElement,
resolveHoverElement,
type TargetDescriptor,
} from "@/lib/describe-target";
import {
evaluateHoverTrigger,
@@ -12,22 +12,8 @@ import {
hasDirectHoverInteractiveSignal,
hasStrongHoverExpansionSignal,
} from "@/lib/hover-trigger-policy";
import {
isRecordCancelMessage,
isRecordStartMessage,
isRecordStopMessage,
RECORD_CANCEL,
RECORD_START,
RECORD_STEP,
RECORD_STOP,
type RecordCancelMessage,
type RecordStartAck,
type RecordStartMessage,
type RecordStepPayload,
type RecordStopAck,
type RecordStopMessage,
} from "@/lib/record-bridge";
import { shouldRecordPress } from "@/lib/trace-reducer";
import type { RecordStepPayload } from "@/lib/record-bridge";
import { shouldRecordPress } from "@/lib/recording/draft-policy";
import {
closestHoverSurfaceCandidate,
collectHoverSurfaceStates,
@@ -37,42 +23,21 @@ import {
isLikelyHoverSurfaceOwner,
} from "./record-hover-surface";
const pendingStepSends = new Map<string, Set<Promise<boolean>>>();
const failedStepDeliveries = new Set<string>();
const knownRecordRequests = new Set<string>();
function sendRecordStep(requestId: string, step: RecordStepPayload): void {
const payload = { type: RECORD_STEP, requestId, step };
const pending = Promise.resolve(chrome.runtime.sendMessage(payload)).then(
() => true,
() => {
failedStepDeliveries.add(requestId);
return false;
},
);
const sends = pendingStepSends.get(requestId) ?? new Set<Promise<boolean>>();
sends.add(pending);
pendingStepSends.set(requestId, sends);
void pending.then(() => {
sends.delete(pending);
if (sends.size === 0) pendingStepSends.delete(requestId);
});
}
export interface RecordCaptureController {
dispose(): void;
}
interface FillSession {
element: FillableElement;
target: TargetDescriptor;
target: CaptureTargetDescriptor;
baselineValue: string;
lastValue: string;
pendingCommit?: "enter" | "suggestion" | "blur";
}
interface HoverCandidate {
element: Element;
target: TargetDescriptor;
target: CaptureTargetDescriptor;
recordedAt: number;
score: number;
eligible: boolean;
@@ -157,6 +122,24 @@ function nearbyFillableFromSearchChrome(target: Element): FillableElement | null
return null;
}
function captureGeometry(el: Element): RecordStepPayload["geometry"] {
const rect = el.getBoundingClientRect();
return {
rect: { x: rect.x, y: rect.y, w: rect.width, h: rect.height },
tag: el.tagName.toLowerCase(),
};
}
function geometryForEventTarget(
target: EventTarget | null,
): RecordStepPayload["geometry"] | undefined {
if (!(target instanceof Element)) return undefined;
const clickable = target.closest(
'a,button,input,select,textarea,[role="button"],[role="link"],[role="menuitem"],[contenteditable="true"]',
);
return captureGeometry(clickable ?? target);
}
function fillableValue(el: FillableElement): string {
if (el instanceof HTMLInputElement || el instanceof HTMLTextAreaElement) {
return el.value;
@@ -232,7 +215,10 @@ function collectHoverTriggerLabelText(root: Element): string {
return normalizeLabelText(text);
}
function compactHoverTargetName(el: Element, desc: TargetDescriptor): TargetDescriptor {
function compactHoverTargetName(
el: Element,
desc: CaptureTargetDescriptor,
): CaptureTargetDescriptor {
if (!desc.name) return desc;
const fullText = normalizeLabelText(el.textContent ?? "");
const compactName = collectHoverTriggerLabelText(el);
@@ -250,7 +236,7 @@ function compactHoverTargetName(el: Element, desc: TargetDescriptor): TargetDesc
return desc;
}
function isWeakHoverTarget(target: TargetDescriptor): boolean {
function isWeakHoverTarget(target: CaptureTargetDescriptor): boolean {
return !target.role && !target.name && target.tag === "div";
}
@@ -261,9 +247,9 @@ function looksLikeAvatarElement(el: Element): boolean {
function normalizeHoverTarget(
el: Element,
desc: TargetDescriptor,
desc: CaptureTargetDescriptor,
decision: HoverTriggerDecision,
): TargetDescriptor {
): CaptureTargetDescriptor {
if (desc.role === "img" && !desc.name && looksLikeAvatarElement(el)) {
return { ...desc, name: "image" };
}
@@ -277,7 +263,7 @@ function normalizeHoverTarget(
return desc;
}
function hoverTriggerSignals(el: Element, desc: TargetDescriptor) {
function hoverTriggerSignals(el: Element, desc: CaptureTargetDescriptor) {
if (!(el instanceof HTMLElement)) return null;
const style = hoverTriggerStyle(el);
return {
@@ -373,6 +359,7 @@ function scheduleInputCompletionCommit(
export function startRecordCapture(
_requestId: string,
sendStep: (step: RecordStepPayload) => void,
options: { captureNavigation?: boolean } = {},
): RecordCaptureController {
const emitStep = (step: RecordStepPayload) => {
sendStep({ page_url: location.href, ...step });
@@ -411,14 +398,17 @@ export function startRecordCapture(
emitStep({
op: "fill",
target: session.target,
geometry: captureGeometry(session.element),
value,
commit: session.pendingCommit ?? "blur",
...(isPassword ? { redacted: true } : {}),
});
};
const commitFillSession = () => {
const commitFillSession = (commit: "enter" | "suggestion" | "blur" = "blur") => {
if (!fillSession || composing) return;
const session = fillSession;
session.pendingCommit = commit;
fillSession = null;
emitFill(session);
committedValues.set(session.element, session.lastValue);
@@ -624,6 +614,7 @@ export function startRecordCapture(
emitStep({
op: "hover",
target: hover.target,
geometry: captureGeometry(hover.element),
});
emittedHoverElements.add(hover.element);
};
@@ -678,6 +669,7 @@ export function startRecordCapture(
emitStep({
op: "click",
target,
geometry: geometryForEventTarget(eventTarget(event)),
expects_navigation: true,
});
};
@@ -779,7 +771,7 @@ export function startRecordCapture(
scheduleInputCompletionCommit(
sessionElement,
syncFillSessionValue,
commitFillSession,
() => commitFillSession("suggestion"),
(el) => fillSession?.element === el,
);
return;
@@ -839,6 +831,7 @@ export function startRecordCapture(
emitStep({
op: "select",
target: desc,
geometry: captureGeometry(target),
values,
labels,
expects_navigation: true,
@@ -879,7 +872,7 @@ export function startRecordCapture(
};
}
if (fillable) {
commitFillSession();
commitFillSession(event.key === "Enter" ? "enter" : "blur");
}
const desc = describeEventTarget(target);
if (!desc && !event.key) return;
@@ -887,7 +880,7 @@ export function startRecordCapture(
emitStep({
op: "press",
key: event.key,
...(desc ? { target: desc } : {}),
...(desc ? { target: desc, geometry: geometryForEventTarget(target) } : {}),
...(modifiers.length ? { modifiers } : {}),
expects_navigation: event.key === "Enter",
});
@@ -903,23 +896,33 @@ export function startRecordCapture(
document.addEventListener("change", onChange, true);
document.addEventListener("keydown", onKeyDown, true);
const urlObserver = new MutationObserver(() => emitNavigateIfChanged());
urlObserver.observe(document, { subtree: true, childList: true });
const captureNavigation = options.captureNavigation ?? true;
const urlObserver = captureNavigation
? new MutationObserver(() => emitNavigateIfChanged())
: null;
urlObserver?.observe(document, { subtree: true, childList: true });
const onUrlEvent = () => emitNavigateIfChanged();
window.addEventListener("hashchange", onUrlEvent);
window.addEventListener("popstate", onUrlEvent);
window.addEventListener("pagehide", commitFillSession);
// Wrap: passing commitFillSession directly would forward the DOM event as
// the `commit` argument and stamp it onto the recorded fill step.
const onPageHide = () => commitFillSession();
if (captureNavigation) {
window.addEventListener("hashchange", onUrlEvent);
window.addEventListener("popstate", onUrlEvent);
}
window.addEventListener("pagehide", onPageHide);
const originalPushState = history.pushState;
const originalReplaceState = history.replaceState;
history.pushState = function (...args: Parameters<History["pushState"]>) {
originalPushState.apply(this, args);
emitNavigateIfChanged(navigationActionPending ? true : undefined);
};
history.replaceState = function (...args: Parameters<History["replaceState"]>) {
originalReplaceState.apply(this, args);
emitNavigateIfChanged(navigationActionPending ? true : undefined);
};
if (captureNavigation) {
history.pushState = function (...args: Parameters<History["pushState"]>) {
originalPushState.apply(this, args);
emitNavigateIfChanged(navigationActionPending ? true : undefined);
};
history.replaceState = function (...args: Parameters<History["replaceState"]>) {
originalReplaceState.apply(this, args);
emitNavigateIfChanged(navigationActionPending ? true : undefined);
};
}
return {
dispose() {
@@ -933,93 +936,16 @@ export function startRecordCapture(
document.removeEventListener("compositionend", onCompositionEnd, true);
document.removeEventListener("change", onChange, true);
document.removeEventListener("keydown", onKeyDown, true);
urlObserver.disconnect();
window.removeEventListener("hashchange", onUrlEvent);
window.removeEventListener("popstate", onUrlEvent);
window.removeEventListener("pagehide", commitFillSession);
history.pushState = originalPushState;
history.replaceState = originalReplaceState;
urlObserver?.disconnect();
if (captureNavigation) {
window.removeEventListener("hashchange", onUrlEvent);
window.removeEventListener("popstate", onUrlEvent);
}
window.removeEventListener("pagehide", onPageHide);
if (captureNavigation) {
history.pushState = originalPushState;
history.replaceState = originalReplaceState;
}
},
};
}
export type RecordContentMessage = RecordStartMessage | RecordStopMessage | RecordCancelMessage;
export function isRecordContentMessage(msg: unknown): msg is RecordContentMessage {
return isRecordStartMessage(msg) || isRecordStopMessage(msg) || isRecordCancelMessage(msg);
}
export function handleRecordContentMessage(
message: RecordContentMessage,
state: {
activeRequestId: string | null;
capture: RecordCaptureController | null;
setActiveRequestId(id: string | null): void;
setCapture(capture: RecordCaptureController | null): void;
onStart(requestId: string, startedAtMs?: number): void;
onStop(): void;
},
sendResponse?: (response: RecordStartAck | RecordStopAck) => void,
): boolean {
if (isRecordStartMessage(message)) {
if (!knownRecordRequests.has(message.requestId)) {
knownRecordRequests.add(message.requestId);
failedStepDeliveries.delete(message.requestId);
}
state.capture?.dispose();
state.setCapture(
startRecordCapture(message.requestId, (step) => {
sendRecordStep(message.requestId, step);
}),
);
state.setActiveRequestId(message.requestId);
state.onStart(message.requestId, message.startedAtMs);
sendResponse?.({ ok: true });
return sendResponse !== undefined;
}
if (isRecordStopMessage(message) || isRecordCancelMessage(message)) {
// Require an active recording that matches this requestId — otherwise a
// stray STOP/CANCEL (e.g. after teardown) would still run finishStop and
// clear overlay state even though nothing was capturing.
if (!state.activeRequestId || state.activeRequestId !== message.requestId) {
return false;
}
state.capture?.dispose();
state.setCapture(null);
const finishStop = () => {
state.onStop();
state.setActiveRequestId(null);
};
if (isRecordStopMessage(message) && sendResponse) {
const pending = [...(pendingStepSends.get(message.requestId) ?? [])];
void Promise.all(pending).then((delivered) => {
finishStop();
const succeeded = delivered.every(Boolean) && !failedStepDeliveries.has(message.requestId);
if (succeeded) {
failedStepDeliveries.delete(message.requestId);
knownRecordRequests.delete(message.requestId);
}
sendResponse(
succeeded
? { ok: true }
: {
ok: false,
error: "failed to deliver one or more recorded steps",
},
);
});
return true;
}
if (isRecordCancelMessage(message)) {
failedStepDeliveries.delete(message.requestId);
knownRecordRequests.delete(message.requestId);
}
finishStop();
return false;
}
return false;
}
export { RECORD_CANCEL, RECORD_START, RECORD_STEP, RECORD_STOP };
@@ -0,0 +1,65 @@
import {
isAcceptedRecordStepAck,
RECORD_STEP,
type RecordStepMessage,
type RecordStepPayload,
} from "@/lib/record-bridge";
export type SendRecordStepMessage = (message: RecordStepMessage) => Promise<unknown>;
export class RecordStepDelivery {
readonly #requestId: string;
readonly #producerId: string;
readonly #send: SendRecordStepMessage;
readonly #pending = new Map<number, RecordStepPayload>();
#nextSequence = 1;
#tail = Promise.resolve();
constructor(
requestId: string,
send: SendRecordStepMessage = (message) => chrome.runtime.sendMessage(message),
producerId: string = crypto.randomUUID(),
) {
this.#requestId = requestId;
this.#producerId = producerId;
this.#send = send;
}
enqueue(step: RecordStepPayload): void {
const sequence = this.#nextSequence;
this.#nextSequence += 1;
this.#pending.set(sequence, step);
void this.#schedule();
}
async flush(): Promise<boolean> {
await this.#tail;
await this.#schedule();
return this.#pending.size === 0;
}
#schedule(): Promise<void> {
const drain = () => this.#drain();
this.#tail = this.#tail.then(drain, drain);
return this.#tail;
}
async #drain(): Promise<void> {
for (const [sequence, step] of this.#pending) {
const message: RecordStepMessage = {
type: RECORD_STEP,
requestId: this.#requestId,
producerId: this.#producerId,
sequence,
step,
};
try {
const response = await this.#send(message);
if (!isAcceptedRecordStepAck(response, sequence)) return;
this.#pending.delete(sequence);
} catch {
return;
}
}
}
}
@@ -0,0 +1,209 @@
import {
markRecordingDocument,
type RecordingDocumentMarker,
waitForDocumentElement,
} from "@/lib/recording/document-marker";
import {
isRecordFrameStartMessage,
RECORD_FRAME_PORT,
RECORD_FRAME_QUERY,
RECORD_FRAME_START,
type RecordFramePortMessage,
type RecordFrameQueryResponse,
type RecordFrameStartMessage,
} from "@/lib/recording/frame-bridge";
import { type RecordCaptureController, startRecordCapture } from "../record-capture";
import { RecordStepDelivery } from "../record-step-delivery";
interface ActiveFrameRecording {
requestId: string;
producerId: string;
port: chrome.runtime.Port;
delivery: RecordStepDelivery;
capture: RecordCaptureController;
marker: RecordingDocumentMarker;
}
const RECORD_FRAME_REGISTER_TIMEOUT_MS = 5_000;
export class RecordFrameAgent {
#active: ActiveFrameRecording | null = null;
#startPromise: Promise<{ ok: true } | { ok: false; error: string }> | null = null;
#disposed = false;
async start(
message: RecordFrameStartMessage,
): Promise<{ ok: true } | { ok: false; error: string }> {
if (this.#active?.requestId === message.requestId) return { ok: true };
if (this.#disposed) return { ok: false, error: "recording document is disposed" };
if (this.#startPromise) {
await this.#startPromise;
return this.start(message);
}
if (this.#active) this.cancel(this.#active.requestId);
const starting = this.#start(message);
this.#startPromise = starting;
try {
return await starting;
} finally {
if (this.#startPromise === starting) this.#startPromise = null;
}
}
cancel(requestId: string): void {
const active = this.#active;
if (!active || active.requestId !== requestId) return;
active.capture.dispose();
active.marker.restore();
active.port.disconnect();
this.#active = null;
}
dispose(): void {
this.#disposed = true;
if (this.#active) this.cancel(this.#active.requestId);
}
async #start(
message: RecordFrameStartMessage,
): Promise<{ ok: true } | { ok: false; error: string }> {
const producerId = crypto.randomUUID();
const root = await waitForDocumentElement();
if (this.#disposed) return { ok: false, error: "recording document is disposed" };
const marker = markRecordingDocument(producerId, root);
const port = chrome.runtime.connect({ name: RECORD_FRAME_PORT });
try {
await registerPort(port, message.requestId, producerId);
} catch {
marker.restore();
port.disconnect();
return { ok: false, error: "failed to register recording document" };
}
if (this.#disposed) {
marker.restore();
port.disconnect();
return { ok: false, error: "recording document is disposed" };
}
const delivery = new RecordStepDelivery(message.requestId, undefined, producerId);
const capture = startRecordCapture(
message.requestId,
(step) => {
marker.ensure();
delivery.enqueue(step);
},
{ captureNavigation: window.top === window },
);
const active: ActiveFrameRecording = {
requestId: message.requestId,
producerId,
port,
delivery,
capture,
marker,
};
this.#active = active;
port.onMessage.addListener((raw: unknown) => {
const command = raw as Partial<RecordFramePortMessage>;
if (command.type === "stop" && command.requestId === active.requestId) {
void this.#stop(active, command.commandId);
} else if (command.type === "cancel" && command.requestId === active.requestId) {
this.cancel(active.requestId);
}
});
port.onDisconnect.addListener(() => {
if (this.#active !== active) return;
active.capture.dispose();
active.marker.restore();
this.#active = null;
});
return { ok: true };
}
async #stop(active: ActiveFrameRecording, commandId: string | undefined): Promise<void> {
if (!commandId || this.#active !== active) return;
active.capture.dispose();
const succeeded = await active.delivery.flush();
try {
active.port.postMessage({
type: "stopped",
requestId: active.requestId,
commandId,
ok: succeeded,
...(succeeded ? {} : { error: "failed to deliver one or more recorded steps" }),
} satisfies RecordFramePortMessage);
} catch {
return;
}
if (!succeeded || this.#active !== active) return;
active.marker.restore();
this.#active = null;
active.port.disconnect();
}
}
function registerPort(
port: chrome.runtime.Port,
requestId: string,
producerId: string,
): Promise<void> {
return new Promise((resolve, reject) => {
const timer = setTimeout(() => {
cleanup();
reject(new Error("recording document registration timed out"));
}, RECORD_FRAME_REGISTER_TIMEOUT_MS);
const cleanup = () => {
clearTimeout(timer);
port.onMessage.removeListener(onMessage);
port.onDisconnect.removeListener(onDisconnect);
};
const onMessage = (raw: unknown) => {
const message = raw as Partial<RecordFramePortMessage>;
if (
message.type !== "ready_ack" ||
message.requestId !== requestId ||
message.producerId !== producerId
) {
return;
}
cleanup();
resolve();
};
const onDisconnect = () => {
cleanup();
reject(new Error("recording document port disconnected"));
};
port.onMessage.addListener(onMessage);
port.onDisconnect.addListener(onDisconnect);
port.postMessage({ type: "ready", requestId, producerId } satisfies RecordFramePortMessage);
});
}
export function attachRecordFrameAgent(): () => void {
const agent = new RecordFrameAgent();
const onMessage = (
message: unknown,
_sender: chrome.runtime.MessageSender,
sendResponse: (response: { ok: true } | { ok: false; error: string }) => void,
) => {
if (!isRecordFrameStartMessage(message)) return false;
void agent.start(message).then(sendResponse);
return true;
};
chrome.runtime.onMessage.addListener(onMessage);
void chrome.runtime
.sendMessage({ type: RECORD_FRAME_QUERY })
.then((response: RecordFrameQueryResponse | undefined) => {
if (!response?.active || !response.requestId || response.startedAtMs === undefined) return;
return agent.start({
type: RECORD_FRAME_START,
requestId: response.requestId,
startedAtMs: response.startedAtMs,
});
})
.catch(() => {});
return () => {
chrome.runtime.onMessage.removeListener(onMessage);
agent.dispose();
};
}
+25 -21
View File
@@ -21,6 +21,7 @@ import {
type OverlayMode,
} from "@/lib/overlay-bridge";
import { POPUP_PORT_NAME, type PopupInbound, type PopupOutbound } from "@/lib/popup-bridge";
import { recordFrameCoordinator } from "@/lib/recording/frame-coordinator";
import { attachSessionsLiveFlag } from "@/lib/sessions-live-flag";
import { createDisconnectCleanup } from "@/session-manager/disconnect-cleanup";
import { attachSessionEventHandler } from "@/session-manager/event-handler";
@@ -38,6 +39,7 @@ import {
attachRecordFinishListener,
attachRecordQueryListener,
attachRecordStepListener,
type RecordRuntimeDeps,
} from "@/tools/record";
import { detectBrowserMeta } from "@/transport/handshake";
import type { Transport } from "@/transport/transport";
@@ -145,10 +147,33 @@ export default defineBackground(() => {
void sessionsLive.syncFromManager();
},
});
const recordDeps = {
tabsApi: chrome.tabs,
cdp,
frameCoordinator: recordFrameCoordinator,
sendToTab: (tabId: number, msg: Parameters<typeof chrome.tabs.sendMessage>[1]) =>
chrome.tabs.sendMessage(tabId, msg),
bypassOverlay: async (tabId: number, enabled: boolean) => {
try {
await chrome.tabs.sendMessage(tabId, {
type: OVERLAY_AUTOMATION_BYPASS,
enabled,
});
} catch {
// Content script may be unavailable on restricted pages.
}
},
} satisfies RecordRuntimeDeps;
recordFrameCoordinator.attach();
attachRecordStepListener(recordDeps);
attachRecordFinishListener(recordDeps);
attachRecordQueryListener(recordDeps);
const dispatcher = new ToolDispatcher({
transport,
sessions,
cdp,
recording: recordDeps,
onSessionsChanged: onOverlaySessionStateChanged,
onBrowserControlResumed,
approveBorrow: (ctx) =>
@@ -170,27 +195,6 @@ export default defineBackground(() => {
}),
});
dispatcher.start();
const recordDeps = {
tabsApi: chrome.tabs,
sendToTab: (tabId: number, msg: Parameters<typeof chrome.tabs.sendMessage>[1]) =>
chrome.tabs.sendMessage(tabId, msg),
bypassOverlay: async (tabId: number, enabled: boolean) => {
try {
await chrome.tabs.sendMessage(tabId, {
type: OVERLAY_AUTOMATION_BYPASS,
enabled,
});
} catch {
// Content script may be unavailable on restricted pages.
}
},
};
// Message listeners stay up (cheap; fire only for record message types).
// Tab / webNavigation observation attaches lazily while a recording is
// active — see ensureBrowserObservationListeners in tools/record.ts.
attachRecordStepListener();
attachRecordFinishListener(recordDeps);
attachRecordQueryListener(recordDeps);
if (typeof chrome.notifications?.onClicked?.addListener === "function") {
attachBorrowNotificationClickHandler({
onClicked: chrome.notifications.onClicked,
+34 -45
View File
@@ -11,11 +11,6 @@ import { createHelpRequestData } from "@/content/help-request";
import overlayCss from "@/content/overlay.css?inline";
import { OverlayController, shouldShowAgentControlOverlay } from "@/content/overlay-controller";
import { RecordOverlay } from "@/content/RecordOverlay";
import {
handleRecordContentMessage,
isRecordContentMessage,
type RecordCaptureController,
} from "@/content/record-capture";
import {
type CaptureSuppressAck,
type CaptureSuppressMessage,
@@ -45,11 +40,17 @@ import {
} from "@/lib/overlay-bridge";
import { sendInterrupt } from "@/lib/overlay-interrupt-client";
import {
isRecordCancelMessage,
isRecordStartMessage,
isRecordStopMessage,
RECORD_FINISH,
RECORD_QUERY,
type RecordCancelMessage,
type RecordQueryResponse,
type RecordStartAck,
type RecordStartMessage,
type RecordStopAck,
type RecordStopMessage,
} from "@/lib/record-bridge";
import type {
BorrowCancelMessage,
@@ -69,7 +70,6 @@ export default defineContentScript({
if (window.top !== window) return;
const overlays = new OverlayController();
let recordCapture: RecordCaptureController | null = null;
let activeRecordRequestId: string | null = null;
let reactRoot: ReactDOM.Root | null = null;
let overlayHost: HTMLElement | null = null;
@@ -228,8 +228,6 @@ export default defineContentScript({
if (previousHelp) {
void sendHelpFinish(previousHelp.id, "cancelled");
}
recordCapture?.dispose();
recordCapture = null;
activeRecordRequestId = null;
renderAll();
}
@@ -259,6 +257,9 @@ export default defineContentScript({
| HelpRequestMessage
| HelpCancelMessage
| CaptureSuppressMessage
| RecordStartMessage
| RecordStopMessage
| RecordCancelMessage
| OverlayAgentOverlayResetMessage
| OverlayAgentStateMessage
| OverlayAutomationBypassMessage,
@@ -269,41 +270,32 @@ export default defineContentScript({
return captureSuppress.handleMessage(message, sendResponse);
}
if (isRecordContentMessage(message)) {
const needsAsync = handleRecordContentMessage(
message,
{
activeRequestId: activeRecordRequestId,
capture: recordCapture,
setActiveRequestId: (id) => {
activeRecordRequestId = id;
},
setCapture: (capture) => {
recordCapture = capture;
},
onStart: (requestId, startedAtMs) => {
overlays.setAgentRecordRequest({
id: requestId,
...(typeof startedAtMs === "number" ? { startedAtMs } : {}),
onFinish: () => {
void chrome.runtime.sendMessage({
type: RECORD_FINISH,
requestId,
});
},
});
renderAll();
},
onStop: () => {
overlays.clearAgentRecordRequest(activeRecordRequestId ?? undefined);
renderAll();
},
if (isRecordStartMessage(message)) {
activeRecordRequestId = message.requestId;
overlays.setAgentRecordRequest({
id: message.requestId,
...(typeof message.startedAtMs === "number" ? { startedAtMs: message.startedAtMs } : {}),
onFinish: () => {
void chrome.runtime.sendMessage({
type: RECORD_FINISH,
requestId: message.requestId,
});
},
sendResponse as unknown as
| ((response: RecordStartAck | RecordStopAck) => void)
| undefined,
);
return needsAsync;
});
renderAll();
(sendResponse as unknown as (response: RecordStartAck) => void)({ ok: true });
return false;
}
if (isRecordStopMessage(message) || isRecordCancelMessage(message)) {
if (activeRecordRequestId !== message.requestId) return false;
overlays.clearAgentRecordRequest(message.requestId);
activeRecordRequestId = null;
renderAll();
if (isRecordStopMessage(message)) {
(sendResponse as unknown as (response: RecordStopAck) => void)({ ok: true });
}
return false;
}
if (
@@ -526,9 +518,6 @@ export default defineContentScript({
chrome.runtime.onMessage.removeListener(onMessage);
chrome.storage.onChanged.removeListener(onStorageChange);
window.removeEventListener("pageshow", onPageShow);
// Restore history hooks / remove capture listeners before the CS unloads.
recordCapture?.dispose();
recordCapture = null;
activeRecordRequestId = null;
});
},
@@ -122,8 +122,8 @@ describe("App", () => {
expect(screen.queryByText(/^扩展 v/)).toBeNull();
expect(screen.queryByText(/^daemon v/)).toBeNull();
expect(screen.getByTitle("扩展版本").textContent).toBe(EXTENSION_VERSION);
expect(screen.getByTitle("bsk 版本").textContent).toBe(mockDaemonVersion);
expect(screen.getByTitle("扩展版本").textContent).toBe(`Ext ${EXTENSION_VERSION}`);
expect(screen.getByTitle("bsk 版本").textContent).toBe(`CLI ${mockDaemonVersion}`);
expect(screen.getByText("03c3e47f")).toBeTruthy();
const copyButton = screen.getByRole("button", { name: "复制实例 ID" });
@@ -200,6 +200,9 @@ describe("App", () => {
expect(copied).toContain("--browser 03c3e47f");
expect(copied).toContain('--purpose "发布 wiki 文档"');
expect(copied).not.toMatch(/bsk record start[^\n]*--url/);
expect(copied).toContain("./trace");
expect(copied).toContain("trace.json");
expect(copied).toContain("states/");
// Button label stays static; a transient toast confirms the copy.
expect(copyButton.textContent).toContain("复制录制指令");
await waitFor(() => expect(screen.getByRole("status").textContent).toContain("已复制"));
@@ -244,6 +247,41 @@ describe("App", () => {
expect(copyButton.getAttribute("disabled")).not.toBeNull();
expect(screen.getByText("连接后可用")).toBeTruthy();
});
it("treats protocol drift as still connected and prompts an upgrade", async () => {
mockUseConnectionState.mockReturnValue({
snapshot: {
...baseSnapshot,
state: "version_skew",
instanceId: "03c3e47f",
handshake: {
server: "bh",
version: mockDaemonVersion,
protocol_version: "1.0",
},
},
statusState: "version_skew",
setLabel,
setConnectionEnabled,
});
render(<App />);
expect(screen.getByText("已连接")).toBeTruthy();
expect(screen.getByText("可升级")).toBeTruthy();
expect(screen.queryByText("协议不一致")).toBeNull();
expect(screen.queryByText("Action needed")).toBeNull();
expect(screen.queryByText("兼容")).toBeNull();
const warning = screen.getByText(/协议版本不同,请及时升级/);
expect(warning.textContent).toContain("CLI 协议");
expect(warning.textContent).toContain("扩展协议");
openRecordView();
const copyButton = screen.getByRole("button", { name: "复制录制指令" });
expect(copyButton.getAttribute("disabled")).toBeNull();
fireEvent.click(copyButton);
expect(navigator.clipboard.writeText).toHaveBeenCalledTimes(1);
});
});
describe("control hints toggle", () => {
+6 -5
View File
@@ -79,6 +79,7 @@ export function App() {
}, [copiedTick]);
const isSkewed = statusState === "version_skew";
const connectionLive = statusState === "connected" || isSkewed;
const daemonVersion = snapshot.handshake?.version ?? "—";
const daemonProtocol = snapshot.handshake?.protocol_version ?? "—";
const extensionVersion = snapshot.extensionVersion || "—";
@@ -90,7 +91,7 @@ export function App() {
setCopiedInstanceId(true);
};
const recordReady = statusState === "connected" && Boolean(snapshot.instanceId);
const recordReady = connectionLive && Boolean(snapshot.instanceId);
const recordPurpose = purposeDraft.trim();
const recordStartUrl = startUrlDraft.trim();
const recordCommand = snapshot.instanceId
@@ -206,12 +207,12 @@ export function App() {
</div>
{isSkewed && (
<p
className="mt-2 text-xs leading-snug text-amber-600 dark:text-amber-400"
className="mt-2 text-xs leading-snug text-muted-foreground"
data-slot="popup-version-skew-warning"
>
{t("popup.versionSkewWarning", {
extensionProtocol: PROTOCOL_VERSION,
daemonProtocol,
cliProtocol: daemonProtocol,
})}
</p>
)}
@@ -266,9 +267,9 @@ export function App() {
data-slot="popup-meta"
>
<div className="flex shrink-0 items-center gap-1">
<span title={t("popup.extensionVersionHint")}>{extensionVersion}</span>
<span title={t("popup.daemonVersionHint")}>CLI {daemonVersion}</span>
<span aria-hidden>/</span>
<span title={t("popup.daemonVersionHint")}>{daemonVersion}</span>
<span title={t("popup.extensionVersionHint")}>Ext {extensionVersion}</span>
</div>
<div className="flex min-w-0 items-center justify-end gap-1">
<span className="shrink-0">{t("popup.instanceTitle")}</span>
@@ -0,0 +1,14 @@
import { attachRecordFrameAgent } from "@/content/recording/frame-agent";
export default defineContentScript({
matches: ["<all_urls>"],
runAt: "document_start",
allFrames: true,
matchAboutBlank: true,
matchOriginAsFallback: true,
main(ctx) {
const dispose = attachRecordFrameAgent();
ctx.onInvalidated(dispose);
},
});
@@ -2,7 +2,7 @@ import { describe, expect, it, vi } from "vitest";
import {
CAPTURE_SUPPRESS,
type CaptureSuppressMessage,
withOverlaysHiddenForCapture,
withExtensionOverlayHidden,
} from "../capture-suppress-bridge";
function recordingSendToTab(events: string[]) {
@@ -12,12 +12,12 @@ function recordingSendToTab(events: string[]) {
});
}
describe("withOverlaysHiddenForCapture", () => {
describe("withExtensionOverlayHidden", () => {
it("sends begin → fn → end and returns fn's result", async () => {
const events: string[] = [];
const sendToTab = recordingSendToTab(events);
const result = await withOverlaysHiddenForCapture(
const result = await withExtensionOverlayHidden(
7,
async () => {
events.push("capture");
@@ -37,7 +37,7 @@ describe("withOverlaysHiddenForCapture", () => {
const sendToTab = recordingSendToTab(events);
await expect(
withOverlaysHiddenForCapture(
withExtensionOverlayHidden(
7,
async () => {
events.push("capture");
@@ -55,7 +55,7 @@ describe("withOverlaysHiddenForCapture", () => {
});
const fn = vi.fn(async () => "shot");
const result = await withOverlaysHiddenForCapture(7, fn, sendToTab);
const result = await withExtensionOverlayHidden(7, fn, sendToTab);
expect(result).toBe("shot");
expect(fn).toHaveBeenCalledTimes(1);
@@ -69,7 +69,7 @@ describe("withOverlaysHiddenForCapture", () => {
return { type: CAPTURE_SUPPRESS, ok: true };
});
const result = await withOverlaysHiddenForCapture(7, async () => "shot", sendToTab);
const result = await withExtensionOverlayHidden(7, async () => "shot", sendToTab);
expect(result).toBe("shot");
expect(sendToTab).toHaveBeenCalledTimes(2);
@@ -26,20 +26,20 @@ function handshake(
}
describe("computeConnectedState (protocol-based compat)", () => {
it("returns connected when protocol strings match", () => {
expect(computeConnectedState(handshake("1.0", "1.0"), MIN_COMPATIBLE_PROTOCOL)).toEqual({
it("returns connected when daemon protocol equals extension protocol", () => {
expect(computeConnectedState(handshake("1.1", "1.0"), MIN_COMPATIBLE_PROTOCOL)).toEqual({
kind: "connected",
});
});
it("returns version_skew when daemon protocol minor is newer", () => {
expect(computeConnectedState(handshake("1.1", "1.0"))).toEqual({
expect(computeConnectedState(handshake("1.2", "1.0"))).toEqual({
kind: "version_skew",
});
});
it("returns version_skew when daemon protocol string differs but floor is satisfied", () => {
expect(computeConnectedState(handshake("1", "1.0"))).toEqual({
expect(computeConnectedState(handshake("1.1.0", "1.0"))).toEqual({
kind: "version_skew",
});
});
@@ -53,7 +53,7 @@ describe("computeConnectedState (protocol-based compat)", () => {
});
it("rejects when extension is below daemon min_compatible_protocol", () => {
const result = computeConnectedState(handshake("1.0", "1.5"));
const result = computeConnectedState(handshake("1.1", "1.5"));
expect(result.kind).toBe("rejected");
if (result.kind === "rejected") {
expect(result.reason).toContain("min_compatible_protocol");
@@ -65,7 +65,7 @@ describe("computeConnectedState (protocol-based compat)", () => {
const result = computeConnectedState({
server: "browser-skill-daemon",
version: "0.1.0",
protocol_version: "1.0",
protocol_version: "1.1",
min_compatible_peer: "0.1.0",
});
expect(result).toEqual({ kind: "connected" });
@@ -80,7 +80,7 @@ describe("computeConnectedState (protocol-based compat)", () => {
});
it("rejects malformed daemon min_compatible_protocol with a daemon-floor reason", () => {
const result = computeConnectedState(handshake("1.0", "not-a-protocol"));
const result = computeConnectedState(handshake("1.1", "not-a-protocol"));
expect(result.kind).toBe("rejected");
if (result.kind === "rejected") {
expect(result.reason).toContain("daemon min_compatible_protocol");
@@ -242,11 +242,11 @@ describe("ConnectionController connectionEnabled", () => {
const second = transport.send.mock.calls[1]?.[0] as { id: string };
expect(second.id).not.toBe(first.id);
transport.emitMessage({ id: first.id, result: handshake("1.0", "1.0") });
transport.emitMessage({ id: first.id, result: handshake("1.1", "1.0") });
await Promise.resolve();
expect(controller.snapshot().state).not.toBe("connected");
transport.emitMessage({ id: second.id, result: handshake("1.0", "1.0") });
transport.emitMessage({ id: second.id, result: handshake("1.1", "1.0") });
await vi.waitFor(() => expect(controller.snapshot().state).toBe("connected"));
});
});
@@ -0,0 +1,101 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import type { CdpRunner } from "@/tools/shared";
import { waitForDocumentSettled } from "../recording/document-settle";
function quietCdp(): { cdp: CdpRunner; send: ReturnType<typeof vi.fn> } {
const send = vi.fn(async () => ({
result: { value: { idleMs: 1_000, readyState: "complete" } },
}));
return {
cdp: { send: send as unknown as CdpRunner["send"] },
send,
};
}
describe("waitForDocumentSettled", () => {
afterEach(() => {
vi.useRealTimers();
});
it("waits through the minimum observation floor before accepting a quiet page", async () => {
vi.useFakeTimers();
const { cdp, send } = quietCdp();
const settled = waitForDocumentSettled(cdp, { target: { tabId: 7 } });
await vi.advanceTimersByTimeAsync(180);
await expect(settled).resolves.toBe("quiet");
expect(send).toHaveBeenCalledTimes(3);
});
it("cancels before probing a superseded observation", async () => {
vi.useFakeTimers();
const { cdp, send } = quietCdp();
const controller = new AbortController();
const settled = waitForDocumentSettled(
cdp,
{ target: { tabId: 7 } },
{
signal: controller.signal,
},
);
controller.abort();
await vi.advanceTimersByTimeAsync(60);
await expect(settled).resolves.toBe("cancelled");
expect(send).not.toHaveBeenCalled();
});
it("probes a same-process iframe through its isolated execution context", async () => {
vi.useFakeTimers();
const send = vi.fn(async (_tabId: number, method: string) => {
if (method === "Page.createIsolatedWorld") return { executionContextId: 91 };
return { result: { value: { idleMs: 1_000, readyState: "complete" } } };
});
const cdp: CdpRunner = { send: send as unknown as CdpRunner["send"] };
const settled = waitForDocumentSettled(cdp, {
target: { tabId: 7 },
frameId: "child-frame",
});
await vi.advanceTimersByTimeAsync(180);
await expect(settled).resolves.toBe("quiet");
expect(send).toHaveBeenCalledWith(
7,
"Page.createIsolatedWorld",
expect.objectContaining({ frameId: "child-frame" }),
);
expect(send).toHaveBeenCalledWith(
7,
"Runtime.evaluate",
expect.objectContaining({ contextId: 91 }),
);
});
it("routes an OOPIF probe through its CDP target session", async () => {
vi.useFakeTimers();
const send = vi.fn();
const sendToTarget = vi.fn(async (_target, method: string) => {
if (method === "Page.createIsolatedWorld") return { executionContextId: 27 };
return { result: { value: { idleMs: 1_000, readyState: "complete" } } };
});
const cdp: CdpRunner = {
send: send as unknown as CdpRunner["send"],
sendToTarget: sendToTarget as unknown as NonNullable<CdpRunner["sendToTarget"]>,
};
const target = { tabId: 7, sessionId: "oopif-session" };
const settled = waitForDocumentSettled(cdp, { target, frameId: "oopif-frame" });
await vi.advanceTimersByTimeAsync(180);
await expect(settled).resolves.toBe("quiet");
expect(send).not.toHaveBeenCalled();
expect(sendToTarget).toHaveBeenCalledWith(
target,
"Runtime.evaluate",
expect.objectContaining({ contextId: 27 }),
);
});
});
@@ -0,0 +1,32 @@
import { describe, expect, it } from "vitest";
import {
isAcceptedRecordStepAck,
isRecordStepMessage,
RECORD_STEP,
type RecordStepMessage,
} from "../record-bridge";
const message: RecordStepMessage = {
type: RECORD_STEP,
requestId: "rec-1",
producerId: "document-1",
sequence: 1,
step: { op: "click", target: { tag: "button", name: "Submit" } },
};
describe("record step delivery protocol", () => {
it("requires a producer and a positive integer sequence", () => {
expect(isRecordStepMessage(message)).toBe(true);
expect(isRecordStepMessage({ ...message, producerId: "" })).toBe(false);
expect(isRecordStepMessage({ ...message, sequence: 0 })).toBe(false);
expect(isRecordStepMessage({ ...message, sequence: 1.5 })).toBe(false);
});
it("accepts only the acknowledgement for the sent sequence", () => {
expect(isAcceptedRecordStepAck({ ok: true, sequence: 2 }, 2)).toBe(true);
expect(isAcceptedRecordStepAck({ ok: true, sequence: 1 }, 2)).toBe(false);
expect(isAcceptedRecordStepAck({ ok: false, expectedSequence: 2, error: "gap" }, 2)).toBe(
false,
);
});
});
@@ -0,0 +1,213 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { RECORD_FRAME_PORT, RECORD_FRAME_QUERY } from "../recording/frame-bridge";
import { RecordFrameCoordinator } from "../recording/frame-coordinator";
class ListenerSet<T extends (...args: never[]) => unknown> {
readonly listeners = new Set<T>();
addListener = (listener: T) => this.listeners.add(listener);
removeListener = (listener: T) => this.listeners.delete(listener);
}
function fakePort(sender: chrome.runtime.MessageSender) {
const messages: unknown[] = [];
const onMessage = new ListenerSet<(message: unknown) => void>();
const onDisconnect = new ListenerSet<() => void>();
return {
port: {
name: RECORD_FRAME_PORT,
sender,
onMessage,
onDisconnect,
postMessage: vi.fn((message: unknown) => messages.push(message)),
disconnect: vi.fn(),
} as unknown as chrome.runtime.Port,
messages,
receive(message: unknown) {
for (const listener of onMessage.listeners) listener(message);
},
};
}
describe("RecordFrameCoordinator", () => {
const onConnect = new ListenerSet<(port: chrome.runtime.Port) => void>();
const onMessage = new ListenerSet<
(
message: unknown,
sender: chrome.runtime.MessageSender,
sendResponse: (response: unknown) => void,
) => boolean
>();
beforeEach(() => {
onConnect.listeners.clear();
onMessage.listeners.clear();
vi.stubGlobal("chrome", {
runtime: { onConnect, onMessage },
});
});
it("answers child-document queries for the initial tab before it is armed", () => {
const coordinator = new RecordFrameCoordinator({
getAllFrames: async () => [],
sendToDocument: async () => ({ ok: true }),
});
coordinator.attach();
coordinator.begin("rec-1", 10, 3);
const query = (tabId: number) => {
let response: unknown;
for (const listener of onMessage.listeners) {
listener(
{ type: RECORD_FRAME_QUERY },
{
tab: { id: tabId },
frameId: 7,
documentId: "child-document",
} as chrome.runtime.MessageSender,
(value) => {
response = value;
},
);
}
return response;
};
expect(query(3)).toEqual({ active: true, requestId: "rec-1", startedAtMs: 10 });
expect(query(4)).toEqual({ active: false });
});
it("starts a child Document that appears after the initial frame snapshot", async () => {
let notifyFrameNavigation:
| ((frame: {
tabId: number;
frameId: number;
documentId?: string;
lifecycle: "committed" | "completed";
}) => void)
| undefined;
const sendToDocument = vi.fn(async () => ({ ok: true }));
const coordinator = new RecordFrameCoordinator({
getAllFrames: async () => [{ frameId: 0, documentId: "top-document" }],
sendToDocument,
subscribeFrameNavigation(listener) {
notifyFrameNavigation = listener;
return () => {};
},
});
coordinator.attach();
const onDocumentReady = vi.fn();
coordinator.begin("rec-1", 10, 3, onDocumentReady);
await coordinator.armTab("rec-1", 3);
sendToDocument.mockClear();
notifyFrameNavigation?.({
tabId: 3,
frameId: 7,
documentId: "late-child",
lifecycle: "completed",
});
await vi.waitFor(() => {
expect(sendToDocument).toHaveBeenCalledWith(
3,
{ type: "bsk-record-frame-start", requestId: "rec-1", startedAtMs: 10 },
{ documentId: "late-child" },
);
});
expect(onDocumentReady).not.toHaveBeenCalled();
const child = fakePort({
tab: { id: 3 },
frameId: 7,
documentId: "late-child",
} as chrome.runtime.MessageSender);
for (const listener of onConnect.listeners) listener(child.port);
child.receive({ type: "ready", requestId: "rec-1", producerId: "producer-1" });
notifyFrameNavigation?.({
tabId: 3,
frameId: 7,
documentId: "late-child",
lifecycle: "completed",
});
await vi.waitFor(() => {
expect(onDocumentReady).toHaveBeenCalledWith({
tabId: 3,
documentId: "late-child",
browserFrameId: 7,
producerId: "producer-1",
});
});
});
it("binds a producer to its sender Document and keeps final steps valid while stopping", async () => {
const sendToDocument = vi.fn(async () => ({ ok: true }));
const coordinator = new RecordFrameCoordinator({
getAllFrames: async () => [
{ frameId: 0, documentId: "top-document" },
{ frameId: 7, documentId: "child-document" },
],
sendToDocument,
});
coordinator.attach();
coordinator.begin("rec-1", 10, 3);
await expect(coordinator.armTab("rec-1", 3)).resolves.toBe(true);
const sender = {
tab: { id: 3 },
frameId: 7,
documentId: "child-document",
} as chrome.runtime.MessageSender;
const frame = fakePort(sender);
for (const listener of onConnect.listeners) listener(frame.port);
frame.receive({ type: "ready", requestId: "rec-1", producerId: "producer-1" });
expect(coordinator.sourceFor("rec-1", "producer-1", sender)).toEqual({
tabId: 3,
documentId: "child-document",
browserFrameId: 7,
producerId: "producer-1",
});
const stopping = coordinator.stop("rec-1");
expect(coordinator.sourceFor("rec-1", "producer-1", sender)).not.toBeNull();
const stop = frame.messages.find(
(message): message is { type: "stop"; commandId: string } =>
typeof message === "object" &&
message !== null &&
(message as { type?: string }).type === "stop",
);
expect(stop).toBeDefined();
frame.receive({
type: "stopped",
requestId: "rec-1",
commandId: stop!.commandId,
ok: true,
});
await expect(stopping).resolves.toBe(true);
expect(coordinator.sourceFor("rec-1", "producer-1", sender)).toBeNull();
});
it("rejects a producer used from another Document", async () => {
const coordinator = new RecordFrameCoordinator({
getAllFrames: async () => [{ frameId: 0, documentId: "top-document" }],
sendToDocument: async () => ({ ok: true }),
});
coordinator.attach();
coordinator.begin("rec-1", 10, 3);
await coordinator.armTab("rec-1", 3);
const sender = {
tab: { id: 3 },
frameId: 0,
documentId: "top-document",
} as chrome.runtime.MessageSender;
const frame = fakePort(sender);
for (const listener of onConnect.listeners) listener(frame.port);
frame.receive({ type: "ready", requestId: "rec-1", producerId: "producer-1" });
expect(
coordinator.sourceFor("rec-1", "producer-1", {
...sender,
documentId: "other-document",
}),
).toBeNull();
});
});
@@ -0,0 +1,70 @@
import { describe, expect, it } from "vitest";
import {
RECORD_DOCUMENT_ATTRIBUTE,
recordingDocumentMarkerValue,
} from "@/shared/recording-document-identity";
import { markRecordingDocument } from "../recording/document-marker";
import { ObservationNodeIndex } from "../recording/observation-capture";
describe("recording document marker", () => {
it("rejects a non-random Document identity", () => {
expect(() => markRecordingDocument("user@example.com")).toThrow(
"recording Document identity must be a random UUID",
);
});
it("restores the Document attribute exactly", () => {
document.documentElement.setAttribute(RECORD_DOCUMENT_ATTRIBUTE, "page-value");
const producerId = "123e4567-e89b-42d3-a456-426614174000";
const marker = markRecordingDocument(producerId);
expect(document.documentElement.getAttribute(RECORD_DOCUMENT_ATTRIBUTE)).toBe(
recordingDocumentMarkerValue(producerId),
);
marker.restore();
expect(document.documentElement.getAttribute(RECORD_DOCUMENT_ATTRIBUTE)).toBe("page-value");
document.documentElement.removeAttribute(RECORD_DOCUMENT_ATTRIBUTE);
});
it("indexes the CDP scope from the safe frame identity", () => {
const index = new ObservationNodeIndex({
rootFrameId: "root",
frames: [
{
frameId: "child",
target: { tabId: 3, sessionId: "oopif-session" },
recordingDocumentId: "producer-1",
},
],
matchNodes: [],
refs: [],
});
expect(index.documentScope("producer-1")).toEqual({
frameId: "child",
target: { tabId: 3, sessionId: "oopif-session" },
});
});
it("fails closed when a recording identity maps to multiple Documents", () => {
const index = new ObservationNodeIndex({
rootFrameId: "root",
frames: [
{
frameId: "left",
target: { tabId: 3, sessionId: "left-session" },
recordingDocumentId: "duplicate",
},
{
frameId: "right",
target: { tabId: 3, sessionId: "right-session" },
recordingDocumentId: "duplicate",
},
],
matchNodes: [],
refs: [],
});
expect(index.documentScope("duplicate")).toBeUndefined();
});
});
@@ -0,0 +1,146 @@
import { describe, expect, it } from "vitest";
import { ObservationNodeIndex } from "../recording/observation-capture";
import { RecordingObservationSession } from "../recording/observation-session";
import { RecordingStateRegistry } from "../recording/state-registry";
import { buildTraceV3 } from "../recording/trace-builder-v3";
import type { RecordingDraftStep } from "../recording/types";
const URL = "https://example.com/login";
function sessionWithInput(redactValues = false): RecordingObservationSession {
const session = new RecordingObservationSession({ redactValues });
const state = session.registry.register({
url: URL,
vomText: '@vom 1\ntextbox "Password" value="••••••" [ref=e1]',
});
session.cursor.lastSettled = {
stateId: state.id,
rootFrameId: "root",
index: new ObservationNodeIndex({
rootFrameId: "root",
matchNodes: [
{
frameId: "root",
backendNodeId: 42,
tag: "input",
rect: { x: 20, y: 40, w: 200, h: 30 },
localRect: { x: 20, y: 40, w: 200, h: 30 },
},
],
refs: [{ ref: "e1", backendNodeId: 42, role: "textbox", name: "Password", line: 1 }],
}),
url: URL,
};
return session;
}
function finalizedFillTrace(value: string, redactValues: boolean) {
const session = sessionWithInput(redactValues);
const draft: RecordingDraftStep = {
op: "fill",
captureTarget: { tag: "input", role: "textbox", name: "Password" },
value,
targetHint: {
geometry: { rect: { x: 20, y: 40, w: 200, h: 30 }, tag: "input" },
},
};
session.bindDraft(draft, 1);
draft.postStateId = draft.preStateId;
return buildTraceV3({
registry: session.registry,
drafts: [draft],
annotations: session.annotations,
startedAt: "2026-08-12T00:00:00.000Z",
stoppedBy: "user_finish",
bskVersion: "test",
redactValues,
});
}
function finalizedFillBody(value: string, redactValues: boolean): string {
return finalizedFillTrace(value, redactValues).states[0]!.body;
}
describe("record observation annotations", () => {
it("omits a fill literal when values are redacted", () => {
const secret = "hunter2-private";
const dumped = JSON.stringify(finalizedFillTrace(secret, true));
expect(dumped).toContain("step 1: fill");
expect(dumped).not.toContain(secret);
});
it("keeps ordinary fill details", () => {
expect(finalizedFillBody("ordinary text", false)).toContain('step 1: fill: "ordinary text"');
});
it("encodes title and URL so line breaks cannot corrupt state metadata", () => {
const registry = new RecordingStateRegistry();
const state = registry.register({
url: "https://example.com/a\nb",
title: "hello\nworld",
vomText: "@vom 1",
});
const trace = buildTraceV3({
registry,
drafts: [{ op: "scroll", preStateId: state.id, postStateId: state.id }],
startedAt: "2026-08-12T00:00:00.000Z",
stoppedBy: "user_finish",
bskVersion: "test",
});
expect(trace.states[0]?.body).toContain('url: "https://example.com/a\\nb"');
expect(trace.states[0]?.body).toContain('title: "hello\\nworld"');
});
});
describe("recording state ownership", () => {
it("deduplicates within one recording and isolates ids between recordings", () => {
const first = new RecordingStateRegistry();
const second = new RecordingStateRegistry();
expect(first.register({ url: URL, vomText: "same" }).id).toBe("s1");
expect(first.register({ url: URL, vomText: "same" }).id).toBe("s1");
expect(second.register({ url: URL, vomText: "other" }).id).toBe("s1");
});
it("enriches metadata when a deduplicated observation becomes more complete", () => {
const registry = new RecordingStateRegistry();
registry.register({ url: URL, vomText: "same" });
const state = registry.register({
url: URL,
title: "Login",
vomText: "same",
truncated: true,
});
expect(state).toMatchObject({ id: "s1", title: "Login", truncated: true });
});
});
describe("draft binding", () => {
it("keeps capture semantics when no observation exists", () => {
const session = new RecordingObservationSession();
const draft: RecordingDraftStep = {
op: "hover",
captureTarget: { tag: "button", role: "button", name: "新建" },
};
session.bindDraft(draft, 1);
expect(draft.matchedTarget).toEqual({ role: "button", name: "新建", unmatched: true });
expect(draft.preStateId).toBeUndefined();
});
it("does not bind an unmatched new action to a stale observation", () => {
const session = sessionWithInput();
const draft: RecordingDraftStep = {
op: "click",
captureTarget: { tag: "button", role: "button", name: "Confirm" },
targetHint: {
geometry: { rect: { x: 400, y: 300, w: 80, h: 30 }, tag: "button" },
},
};
session.bindDraft(draft, 2, true);
expect(draft.preStateId).toBeUndefined();
expect("matchedTarget" in draft ? draft.matchedTarget : undefined).toEqual({
role: "button",
name: "Confirm",
unmatched: true,
});
});
});
@@ -0,0 +1,289 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import type { CdpRunner, ChromeTabsApi } from "@/tools/shared";
const captureRecordingObservation = vi.hoisted(() => vi.fn());
const waitForDocumentSettled = vi.hoisted(() => vi.fn(async () => "quiet" as const));
vi.mock("../recording/observation-capture", async (importOriginal) => {
const actual = await importOriginal<typeof import("../recording/observation-capture")>();
return { ...actual, captureRecordingObservation };
});
vi.mock("../recording/document-settle", async (importOriginal) => {
const actual = await importOriginal<typeof import("../recording/document-settle")>();
return { ...actual, waitForDocumentSettled };
});
import { ObservationNodeIndex } from "../recording/observation-capture";
import { RecordingObservationRuntime } from "../recording/recording-runtime";
import type { RecordingDraftStep } from "../recording/types";
function observation(url = "https://example.com/") {
return {
rootFrameId: "root",
index: new ObservationNodeIndex({ rootFrameId: "root", matchNodes: [], refs: [] }),
url,
title: "Example",
vomText: '@vom 1\nRootWebArea "Example"',
truncated: false,
};
}
function runtime(): RecordingObservationRuntime {
return new RecordingObservationRuntime({
cdp: { send: vi.fn() as unknown as CdpRunner["send"] },
tabsApi: { get: vi.fn(), query: vi.fn() } as unknown as ChromeTabsApi,
});
}
describe("RecordingObservationRuntime", () => {
afterEach(() => {
captureRecordingObservation.mockReset();
waitForDocumentSettled.mockClear();
});
it("shares one initial capture and includes it in flush", async () => {
let release!: (value: ReturnType<typeof observation>) => void;
captureRecordingObservation.mockReturnValueOnce(
new Promise((resolve) => {
release = resolve;
}),
);
const recording = runtime();
const first = recording.captureInitial(7);
const second = recording.captureInitial(7);
const flushed = recording.flush();
let flushCompleted = false;
void flushed.then(() => {
flushCompleted = true;
});
await Promise.resolve();
expect(captureRecordingObservation).toHaveBeenCalledTimes(1);
expect(flushCompleted).toBe(false);
release(observation());
await Promise.all([first, second, flushed]);
expect(flushCompleted).toBe(true);
});
it("cancels an in-flight initial capture", async () => {
captureRecordingObservation.mockImplementationOnce(
(input: { signal?: AbortSignal }) =>
new Promise((_, reject) => {
input.signal?.addEventListener(
"abort",
() => reject(new DOMException("observation aborted", "AbortError")),
{ once: true },
);
}),
);
const recording = runtime();
const pending = recording.captureInitial(7);
recording.cancel();
await expect(pending).rejects.toMatchObject({ name: "AbortError" });
});
it("retries a failed initial capture when the recording settles at stop", async () => {
captureRecordingObservation
.mockRejectedValueOnce(new Error("document swapped"))
.mockResolvedValueOnce(observation());
const recording = runtime();
await expect(recording.captureInitial(7)).rejects.toThrow("document swapped");
await recording.settleTrailing(7, []);
const trace = recording.buildTrace({
drafts: [],
startedAt: "2026-08-19T00:00:00.000Z",
stoppedBy: "user_finish",
bskVersion: "0.1.6",
});
expect(captureRecordingObservation).toHaveBeenCalledTimes(2);
expect(trace.states).toHaveLength(1);
});
it("keeps runtime matching data out of the serialized trace", async () => {
const captured = observation();
captured.index = new ObservationNodeIndex({
rootFrameId: "root",
matchNodes: [
{
frameId: "root",
backendNodeId: 42,
tag: "button",
rect: { x: 10, y: 20, w: 100, h: 30 },
localRect: { x: 10, y: 20, w: 100, h: 30 },
},
],
refs: [{ ref: "e1", backendNodeId: 42, role: "button", name: "Submit", line: 1 }],
});
captureRecordingObservation.mockResolvedValueOnce(captured);
const recording = runtime();
await recording.captureInitial(7);
const dumped = JSON.stringify(
recording.buildTrace({
drafts: [],
startedAt: "2026-08-19T00:00:00.000Z",
stoppedBy: "user_finish",
bskVersion: "0.1.6",
}),
);
expect(dumped).toContain("RootWebArea");
expect(dumped).not.toContain("matchNodes");
expect(dumped).not.toContain("localRect");
expect(dumped).not.toContain("backendNodeId");
expect(dumped).not.toContain("ObservationNodeIndex");
});
it("captures a tab transition without sharing observation cursors", async () => {
captureRecordingObservation
.mockResolvedValueOnce(observation("https://example.com/first"))
.mockResolvedValueOnce(observation("https://example.com/second"));
const recording = runtime();
await recording.captureInitial(4);
const transition = await recording.captureTabTransition(4, 5);
await recording.captureInitial(4);
await recording.captureInitial(5);
expect(transition).toEqual({
preStateId: "s1",
postStateId: "s2",
targetUrl: "https://example.com/second",
});
expect(captureRecordingObservation.mock.calls.map(([input]) => input.tabId)).toEqual([4, 5]);
});
it("binds an iframe draft to its marked CDP Document scope", async () => {
captureRecordingObservation.mockResolvedValueOnce({
...observation(),
index: new ObservationNodeIndex({
rootFrameId: "root",
frames: [
{
frameId: "child",
target: { tabId: 7, sessionId: "oopif-session" },
recordingDocumentId: "producer-1",
},
],
matchNodes: [
{
backendNodeId: 42,
frameId: "child",
tag: "button",
rect: { x: 410, y: 20, w: 100, h: 30 },
localRect: { x: 10, y: 20, w: 100, h: 30 },
},
],
refs: [{ ref: "e1", backendNodeId: 42, frameId: "child", line: 1 }],
}),
});
const recording = runtime();
const drafts: RecordingDraftStep[] = [
{
op: "click" as const,
captureTarget: { tag: "button", role: "button", name: "Save" },
targetHint: {
geometry: { rect: { x: 10, y: 20, w: 100, h: 30 }, tag: "button" },
},
},
];
await recording.captureInitial(7);
await recording.processDraft(7, drafts, 0, "producer-1");
recording.cancel();
const draft = drafts[0];
expect(draft?.op).toBe("click");
if (!draft || draft.op !== "click") throw new Error("expected click draft");
expect(draft.targetHint).toMatchObject({
frameId: "child",
geometrySpace: "local",
});
expect(draft.matchedTarget?.ref).toBe("e1");
});
it("refreshes a late iframe before matching its first action", async () => {
const iframeObservation = {
...observation(),
index: new ObservationNodeIndex({
rootFrameId: "root",
frames: [
{
frameId: "child",
target: { tabId: 7, sessionId: "oopif-session" },
recordingDocumentId: "producer-1",
},
],
matchNodes: [
{
backendNodeId: 42,
frameId: "child",
tag: "button",
rect: { x: 410, y: 20, w: 100, h: 30 },
localRect: { x: 10, y: 20, w: 100, h: 30 },
},
],
refs: [
{
ref: "e1",
backendNodeId: 42,
frameId: "child",
role: "button",
name: "表格视图",
line: 1,
},
],
}),
};
captureRecordingObservation
.mockResolvedValueOnce(observation())
.mockResolvedValueOnce(iframeObservation)
.mockResolvedValueOnce(iframeObservation);
const recording = runtime();
await recording.captureInitial(7);
await recording.refreshDocument(7, "producer-1");
const drafts: RecordingDraftStep[] = [
{
op: "click",
captureTarget: { tag: "button", role: "button", name: "表格视图" },
targetHint: {
geometry: { rect: { x: 10, y: 20, w: 100, h: 30 }, tag: "button" },
},
},
];
await recording.processDraft(7, drafts, 0, "producer-1");
recording.cancel();
expect(captureRecordingObservation).toHaveBeenCalledTimes(3);
expect(waitForDocumentSettled).toHaveBeenCalledWith(
expect.anything(),
{ frameId: "child", target: { tabId: 7, sessionId: "oopif-session" } },
{ signal: expect.any(AbortSignal) },
);
const draft = drafts[0];
expect(draft?.op).toBe("click");
if (!draft || draft.op !== "click") throw new Error("expected click draft");
expect(draft.matchedTarget).toMatchObject({ ref: "e1", name: "表格视图" });
});
it("does not poison final flush when a frame readiness refresh fails", async () => {
captureRecordingObservation
.mockResolvedValueOnce(observation())
.mockRejectedValueOnce(new Error("child Document replaced"));
const recording = runtime();
await recording.captureInitial(7);
await expect(recording.refreshDocument(7, "producer-1")).rejects.toThrow(
"child Document replaced",
);
await expect(recording.flush()).resolves.toBeUndefined();
});
});
@@ -1,80 +0,0 @@
import { describe, expect, it } from "vitest";
import { appendRecordedPayload, observeRecordedNavigation } from "../recording-step-buffer";
describe("recording-step-buffer", () => {
it("stores semantic click without summary", () => {
const buffer = { steps: [], pendingNavigation: false };
appendRecordedPayload(buffer, {
op: "click",
target: { tag: "button", role: "button", name: "发布" },
expects_navigation: true,
});
expect(buffer.steps).toEqual([
{
op: "click",
target: { tag: "button", role: "button", name: "发布" },
},
]);
expect(buffer.pendingNavigation).toBe(true);
});
it("annotates navigated_to on action-caused navigation instead of wait_for_navigation", () => {
const buffer = {
steps: [
{
op: "click" as const,
target: { tag: "button", role: "button", name: "发布" },
},
],
currentUrl: "https://example.com/a",
pendingNavigation: true,
pendingNavigationDeadline: Date.now() + 5_000,
};
observeRecordedNavigation(buffer, "https://example.com/b", true);
expect(buffer.steps).toEqual([
{
op: "click",
target: { tag: "button", role: "button", name: "发布" },
navigated_to: "https://example.com/b",
},
]);
expect(JSON.stringify(buffer.steps)).not.toContain("wait_for_navigation");
});
it("annotates navigated_to onto a select that auto-submits a navigation", () => {
const buffer = {
steps: [
{
op: "select" as const,
target: { tag: "select", role: "combobox", name: "分类" },
values: ["tech"],
},
],
currentUrl: "https://example.com/list",
pendingNavigation: true,
pendingNavigationDeadline: Date.now() + 5_000,
};
observeRecordedNavigation(buffer, "https://example.com/list?cat=tech", true);
expect(buffer.steps).toEqual([
{
op: "select",
target: { tag: "select", role: "combobox", name: "分类" },
values: ["tech"],
navigated_to: "https://example.com/list?cat=tech",
},
]);
});
it("emits navigate for uncaused URL changes", () => {
const buffer = {
steps: [],
currentUrl: "https://example.com/a",
pendingNavigation: false,
};
observeRecordedNavigation(buffer, "https://example.com/b", false);
expect(buffer.steps[0]).toMatchObject({
op: "navigate",
url: "https://example.com/b",
});
});
});
@@ -0,0 +1,136 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import type { CdpRunner, ChromeTabsApi } from "@/tools/shared";
import { ObservationNodeIndex, type RegisteredObservation } from "../recording/observation-capture";
import { RecordingObservationSession } from "../recording/observation-session";
import { SettleController } from "../recording/settle-controller";
import type { RecordingDraftStep } from "../recording/types";
const OBSERVATION: RegisteredObservation = {
stateId: "s-next",
rootFrameId: "root",
index: new ObservationNodeIndex({ rootFrameId: "root", matchNodes: [], refs: [] }),
url: "https://example.com/next",
};
describe("SettleController", () => {
afterEach(() => vi.useRealTimers());
it("aborts superseded capture work and lands the prior action on the next origin", async () => {
vi.useFakeTimers();
const cdp: CdpRunner = {
send: vi.fn(async () => ({
result: { value: { idleMs: 1_000, readyState: "complete" } },
})) as unknown as CdpRunner["send"],
};
const tabsApi: ChromeTabsApi = {
get: vi.fn(async () => ({ id: 7, url: OBSERVATION.url }) as chrome.tabs.Tab),
query: vi.fn(async () => []),
};
const session = new RecordingObservationSession();
let firstSignal: AbortSignal | undefined;
vi.spyOn(session, "capture")
.mockImplementationOnce(async (_cdp, _tabs, _tabId, signal) => {
firstSignal = signal;
return new Promise<RegisteredObservation>((_resolve, reject) => {
signal?.addEventListener(
"abort",
() => reject(new DOMException("observation aborted", "AbortError")),
{ once: true },
);
});
})
.mockResolvedValueOnce(OBSERVATION);
const drafts: RecordingDraftStep[] = [
{ op: "click", captureTarget: { tag: "button", name: "First" }, preStateId: "s1" },
];
const controller = new SettleController({ session, cdp, tabsApi, tabId: 7 });
controller.schedule(drafts, 0);
await vi.advanceTimersByTimeAsync(180);
drafts.push({
op: "click",
captureTarget: { tag: "button", name: "Second" },
preStateId: "s-next",
});
controller.schedule(drafts, 1);
expect(firstSignal?.aborted).toBe(true);
expect(drafts[0]?.postStateId).toBe("s-next");
await vi.advanceTimersByTimeAsync(500);
await controller.flush();
expect(drafts[1]?.postStateId).toBe("s-next");
});
it("does not consume a newer redirect while reading the prior landing URL", async () => {
vi.useFakeTimers();
const cdp: CdpRunner = {
send: vi.fn(async () => ({
result: { value: { idleMs: 1_000, readyState: "complete" } },
})) as unknown as CdpRunner["send"],
};
let resolveFirstTab!: (tab: chrome.tabs.Tab) => void;
const firstTab = new Promise<chrome.tabs.Tab>((resolve) => {
resolveFirstTab = resolve;
});
const tabsApi: ChromeTabsApi = {
get: vi
.fn()
.mockImplementationOnce(() => firstTab)
.mockResolvedValue({ id: 7, url: OBSERVATION.url }),
query: vi.fn(async () => []),
};
const session = new RecordingObservationSession();
session.cursor.lastSettled = OBSERVATION;
const drafts: RecordingDraftStep[] = [];
const controller = new SettleController({ session, cdp, tabsApi, tabId: 7 });
controller.scheduleRedirect(drafts, "https://example.com/intermediate");
await vi.advanceTimersByTimeAsync(180);
expect(tabsApi.get).toHaveBeenCalledTimes(1);
controller.scheduleRedirect(drafts, OBSERVATION.url);
resolveFirstTab({ id: 7, url: "https://example.com/intermediate" } as chrome.tabs.Tab);
await vi.advanceTimersByTimeAsync(500);
await controller.flushRedirects();
expect(drafts).toEqual([]);
expect(controller.hasPending).toBe(false);
});
it("settles an action in its own OOPIF document scope", async () => {
vi.useFakeTimers();
const send = vi.fn();
const sendToTarget = vi.fn(async (_target, method: string) => {
if (method === "Page.createIsolatedWorld") return { executionContextId: 39 };
return { result: { value: { idleMs: 1_000, readyState: "complete" } } };
});
const cdp: CdpRunner = {
send: send as unknown as CdpRunner["send"],
sendToTarget: sendToTarget as unknown as NonNullable<CdpRunner["sendToTarget"]>,
};
const tabsApi: ChromeTabsApi = {
get: vi.fn(async () => ({ id: 7, url: OBSERVATION.url }) as chrome.tabs.Tab),
query: vi.fn(async () => []),
};
const session = new RecordingObservationSession();
vi.spyOn(session, "capture").mockResolvedValue(OBSERVATION);
const drafts: RecordingDraftStep[] = [
{ op: "click", captureTarget: { tag: "button", name: "Inside frame" } },
];
const controller = new SettleController({ session, cdp, tabsApi, tabId: 7 });
const target = { tabId: 7, sessionId: "oopif-session" };
controller.schedule(drafts, 0, { target, frameId: "oopif-frame" });
await vi.advanceTimersByTimeAsync(180);
await controller.flush();
expect(send).not.toHaveBeenCalled();
expect(sendToTarget).toHaveBeenCalledWith(
target,
"Runtime.evaluate",
expect.objectContaining({ contextId: 39 }),
);
expect(drafts[0]?.postStateId).toBe("s-next");
});
});
@@ -0,0 +1,164 @@
import { describe, expect, it } from "vitest";
import { appendRecordedPayload, observeRecordedNavigation } from "../recording/step-buffer";
describe("recording-step-buffer", () => {
it("stores semantic click without summary", () => {
const buffer = { steps: [], navigation: { pendingNavigation: false } };
appendRecordedPayload(buffer, {
op: "click",
target: { tag: "button", role: "button", name: "发布" },
expects_navigation: true,
});
expect(buffer.steps).toEqual([
{
op: "click",
captureTarget: { tag: "button", role: "button", name: "发布" },
},
]);
expect(buffer.navigation.pendingNavigation).toBe(true);
});
it("keeps the hovered element description as a capture fallback", () => {
const buffer = { steps: [], navigation: { pendingNavigation: false } };
appendRecordedPayload(
buffer,
{
op: "hover",
target: { tag: "button", role: "button", name: "新建" },
geometry: {
rect: { x: 900, y: 8, w: 60, h: 32 },
tag: "button",
},
},
{
geometry: {
rect: { x: 900, y: 8, w: 60, h: 32 },
tag: "button",
},
},
);
expect(buffer.steps).toEqual([
{
op: "hover",
captureTarget: { tag: "button", role: "button", name: "新建" },
targetHint: {
geometry: {
rect: { x: 900, y: 8, w: 60, h: 32 },
tag: "button",
},
},
},
]);
});
it("annotates navigated_to on action-caused navigation instead of wait_for_navigation", () => {
const buffer = {
steps: [
{
op: "click" as const,
captureTarget: { tag: "button", role: "button", name: "发布" },
},
],
navigation: {
currentUrl: "https://example.com/a",
pendingNavigation: true,
pendingNavigationDeadline: Date.now() + 5_000,
},
};
observeRecordedNavigation(buffer, "https://example.com/b", true);
expect(buffer.steps).toEqual([
{
op: "click",
captureTarget: { tag: "button", role: "button", name: "发布" },
navigatedTo: "https://example.com/b",
},
]);
expect(JSON.stringify(buffer.steps)).not.toContain("wait_for_navigation");
});
it("annotates navigated_to onto a select that auto-submits a navigation", () => {
const buffer = {
steps: [
{
op: "select" as const,
captureTarget: { tag: "select", role: "combobox", name: "分类" },
values: ["tech"],
},
],
navigation: {
currentUrl: "https://example.com/list",
pendingNavigation: true,
pendingNavigationDeadline: Date.now() + 5_000,
},
};
observeRecordedNavigation(buffer, "https://example.com/list?cat=tech", true);
expect(buffer.steps).toEqual([
{
op: "select",
captureTarget: { tag: "select", role: "combobox", name: "分类" },
values: ["tech"],
navigatedTo: "https://example.com/list?cat=tech",
},
]);
});
it("emits navigate for uncaused URL changes", () => {
const buffer = {
steps: [],
navigation: { currentUrl: "https://example.com/a", pendingNavigation: false },
};
const result = observeRecordedNavigation(buffer, "https://example.com/b", false);
expect(result).toEqual({ kind: "appended", index: 0 });
expect(buffer.steps[0]).toMatchObject({
op: "navigate",
url: "https://example.com/b",
});
});
it("asks the recorder to coalesce redirect hops instead of emitting each one", () => {
const buffer = {
steps: [],
navigation: {
currentUrl: "https://passport.example/login",
pendingNavigation: false,
},
};
const hop1 = observeRecordedNavigation(
buffer,
"https://passport.example/callback",
false,
"link",
["server_redirect"],
);
expect(hop1).toEqual({
kind: "coalesce_redirect",
url: "https://passport.example/callback",
});
expect(buffer.steps).toEqual([]);
const hop2 = observeRecordedNavigation(buffer, "https://app.example/dashboard", false, "link", [
"client_redirect",
]);
expect(hop2).toEqual({
kind: "coalesce_redirect",
url: "https://app.example/dashboard",
});
expect(buffer.steps).toEqual([]);
expect(buffer.navigation.currentUrl).toBe("https://app.example/dashboard");
});
it("does not reuse redirect metadata for a later content-observed URL change", () => {
const buffer = {
steps: [],
navigation: {
currentUrl: "https://example.com/redirect",
pendingNavigation: false,
},
};
const result = observeRecordedNavigation(buffer, "https://example.com/spa");
expect(result).toEqual({ kind: "appended", index: 0 });
expect(buffer.steps[0]).toMatchObject({ op: "navigate", url: "https://example.com/spa" });
});
});
@@ -0,0 +1,39 @@
import { describe, expect, it } from "vitest";
import { RecordingTabCoordinator } from "../recording/tab-coordinator";
describe("RecordingTabCoordinator", () => {
it("keeps navigation state isolated per tab", () => {
const tabs = new RecordingTabCoordinator(4, "https://example.com/first");
const first = tabs.navigation(4);
first.pendingNavigation = true;
const second = tabs.navigation(5, "https://example.com/second");
expect(second).toEqual({
currentUrl: "https://example.com/second",
pendingNavigation: false,
});
expect(tabs.navigation(4)).toBe(first);
expect(tabs.navigation(4).pendingNavigation).toBe(true);
});
it("invalidates an earlier activation when a newer tab becomes active", () => {
const tabs = new RecordingTabCoordinator(4);
const second = tabs.noteActivation(5);
const third = tabs.noteActivation(6);
expect(tabs.isLatest(second)).toBe(false);
expect(tabs.isLatest(third)).toBe(true);
tabs.commit(6);
expect(tabs.currentTabId).toBe(6);
});
it("commits the freshly observed URL when returning to an existing tab", () => {
const tabs = new RecordingTabCoordinator(4);
tabs.navigation(5, "https://example.com/old");
tabs.commit(5, "https://example.com/new");
expect(tabs.currentTabId).toBe(5);
expect(tabs.navigation(5).currentUrl).toBe("https://example.com/new");
});
});
@@ -0,0 +1,159 @@
import type { RenderedRef } from "@browser-skill/vom";
import { describe, expect, it } from "vitest";
import type { CaptureVomMatchNode } from "@/tools/capture-vom-observation";
import { ObservationNodeIndex, type RegisteredObservation } from "../recording/observation-capture";
import { matchObservationTarget } from "../recording/target-matcher";
function node(backendNodeId: number, frameId: string, x = 10): CaptureVomMatchNode {
return {
backendNodeId,
frameId,
tag: "button",
rect: { x, y: 20, w: 100, h: 30 },
localRect: { x, y: 20, w: 100, h: 30 },
};
}
function observation(
matchNodes: CaptureVomMatchNode[],
refs: RenderedRef[],
): RegisteredObservation {
return {
stateId: "s1",
rootFrameId: "root",
index: new ObservationNodeIndex({ rootFrameId: "root", matchNodes, refs }),
url: "https://example.com",
};
}
describe("matchObservationTarget", () => {
it("indexes the safe geometry DTO without discarding frame-local geometry", () => {
const geometry = node(42, "child");
geometry.localRect = { x: 5, y: 6, w: 100, h: 30 };
const ref: RenderedRef = {
ref: "e1",
backendNodeId: 42,
frameId: "child",
line: 1,
};
const candidate = observation([geometry], [ref]).index.candidates("child", "button")[0];
expect(candidate).toEqual({ frameId: "child", geometry, ref });
});
it("matches a unique node using canonical top-level viewport geometry", () => {
const target = matchObservationTarget({
observation: observation(
[node(42, "root")],
[{ ref: "e1", backendNodeId: 42, role: "button", name: "发布", line: 1 }],
),
hint: { geometry: { rect: { x: 10, y: 20, w: 100, h: 30 }, tag: "button" } },
});
expect(target).toEqual({ ref: "e1", role: "button", name: "发布" });
});
it("uses frame id with backend node id so sibling frames cannot collide", () => {
const target = matchObservationTarget({
observation: observation(
[node(42, "left"), node(42, "right")],
[
{ ref: "e1", backendNodeId: 42, frameId: "left", line: 1 },
{ ref: "e2", backendNodeId: 42, frameId: "right", line: 2 },
],
),
hint: {
frameId: "right",
geometry: { rect: { x: 10, y: 20, w: 100, h: 30 }, tag: "button" },
},
});
expect(target.ref).toBe("e2");
});
it("matches iframe capture geometry in the child viewport coordinate space", () => {
const child = node(42, "child", 410);
child.localRect = { x: 10, y: 20, w: 100, h: 30 };
const target = matchObservationTarget({
observation: observation(
[child],
[{ ref: "e1", backendNodeId: 42, frameId: "child", line: 1 }],
),
hint: {
frameId: "child",
geometrySpace: "local",
geometry: { rect: { x: 10, y: 20, w: 100, h: 30 }, tag: "button" },
},
});
expect(target.ref).toBe("e1");
});
it("restricts a missing frame hint to the root frame", () => {
const target = matchObservationTarget({
observation: observation(
[node(42, "child")],
[{ ref: "e1", backendNodeId: 42, frameId: "child", line: 1 }],
),
hint: { geometry: { rect: { x: 10, y: 20, w: 100, h: 30 }, tag: "button" } },
fallback: { tag: "button", name: "发布" },
});
expect(target).toEqual({ name: "发布", unmatched: true });
});
it("fails closed when the source Document has no VOM frame mapping", () => {
const target = matchObservationTarget({
observation: observation(
[node(42, "root")],
[{ ref: "e1", backendNodeId: 42, role: "button", name: "发布", line: 1 }],
),
hint: {
frameId: null,
geometry: { rect: { x: 10, y: 20, w: 100, h: 30 }, tag: "button" },
},
fallback: { tag: "button", role: "button", name: "发布" },
});
expect(target).toEqual({ role: "button", name: "发布", unmatched: true });
});
it("returns unmatched for ambiguous geometry", () => {
const target = matchObservationTarget({
observation: observation(
[node(42, "root"), node(43, "root")],
[
{ ref: "e1", backendNodeId: 42, line: 1 },
{ ref: "e2", backendNodeId: 43, line: 2 },
],
),
hint: { geometry: { rect: { x: 10, y: 20, w: 100, h: 30 }, tag: "button" } },
});
expect(target.unmatched).toBe(true);
});
it("uses semantics when geometry is unavailable without crossing frame boundaries", () => {
const target = matchObservationTarget({
observation: observation(
[node(41, "root"), node(42, "child")],
[
{ ref: "e1", backendNodeId: 41, role: "button", name: "保存", line: 1 },
{ ref: "e2", backendNodeId: 42, frameId: "child", role: "button", name: "保存", line: 2 },
],
),
hint: { frameId: "child" },
fallback: { tag: "button", role: "button", name: "保存" },
});
expect(target.ref).toBe("e2");
});
it("uses semantics to disambiguate equal geometry in one frame", () => {
const target = matchObservationTarget({
observation: observation(
[node(42, "root"), node(43, "root")],
[
{ ref: "e1", backendNodeId: 42, role: "button", name: "保存", line: 1 },
{ ref: "e2", backendNodeId: 43, role: "button", name: "取消", line: 2 },
],
),
hint: { geometry: { rect: { x: 10, y: 20, w: 100, h: 30 }, tag: "button" } },
fallback: { tag: "button", role: "button", name: "取消" },
});
expect(target.ref).toBe("e2");
});
});
@@ -1,6 +1,15 @@
import { describe, expect, it } from "vitest";
import type { DraftTraceStep } from "@/transport/types";
import { reduceTraceSteps, resolveTraceStartUrl, shouldRecordPress } from "../trace-reducer";
import type { RecordingDraftStep } from "@/lib/recording/types";
import { shouldRecordPress } from "../recording/draft-policy";
import { buildTraceV2 } from "../recording/trace-reducer-v2";
function reduceTraceSteps(steps: RecordingDraftStep[], startUrl?: string) {
return buildTraceV2({
steps,
startedAt: "2026-01-01T00:00:00.000Z",
...(startUrl ? { startUrl } : {}),
});
}
describe("shouldRecordPress", () => {
it("keeps Enter and Escape", () => {
@@ -18,36 +27,47 @@ describe("shouldRecordPress", () => {
});
describe("reduceTraceSteps", () => {
it("does not expose internal tab transitions in trace v2", () => {
const trace = reduceTraceSteps([
{
op: "switch_tab",
preStateId: "s1",
postStateId: "s2",
},
]);
expect(trace.steps).toEqual([]);
});
it("builds steps with pages dictionary and page id references", () => {
const drafts: DraftTraceStep[] = [
const drafts: RecordingDraftStep[] = [
{
op: "navigate",
url: "https://example.com/search?q=hello&utm_source=x",
page_url: "https://example.com/search?q=hello&utm_source=x",
pageUrl: "https://example.com/search?q=hello&utm_source=x",
},
{
op: "fill",
target: { tag: "input", role: "textbox", name: "搜索", name_attr: "q" },
captureTarget: { tag: "input", role: "textbox", name: "搜索", name_attr: "q" },
value: "browser skill",
page_url: "https://example.com/search?q=hello&utm_source=x",
pageUrl: "https://example.com/search?q=hello&utm_source=x",
},
{
op: "press",
key: "Enter",
target: { tag: "input", role: "textbox", name: "搜索", name_attr: "q" },
navigated_to: "https://example.com/results/42",
page_url: "https://example.com/search?q=hello&utm_source=x",
captureTarget: { tag: "input", role: "textbox", name: "搜索", name_attr: "q" },
navigatedTo: "https://example.com/results/42",
pageUrl: "https://example.com/search?q=hello&utm_source=x",
},
{
op: "click",
target: { tag: "button", role: "button", name: "发布" },
navigated_to: "https://example.com/p/99",
page_url: "https://example.com/results/42",
captureTarget: { tag: "button", role: "button", name: "发布" },
navigatedTo: "https://example.com/p/99",
pageUrl: "https://example.com/results/42",
},
{
op: "press",
key: "a",
page_url: "https://example.com/p/99",
pageUrl: "https://example.com/p/99",
},
];
@@ -109,9 +129,9 @@ describe("reduceTraceSteps", () => {
[
{
op: "fill",
target: { tag: "input", role: "textbox", name: "Search query" },
captureTarget: { tag: "input", role: "textbox", name: "Search query" },
value: "",
page_url: "https://example.com/search",
pageUrl: "https://example.com/search",
},
],
"https://example.com/search",
@@ -125,16 +145,16 @@ describe("reduceTraceSteps", () => {
]);
});
it("maps select navigated_to onto effect.navigated_to (page id)", () => {
it("maps select navigatedTo onto effect.navigatedTo (page id)", () => {
const { pages, steps } = reduceTraceSteps(
[
{
op: "select",
target: { tag: "select", role: "combobox", name: "分类" },
captureTarget: { tag: "select", role: "combobox", name: "分类" },
values: ["tech"],
labels: ["技术"],
navigated_to: "https://example.com/list?cat=tech",
page_url: "https://example.com/list",
navigatedTo: "https://example.com/list?cat=tech",
pageUrl: "https://example.com/list",
},
],
"https://example.com/list",
@@ -151,36 +171,32 @@ describe("reduceTraceSteps", () => {
});
});
it("keeps hover steps before menu clicks", () => {
it("drops hover steps, which only exist in trace v3", () => {
const { steps } = reduceTraceSteps(
[
{
op: "hover",
target: { tag: "span", role: "button", name: "Account" },
page_url: "https://example.com/app",
captureTarget: { tag: "span", role: "button", name: "Account" },
pageUrl: "https://example.com/app",
},
{
op: "click",
target: { tag: "a", role: "link", name: "Profile" },
page_url: "https://example.com/app",
captureTarget: { tag: "a", role: "link", name: "Profile" },
pageUrl: "https://example.com/app",
},
],
"https://example.com/app",
);
expect(steps.map((s) => s.op)).toEqual(["hover", "click"]);
expect(steps[0]).toMatchObject({
op: "hover",
page: "p1",
target: { name: "Account" },
});
expect(steps.map((s) => s.op)).toEqual(["click"]);
});
it("resolveTraceStartUrl prefers explicit start URL", () => {
expect(
resolveTraceStartUrl(
[{ op: "navigate", url: "https://example.com/other" }],
"https://example.com/start",
),
buildTraceV2({
steps: [{ op: "navigate", url: "https://example.com/other" }],
startedAt: "2026-01-01T00:00:00.000Z",
startUrl: "https://example.com/start",
}).entry.start_url,
).toBe("https://example.com/start");
});
});
@@ -0,0 +1,132 @@
import { describe, expect, it } from "vitest";
import { TRACE_VERSION_V3, VOM_FORMAT_VERSION } from "@/transport/types";
import { RecordingStateRegistry } from "../recording/state-registry";
import { buildTraceV3 } from "../recording/trace-builder-v3";
import { reduceTraceStepsV3 } from "../recording/trace-reducer-v3";
import type { RecordingDraftStep } from "../recording/types";
describe("trace reducer v3", () => {
it("removes form literals when value redaction is requested", () => {
const reduced = reduceTraceStepsV3(
[
{
op: "fill",
value: "private@example.com",
preStateId: "s1",
postStateId: "s1",
},
{
op: "select",
values: ["private-account-id"],
labels: ["Private account"],
preStateId: "s1",
postStateId: "s1",
},
],
{ redactValues: true },
);
expect(reduced.steps[0]).toMatchObject({ op: "fill", value: "***", redacted: true });
expect(reduced.steps[1]).toMatchObject({ op: "select" });
expect(reduced.steps[1]).not.toHaveProperty("selection");
expect(JSON.stringify(reduced.steps)).not.toContain("private@example.com");
expect(JSON.stringify(reduced.steps)).not.toContain("private-account-id");
});
it("emits tab transitions only for callers that advertised support", () => {
const drafts: RecordingDraftStep[] = [
{
op: "switch_tab",
preStateId: "s1",
postStateId: "s2",
},
];
expect(reduceTraceStepsV3(drafts).steps).toEqual([]);
expect(reduceTraceStepsV3(drafts, { includeTabSwitches: true }).steps).toEqual([
{ op: "switch_tab", id: 1, state: "s1", result: { state: "s2" } },
]);
});
it("keeps hover steps before menu clicks", () => {
const drafts: RecordingDraftStep[] = [
{
op: "hover",
captureTarget: { tag: "span", role: "button", name: "Account" },
preStateId: "s1",
postStateId: "s1",
},
{
op: "click",
captureTarget: { tag: "a", role: "link", name: "Profile" },
preStateId: "s1",
postStateId: "s1",
},
];
expect(reduceTraceStepsV3(drafts).steps.map((s) => s.op)).toEqual(["hover", "click"]);
});
it("does not export a tab transition without both observation endpoints", () => {
const sourceOnly: RecordingDraftStep = { op: "switch_tab", preStateId: "s1" };
const targetOnly: RecordingDraftStep = { op: "switch_tab", postStateId: "s2" };
expect(
reduceTraceStepsV3([sourceOnly, targetOnly], { includeTabSwitches: true }).steps,
).toEqual([]);
});
it("collapses redirect hops while retaining draft-to-step identity", () => {
const drafts: RecordingDraftStep[] = [
{ op: "navigate", url: "https://example.com/start", preStateId: "s1", postStateId: "s2" },
{
op: "navigate",
url: "https://example.com/final",
transitionQualifiers: ["server_redirect"],
preStateId: "s2",
postStateId: "s3",
},
];
const before = structuredClone(drafts);
const reduced = reduceTraceStepsV3(drafts);
expect(drafts).toEqual(before);
expect(reduced.steps).toEqual([
expect.objectContaining({
id: 1,
op: "navigate",
state: "s1",
to: "https://example.com/final",
result: { state: "s3" },
}),
]);
expect(reduced.stepIdByDraftId.get(1)).toBe(1);
expect(reduced.stepIdByDraftId.get(2)).toBe(1);
});
it("builds the wire model from protocol constants", () => {
const registry = new RecordingStateRegistry();
const state = registry.register({ url: "https://example.com", vomText: "@vom 1" });
const trace = buildTraceV3({
registry,
drafts: [
{
op: "click",
captureTarget: { tag: "button", role: "button", name: "Save" },
preStateId: state.id,
postStateId: state.id,
},
],
startedAt: "2026-08-12T00:00:00.000Z",
stoppedBy: "user_finish",
bskVersion: "test",
});
expect(trace.version).toBe(TRACE_VERSION_V3);
expect(trace.recorder.vom).toBe(VOM_FORMAT_VERSION);
expect(trace.steps[0]).toMatchObject({
op: "click",
target: { role: "button", name: "Save", unmatched: true },
});
});
});
@@ -1,9 +1,9 @@
/**
* Wire protocol and background-side helper for hiding the extension's
* in-page overlay while a screenshot is captured, so captured frames only
* contain the page itself.
* Wire protocol and background-side helper for temporarily excluding the
* extension's in-page overlay. Screenshots use it to avoid captured chrome;
* coordinate-driven native input uses it so Chrome hit-tests the page.
*
* The background wraps every capture in `withOverlaysHiddenForCapture`,
* The background wraps each bounded operation in `withExtensionOverlayHidden`,
* which sends `begin` (the content script hides the overlay host and only
* acks once the compositor has produced an overlay-free frame) and always
* follows up with `end`, even when the capture throws. Tabs without the
@@ -46,7 +46,7 @@ const defaultSendToTab: CaptureSuppressSendToTab = (tabId, message) =>
* (or any other `begin` failure) simply skips suppression — there is no
* overlay to hide in that tab.
*/
export async function withOverlaysHiddenForCapture<T>(
export async function withExtensionOverlayHidden<T>(
tabId: number,
fn: () => Promise<T>,
sendToTab: CaptureSuppressSendToTab = defaultSendToTab,
+6 -5
View File
@@ -1,12 +1,13 @@
/**
* Build a semantic TargetDescriptor for an interacted element.
* Build a semantic capture descriptor for an interacted element.
*
* Trace steps are an LLM *textbook*: each click must say what to look for
* on screen (usually a short visible name). Tag-only noise like
* `{ "tag": "div" }` / “点击div” is useless and must not be recorded.
*/
export interface TargetDescriptor {
/** Content-script capture descriptor before VOM geometric matching. */
export interface CaptureTargetDescriptor {
role?: string;
name?: string;
tag: string;
@@ -247,7 +248,7 @@ export function resolveHoverElement(target: Element): Element | null {
* (visible name), or at least a form `name_attr` for checkbox/radio.
* Recording “点击div” with no name fails this bar.
*/
export function isMeaningfulClickTarget(target: TargetDescriptor): boolean {
export function isMeaningfulClickTarget(target: CaptureTargetDescriptor): boolean {
const name = target.name?.trim();
if (name && isActionableLabel(name)) return true;
if (
@@ -323,7 +324,7 @@ function nearbyLabelText(el: Element): string | undefined {
return undefined;
}
export function describeTarget(el: Element): TargetDescriptor {
export function describeTarget(el: Element): CaptureTargetDescriptor {
const tag = el.tagName.toLowerCase();
const role = inferRole(el);
const name = accessibleName(el);
@@ -350,7 +351,7 @@ export function describeTarget(el: Element): TargetDescriptor {
};
}
export function describeEventTarget(target: EventTarget | null): TargetDescriptor | null {
export function describeEventTarget(target: EventTarget | null): CaptureTargetDescriptor | null {
if (!(target instanceof Element)) return null;
const clickable = resolveClickableElement(target);
if (!clickable) return null;
+39 -3
View File
@@ -3,7 +3,7 @@
* service worker and a tab's content script.
*/
import type { TargetDescriptor } from "./describe-target";
import type { CaptureTargetDescriptor } from "./describe-target";
export const RECORD_START = "bsk-record-start";
export const RECORD_STEP = "bsk-record-step";
@@ -16,6 +16,10 @@ export interface RecordStartAck {
ok: true;
}
export type RecordStepAck =
| { ok: true; sequence: number }
| { ok: false; expectedSequence: number; error: string };
export type RecordStopAck = { ok: true } | { ok: false; error: string };
export interface RecordQueryMessage {
@@ -42,7 +46,7 @@ export interface RecordStartMessage {
export interface RecordStepPayload {
op: "click" | "hover" | "fill" | "press" | "select" | "navigate";
target?: TargetDescriptor;
target?: CaptureTargetDescriptor;
value?: string;
key?: string;
modifiers?: Array<"alt" | "ctrl" | "meta" | "shift">;
@@ -50,17 +54,30 @@ export interface RecordStepPayload {
labels?: string[];
url?: string;
redacted?: boolean;
commit?: "enter" | "suggestion" | "blur";
/** Page URL when the step was captured. */
page_url?: string;
/** Event-target bounds in the source frame's viewport coordinate space. */
geometry?: {
rect: { x: number; y: number; w: number; h: number };
tag: string;
};
/** Capture-only hint; never persisted unless converted to navigated_to. */
expects_navigation?: boolean;
/** Whether an observed URL change was synchronously caused by the action. */
navigation_caused_by_action?: boolean;
/** Raw webNavigation transition metadata for cause mapping. */
transitionType?: string;
transitionQualifiers?: string[];
}
export interface RecordStepMessage {
type: typeof RECORD_STEP;
requestId: string;
/** Stable for the lifetime of one content-script document. */
producerId: string;
/** Monotonic within one content-script producer and recording request. */
sequence: number;
step: RecordStepPayload;
}
@@ -112,8 +129,27 @@ export function isRecordQueryMessage(msg: unknown): msg is RecordQueryMessage {
export function isRecordStepMessage(msg: unknown): msg is RecordStepMessage {
if (typeof msg !== "object" || msg === null) return false;
const m = msg as Record<string, unknown>;
if (m.type !== RECORD_STEP || typeof m.requestId !== "string") return false;
if (
m.type !== RECORD_STEP ||
typeof m.requestId !== "string" ||
typeof m.producerId !== "string" ||
m.producerId.length === 0 ||
typeof m.sequence !== "number" ||
!Number.isSafeInteger(m.sequence) ||
m.sequence < 1
) {
return false;
}
const step = m.step;
if (typeof step !== "object" || step === null) return false;
return typeof (step as RecordStepPayload).op === "string";
}
export function isAcceptedRecordStepAck(
value: unknown,
sequence: number,
): value is Extract<RecordStepAck, { ok: true }> {
if (typeof value !== "object" || value === null) return false;
const ack = value as Partial<RecordStepAck>;
return ack.ok === true && ack.sequence === sequence;
}
@@ -1,136 +0,0 @@
import type { DraftTraceStep } from "@/transport/types";
import type { RecordStepPayload } from "./record-bridge";
export interface RecordingStepBuffer {
steps: DraftTraceStep[];
currentUrl?: string;
pendingNavigation: boolean;
pendingNavigationDeadline?: number;
}
const NAVIGATION_TRIGGER_WINDOW_MS = 3_000;
function toDraftStep(payload: RecordStepPayload): DraftTraceStep | null {
const pageUrl = payload.page_url;
switch (payload.op) {
case "click":
return payload.target
? {
op: "click",
target: payload.target,
...(pageUrl ? { page_url: pageUrl } : {}),
}
: null;
case "hover":
return payload.target
? {
op: "hover",
target: payload.target,
...(pageUrl ? { page_url: pageUrl } : {}),
}
: null;
case "fill":
return payload.target
? {
op: "fill",
target: payload.target,
value: payload.value ?? "",
...(payload.redacted ? { redacted: true } : {}),
...(pageUrl ? { page_url: pageUrl } : {}),
}
: null;
case "press":
return payload.key
? {
op: "press",
key: payload.key,
...(payload.target ? { target: payload.target } : {}),
...(payload.modifiers?.length ? { modifiers: payload.modifiers } : {}),
...(pageUrl ? { page_url: pageUrl } : {}),
}
: null;
case "select":
return payload.target && payload.values
? {
op: "select",
target: payload.target,
values: payload.values,
...(payload.labels?.length ? { labels: payload.labels } : {}),
...(pageUrl ? { page_url: pageUrl } : {}),
}
: null;
case "navigate":
return null;
}
}
function annotateLastStepNavigation(buffer: RecordingStepBuffer, url: string): boolean {
for (let i = buffer.steps.length - 1; i >= 0; i -= 1) {
const step = buffer.steps[i];
if (!step) continue;
if (step.op === "click" || step.op === "press" || step.op === "select") {
buffer.steps[i] = { ...step, navigated_to: url };
return true;
}
break;
}
return false;
}
export function observeRecordedNavigation(
buffer: RecordingStepBuffer,
url: string,
causedByAction?: boolean,
): void {
if (!url || url === buffer.currentUrl) return;
buffer.currentUrl = url;
const pendingIsCurrent =
buffer.pendingNavigation &&
(buffer.pendingNavigationDeadline === undefined ||
buffer.pendingNavigationDeadline >= Date.now());
if (causedByAction === true || (causedByAction === undefined && pendingIsCurrent)) {
buffer.pendingNavigation = false;
buffer.pendingNavigationDeadline = undefined;
if (!annotateLastStepNavigation(buffer, url)) {
buffer.steps.push({
op: "navigate",
url,
page_url: url,
});
}
return;
}
buffer.pendingNavigation = false;
buffer.pendingNavigationDeadline = undefined;
buffer.steps.push({
op: "navigate",
url,
page_url: url,
});
}
export function appendRecordedPayload(
buffer: RecordingStepBuffer,
payload: RecordStepPayload,
): void {
if (payload.op === "navigate") {
if (payload.url) {
observeRecordedNavigation(buffer, payload.url, payload.navigation_caused_by_action);
}
return;
}
const step = toDraftStep({
...payload,
page_url: payload.page_url ?? buffer.currentUrl,
});
if (!step) return;
buffer.steps.push(step);
if (step.op === "click" || step.op === "press" || step.op === "select" || step.op === "fill") {
buffer.pendingNavigation = payload.expects_navigation === true;
buffer.pendingNavigationDeadline = buffer.pendingNavigation
? Date.now() + NAVIGATION_TRIGGER_WINDOW_MS
: undefined;
}
}
@@ -0,0 +1,39 @@
import {
RECORD_DOCUMENT_ATTRIBUTE,
recordingDocumentMarkerValue,
} from "@/shared/recording-document-identity";
export interface RecordingDocumentMarker {
restore(): void;
ensure(): void;
}
export function waitForDocumentElement(): Promise<HTMLElement> {
if (document.documentElement) return Promise.resolve(document.documentElement);
return new Promise((resolve) => {
const observer = new MutationObserver(() => {
if (!document.documentElement) return;
observer.disconnect();
resolve(document.documentElement);
});
observer.observe(document, { childList: true });
});
}
export function markRecordingDocument(
producerId: string,
root: HTMLElement = document.documentElement,
): RecordingDocumentMarker {
const previous = root.getAttribute(RECORD_DOCUMENT_ATTRIBUTE);
const markerValue = recordingDocumentMarkerValue(producerId);
const ensure = () => root.setAttribute(RECORD_DOCUMENT_ATTRIBUTE, markerValue);
ensure();
return {
ensure,
restore() {
if (root.getAttribute(RECORD_DOCUMENT_ATTRIBUTE) !== markerValue) return;
if (previous === null) root.removeAttribute(RECORD_DOCUMENT_ATTRIBUTE);
else root.setAttribute(RECORD_DOCUMENT_ATTRIBUTE, previous);
},
};
}
@@ -0,0 +1,134 @@
import type { CdpTarget } from "@/browser-driver/frame-graph";
import { type CdpRunner, sendToCdpTarget } from "@/tools/shared";
const SETTLE_MIN_MS = 150;
const SETTLE_QUIET_MS = 250;
const SETTLE_MAX_MS = 2_000;
const SETTLE_POLL_MS = 60;
const SETTLE_WORLD_NAME = "__bsk_record_settle__";
const QUIET_PROBE = `(() => {
const scope = window;
let probe = scope.__bskRecordQuiet;
if (!probe) {
probe = { changedAt: Date.now() };
const observer = new MutationObserver(() => {
probe.changedAt = Date.now();
});
observer.observe(document.documentElement, {
subtree: true,
childList: true,
attributes: true,
characterData: true,
});
scope.__bskRecordQuiet = probe;
}
return { idleMs: Date.now() - probe.changedAt, readyState: document.readyState };
})()`;
export interface DocumentSettleScope {
target: CdpTarget;
/** Omit only for the target's root document. */
frameId?: string;
}
type SettleOutcome = "quiet" | "timeout" | "cancelled";
interface QuietProbe {
idleMs: number;
readyState: string;
}
interface ProbeContext {
executionContextId?: number;
}
function sleep(ms: number, signal?: AbortSignal): Promise<void> {
return new Promise((resolve) => {
if (signal?.aborted) return resolve();
const timer = setTimeout(resolve, ms);
signal?.addEventListener(
"abort",
() => {
clearTimeout(timer);
resolve();
},
{ once: true },
);
});
}
async function executionContextForScope(
cdp: CdpRunner,
scope: DocumentSettleScope,
): Promise<number | undefined> {
if (!scope.frameId) return undefined;
const result = await sendToCdpTarget<{ executionContextId?: number }>(
cdp,
scope.target,
"Page.createIsolatedWorld",
{
frameId: scope.frameId,
worldName: SETTLE_WORLD_NAME,
grantUniveralAccess: false,
},
);
return result.executionContextId;
}
async function readQuietProbe(
cdp: CdpRunner,
scope: DocumentSettleScope,
context: ProbeContext,
): Promise<QuietProbe | null> {
try {
if (scope.frameId && context.executionContextId === undefined) {
context.executionContextId = await executionContextForScope(cdp, scope);
if (context.executionContextId === undefined) return null;
}
const reply = await sendToCdpTarget<{ result?: { value?: unknown } }>(
cdp,
scope.target,
"Runtime.evaluate",
{
expression: QUIET_PROBE,
returnByValue: true,
...(context.executionContextId !== undefined
? { contextId: context.executionContextId }
: {}),
},
);
const value = reply.result?.value;
if (!value || typeof value !== "object") return null;
const { idleMs, readyState } = value as { idleMs?: unknown; readyState?: unknown };
if (typeof idleMs !== "number" || typeof readyState !== "string") return null;
return { idleMs, readyState };
} catch {
context.executionContextId = undefined;
return null;
}
}
export async function waitForDocumentSettled(
cdp: CdpRunner,
scope: DocumentSettleScope,
options: { signal?: AbortSignal } = {},
): Promise<SettleOutcome> {
const startedAt = Date.now();
const floor = startedAt + SETTLE_MIN_MS;
const deadline = startedAt + SETTLE_MAX_MS;
const context: ProbeContext = {};
for (;;) {
if (options.signal?.aborted) return "cancelled";
await sleep(SETTLE_POLL_MS, options.signal);
if (options.signal?.aborted) return "cancelled";
const probe = await readQuietProbe(cdp, scope, context);
const now = Date.now();
if (now < floor) continue;
if (now >= deadline) return "timeout";
if (!probe || probe.readyState === "loading") continue;
if (probe.idleMs >= SETTLE_QUIET_MS) return "quiet";
}
}
@@ -0,0 +1,34 @@
import type { RecordingDraftStep } from "./types";
const CLIPBOARD_KEYS = new Set(["a", "c", "v", "x", "A", "C", "V", "X"]);
const MODIFIER_ONLY_KEYS = new Set(["Meta", "Control", "Alt", "Shift", "OS", "Hyper", "Super"]);
export function shouldRecordPress(
key: string,
modifiers?: Array<"alt" | "ctrl" | "meta" | "shift">,
): boolean {
if (MODIFIER_ONLY_KEYS.has(key)) return false;
const mods = modifiers ?? [];
const hasCtrlOrMeta = mods.includes("ctrl") || mods.includes("meta");
if (hasCtrlOrMeta && CLIPBOARD_KEYS.has(key)) return false;
if (key === "Enter" || key === "Escape") return true;
if (key.length === 1 && !hasCtrlOrMeta && !mods.includes("alt")) return false;
return false;
}
export function shouldIncludeDraft(step: RecordingDraftStep): boolean {
return step.op !== "press" || shouldRecordPress(step.key, step.modifiers);
}
export function resolveDraftStartUrl(
drafts: RecordingDraftStep[],
explicitStartUrl?: string,
fallbackUrl?: string,
): string {
if (explicitStartUrl) return explicitStartUrl;
const navigation = drafts.find(
(step): step is Extract<RecordingDraftStep, { op: "navigate" }> => step.op === "navigate",
);
if (navigation) return navigation.url;
return drafts.find((step) => step.pageUrl)?.pageUrl ?? fallbackUrl ?? "about:blank";
}
@@ -0,0 +1,65 @@
export const RECORD_FRAME_PORT = "bsk-record-frame";
export const RECORD_FRAME_QUERY = "bsk-record-frame-query";
export const RECORD_FRAME_START = "bsk-record-frame-start";
export interface RecordFrameQueryMessage {
type: typeof RECORD_FRAME_QUERY;
}
export interface RecordFrameQueryResponse {
active: boolean;
requestId?: string;
startedAtMs?: number;
}
export interface RecordFrameStartMessage {
type: typeof RECORD_FRAME_START;
requestId: string;
startedAtMs: number;
}
export type RecordFramePortMessage =
| {
type: "ready";
requestId: string;
producerId: string;
}
| {
type: "ready_ack";
requestId: string;
producerId: string;
}
| {
type: "stop";
requestId: string;
commandId: string;
}
| {
type: "cancel";
requestId: string;
}
| {
type: "stopped";
requestId: string;
commandId: string;
ok: boolean;
error?: string;
};
export function isRecordFrameQueryMessage(value: unknown): value is RecordFrameQueryMessage {
return (
typeof value === "object" &&
value !== null &&
(value as { type?: unknown }).type === RECORD_FRAME_QUERY
);
}
export function isRecordFrameStartMessage(value: unknown): value is RecordFrameStartMessage {
if (typeof value !== "object" || value === null) return false;
const message = value as Partial<RecordFrameStartMessage>;
return (
message.type === RECORD_FRAME_START &&
typeof message.requestId === "string" &&
typeof message.startedAtMs === "number"
);
}
@@ -0,0 +1,399 @@
import {
isRecordFrameQueryMessage,
RECORD_FRAME_PORT,
RECORD_FRAME_START,
type RecordFramePortMessage,
type RecordFrameQueryResponse,
type RecordFrameStartMessage,
} from "./frame-bridge";
export interface RecordingCaptureScope {
tabId: number;
documentId: string;
browserFrameId: number;
producerId: string;
}
interface ArmedRecording {
requestId: string;
startedAtMs: number;
tabIds: Set<number>;
agents: Map<string, FrameAgent>;
onDocumentReady?: (scope: RecordingCaptureScope) => void | Promise<void>;
finishing: boolean;
}
interface FrameAgent extends RecordingCaptureScope {
requestId: string;
port: chrome.runtime.Port;
stopWaiters: Map<string, (ok: boolean) => void>;
}
interface BrowserFrame {
frameId: number;
documentId?: string;
}
interface BrowserFrameNavigation extends BrowserFrame {
tabId: number;
lifecycle: "committed" | "completed";
}
const RECORD_FRAME_STOP_TIMEOUT_MS = 5_000;
export interface RecordFrameCoordinatorDeps {
getAllFrames(tabId: number): Promise<BrowserFrame[]>;
sendToDocument(
tabId: number,
message: RecordFrameStartMessage,
target: { documentId?: string; frameId?: number },
): Promise<unknown>;
subscribeFrameNavigation?(listener: (frame: BrowserFrameNavigation) => void): () => void;
}
function documentKey(tabId: number, documentId: string): string {
return `${tabId}:${documentId}`;
}
function senderAddress(
sender: chrome.runtime.MessageSender,
): { tabId: number; documentId: string; browserFrameId: number } | null {
const tabId = sender.tab?.id;
const documentId = sender.documentId;
const browserFrameId = sender.frameId;
if (
typeof tabId !== "number" ||
typeof documentId !== "string" ||
typeof browserFrameId !== "number"
) {
return null;
}
return { tabId, documentId, browserFrameId };
}
function defaultDeps(): RecordFrameCoordinatorDeps {
return {
async getAllFrames(tabId) {
return (await chrome.webNavigation.getAllFrames({ tabId })) ?? [];
},
sendToDocument(tabId, message, target) {
return chrome.tabs.sendMessage(tabId, message, target);
},
subscribeFrameNavigation(listener) {
const onCommitted = (details: chrome.webNavigation.WebNavigationTransitionCallbackDetails) =>
listener({ ...details, lifecycle: "committed" });
const onCompleted = (details: chrome.webNavigation.WebNavigationFramedCallbackDetails) =>
listener({ ...details, lifecycle: "completed" });
chrome.webNavigation.onCommitted.addListener(onCommitted);
chrome.webNavigation.onCompleted.addListener(onCompleted);
return () => {
chrome.webNavigation.onCommitted.removeListener(onCommitted);
chrome.webNavigation.onCompleted.removeListener(onCompleted);
};
},
};
}
export class RecordFrameCoordinator {
readonly #deps: RecordFrameCoordinatorDeps;
readonly #recordings = new Map<string, ArmedRecording>();
#attached = false;
#detachFrameNavigation: (() => void) | null = null;
constructor(deps: RecordFrameCoordinatorDeps = defaultDeps()) {
this.#deps = deps;
}
attach(): () => void {
if (this.#attached) return () => {};
this.#attached = true;
chrome.runtime.onConnect.addListener(this.#onConnect);
chrome.runtime.onMessage.addListener(this.#onMessage);
this.#detachFrameNavigation =
this.#deps.subscribeFrameNavigation?.(this.#onFrameNavigation) ?? null;
return () => {
if (!this.#attached) return;
this.#attached = false;
chrome.runtime.onConnect.removeListener(this.#onConnect);
chrome.runtime.onMessage.removeListener(this.#onMessage);
this.#detachFrameNavigation?.();
this.#detachFrameNavigation = null;
for (const recording of this.#recordings.values()) this.#cancelAgents(recording);
this.#recordings.clear();
};
}
begin(
requestId: string,
startedAtMs: number,
initialTabId: number,
onDocumentReady?: (scope: RecordingCaptureScope) => void | Promise<void>,
): void {
const previous = this.#recordings.get(requestId);
if (previous) this.#cancelAgents(previous);
this.#recordings.set(requestId, {
requestId,
startedAtMs,
// Register the initial tab before navigation begins. A newly-created
// child document can query at document_start before armTab's frame
// snapshot sees it, especially when Chromium promotes it to an OOPIF.
tabIds: new Set([initialTabId]),
agents: new Map(),
...(onDocumentReady ? { onDocumentReady } : {}),
finishing: false,
});
}
async armTab(requestId: string, tabId: number): Promise<boolean> {
const recording = this.#recordings.get(requestId);
if (!recording || recording.finishing) return false;
recording.tabIds.add(tabId);
let frames: BrowserFrame[];
try {
frames = await this.#deps.getAllFrames(tabId);
} catch {
frames = [{ frameId: 0 }];
}
if (!frames.some((frame) => frame.frameId === 0)) frames.unshift({ frameId: 0 });
const results = await Promise.all(
frames.map(async (frame) => {
const started = await this.#startFrame(recording, tabId, frame);
return { frameId: frame.frameId, started };
}),
);
return results.some((result) => result.frameId === 0 && result.started);
}
sourceFor(
requestId: string,
producerId: string,
sender: chrome.runtime.MessageSender,
): RecordingCaptureScope | null {
const address = senderAddress(sender);
const recording = this.#recordings.get(requestId);
if (!address || !recording) return null;
const agent = recording.agents.get(documentKey(address.tabId, address.documentId));
if (!agent || agent.producerId !== producerId) return null;
return {
tabId: agent.tabId,
documentId: agent.documentId,
browserFrameId: agent.browserFrameId,
producerId: agent.producerId,
};
}
async stop(requestId: string): Promise<boolean> {
const recording = this.#recordings.get(requestId);
if (!recording) return true;
recording.finishing = true;
const commandId = crypto.randomUUID();
const results = await Promise.all(
[...recording.agents.values()].map((agent) => this.#stopAgent(agent, requestId, commandId)),
);
const succeeded = results.every(Boolean);
if (succeeded) this.#recordings.delete(requestId);
else recording.finishing = false;
return succeeded;
}
cancel(requestId: string): void {
const recording = this.#recordings.get(requestId);
if (!recording) return;
this.#cancelAgents(recording);
this.#recordings.delete(requestId);
}
readonly #onMessage = (
message: unknown,
sender: chrome.runtime.MessageSender,
sendResponse: (response: RecordFrameQueryResponse) => void,
): boolean => {
if (!isRecordFrameQueryMessage(message)) return false;
const tabId = sender.tab?.id;
const recording =
typeof tabId === "number"
? [...this.#recordings.values()].find(
(candidate) => !candidate.finishing && candidate.tabIds.has(tabId),
)
: undefined;
sendResponse(
recording
? {
active: true,
requestId: recording.requestId,
startedAtMs: recording.startedAtMs,
}
: { active: false },
);
return false;
};
readonly #onFrameNavigation = (frame: BrowserFrameNavigation): void => {
if (frame.frameId === 0) return;
for (const recording of this.#recordings.values()) {
if (recording.finishing || !recording.tabIds.has(frame.tabId)) continue;
// onCommitted can race document_start, while onCompleted runs after the
// content script is available. Starting twice is safe and closes both
// the early-query and late-SPA-frame gaps.
void (async () => {
const started = await this.#startFrame(recording, frame.tabId, frame);
if (!started || frame.lifecycle !== "completed" || recording.finishing) return;
const agent = frame.documentId
? recording.agents.get(documentKey(frame.tabId, frame.documentId))
: [...recording.agents.values()].find(
(candidate) =>
candidate.tabId === frame.tabId && candidate.browserFrameId === frame.frameId,
);
if (!agent) return;
await recording.onDocumentReady?.({
tabId: agent.tabId,
documentId: agent.documentId,
browserFrameId: agent.browserFrameId,
producerId: agent.producerId,
});
})();
}
};
readonly #onConnect = (port: chrome.runtime.Port): void => {
if (port.name !== RECORD_FRAME_PORT) return;
const address = senderAddress(port.sender ?? {});
if (!address) {
port.disconnect();
return;
}
let registered: FrameAgent | null = null;
port.onMessage.addListener((raw: unknown) => {
const message = raw as Partial<RecordFramePortMessage>;
if (message.type === "ready") {
if (typeof message.requestId !== "string" || typeof message.producerId !== "string") {
port.disconnect();
return;
}
const requestId = message.requestId;
const producerId = message.producerId;
if (registered) {
if (registered.requestId === requestId && registered.producerId === producerId) {
port.postMessage({
type: "ready_ack",
requestId,
producerId,
} satisfies RecordFramePortMessage);
} else {
port.disconnect();
}
return;
}
const recording = this.#recordings.get(requestId);
if (
!recording ||
recording.finishing ||
!recording.tabIds.has(address.tabId) ||
producerId.length === 0
) {
port.disconnect();
return;
}
const agent: FrameAgent = {
...address,
requestId,
producerId,
port,
stopWaiters: new Map(),
};
registered = agent;
const key = documentKey(address.tabId, address.documentId);
const previous = recording.agents.get(key);
if (previous && previous !== agent) previous.port.disconnect();
recording.agents.set(key, agent);
port.postMessage({
type: "ready_ack",
requestId,
producerId,
} satisfies RecordFramePortMessage);
return;
}
if (
message.type === "stopped" &&
registered &&
message.requestId === registered.requestId &&
typeof message.commandId === "string"
) {
registered.stopWaiters.get(message.commandId)?.(message.ok === true);
registered.stopWaiters.delete(message.commandId);
}
});
port.onDisconnect.addListener(() => {
if (!registered) return;
for (const resolve of registered.stopWaiters.values()) resolve(true);
const recording = this.#recordings.get(registered.requestId);
const key = documentKey(registered.tabId, registered.documentId);
if (recording?.agents.get(key) === registered) recording.agents.delete(key);
});
};
#stopAgent(agent: FrameAgent, requestId: string, commandId: string): Promise<boolean> {
return new Promise((resolve) => {
const timer = setTimeout(() => {
agent.stopWaiters.delete(commandId);
resolve(false);
}, RECORD_FRAME_STOP_TIMEOUT_MS);
agent.stopWaiters.set(commandId, (ok) => {
clearTimeout(timer);
resolve(ok);
});
try {
agent.port.postMessage({
type: "stop",
requestId,
commandId,
} satisfies RecordFramePortMessage);
} catch {
clearTimeout(timer);
agent.stopWaiters.delete(commandId);
resolve(true);
}
});
}
async #startFrame(
recording: ArmedRecording,
tabId: number,
frame: BrowserFrame,
): Promise<boolean> {
const message: RecordFrameStartMessage = {
type: RECORD_FRAME_START,
requestId: recording.requestId,
startedAtMs: recording.startedAtMs,
};
try {
const response = await this.#deps.sendToDocument(tabId, message, {
...(frame.documentId ? { documentId: frame.documentId } : { frameId: frame.frameId }),
});
return isStarted(response);
} catch {
return false;
}
}
#cancelAgents(recording: ArmedRecording): void {
for (const agent of recording.agents.values()) {
try {
agent.port.postMessage({
type: "cancel",
requestId: recording.requestId,
} satisfies RecordFramePortMessage);
} catch {
// A destroyed Document has no remaining capture state to cancel.
}
}
}
}
function isStarted(value: unknown): boolean {
return typeof value === "object" && value !== null && (value as { ok?: unknown }).ok === true;
}
export const recordFrameCoordinator = new RecordFrameCoordinator();
@@ -0,0 +1,5 @@
const REDIRECT_QUALIFIERS = new Set(["client_redirect", "server_redirect"]);
export function hasRedirectQualifier(qualifiers?: readonly string[]): boolean {
return (qualifiers ?? []).some((qualifier) => REDIRECT_QUALIFIERS.has(qualifier));
}
@@ -0,0 +1,140 @@
import type { RenderedRef } from "@browser-skill/vom";
import type { CdpTarget } from "@/browser-driver/frame-graph";
import {
type CaptureVomMatchNode,
type CaptureVomObservationResult,
captureVomObservation,
} from "@/tools/capture-vom-observation";
import type { CdpRunner, ChromeTabsApi } from "@/tools/shared";
export interface IndexedObservationNode {
frameId: string;
geometry: CaptureVomMatchNode;
ref?: RenderedRef;
}
export interface CapturedRecordingObservation {
rootFrameId: string;
index: ObservationNodeIndex;
url: string;
title?: string;
vomText: string;
truncated: boolean;
}
export interface RegisteredObservation {
stateId: string;
rootFrameId: string;
index: ObservationNodeIndex;
url: string;
}
export interface RecordingDocumentScope {
frameId: string;
target: CdpTarget;
}
function nodeKey(frameId: string, backendNodeId: number): string {
return `${frameId}:${backendNodeId}`;
}
function frameTagKey(frameId: string, tag: string): string {
return `${frameId}:${tag.toLowerCase()}`;
}
export class ObservationNodeIndex {
readonly #nodesByFrameTag = new Map<string, IndexedObservationNode[]>();
readonly #refById = new Map<string, RenderedRef>();
readonly #refsByFrame = new Map<string, RenderedRef[]>();
readonly #scopeByProducer = new Map<string, RecordingDocumentScope | null>();
constructor(
input: Pick<CaptureVomObservationResult, "rootFrameId" | "matchNodes" | "refs"> &
Partial<Pick<CaptureVomObservationResult, "frames">>,
) {
const refByNode = new Map<string, RenderedRef>();
for (const ref of input.refs) {
const frameId = ref.frameId ?? input.rootFrameId;
refByNode.set(nodeKey(frameId, ref.backendNodeId), ref);
this.#refById.set(ref.ref, ref);
const frameRefs = this.#refsByFrame.get(frameId) ?? [];
frameRefs.push(ref);
this.#refsByFrame.set(frameId, frameRefs);
}
for (const frame of input.frames ?? []) {
const documentId = frame.recordingDocumentId;
if (!documentId) continue;
this.#scopeByProducer.set(
documentId,
this.#scopeByProducer.has(documentId)
? null
: { frameId: frame.frameId, target: frame.target },
);
}
for (const geometry of input.matchNodes) {
const { frameId } = geometry;
const entry = {
frameId,
geometry,
ref: refByNode.get(nodeKey(frameId, geometry.backendNodeId)),
};
const key = frameTagKey(frameId, geometry.tag);
const bucket = this.#nodesByFrameTag.get(key) ?? [];
bucket.push(entry);
this.#nodesByFrameTag.set(key, bucket);
}
}
candidates(frameId: string, tag: string): readonly IndexedObservationNode[] {
return this.#nodesByFrameTag.get(frameTagKey(frameId, tag)) ?? [];
}
ref(refId: string): RenderedRef | undefined {
return this.#refById.get(refId);
}
refs(frameId: string): readonly RenderedRef[] {
return this.#refsByFrame.get(frameId) ?? [];
}
documentScope(producerId: string): RecordingDocumentScope | undefined {
return this.#scopeByProducer.get(producerId) ?? undefined;
}
}
async function readTabMeta(
tabsApi: ChromeTabsApi,
tabId: number,
): Promise<{ url: string; title?: string }> {
try {
const tab = await tabsApi.get(tabId);
return { url: tab.url ?? "about:blank", title: tab.title };
} catch {
return { url: "about:blank" };
}
}
export async function captureRecordingObservation(input: {
cdp: CdpRunner;
tabsApi: ChromeTabsApi;
tabId: number;
maxTokens: number;
redactValues: boolean;
signal?: AbortSignal;
}): Promise<CapturedRecordingObservation> {
const { url, title } = await readTabMeta(input.tabsApi, input.tabId);
const captured = await captureVomObservation(input.cdp, input.tabId, url, {
maxTokens: input.maxTokens,
redactValues: input.redactValues,
conditionalSurfaceProbe: false,
signal: input.signal,
});
return {
rootFrameId: captured.rootFrameId,
index: new ObservationNodeIndex(captured),
url,
title,
vomText: captured.text,
truncated: captured.truncated,
};
}
@@ -0,0 +1,128 @@
import type { CdpRunner, ChromeTabsApi } from "@/tools/shared";
import { captureRecordingObservation, type RegisteredObservation } from "./observation-capture";
import { RecordingStateRegistry } from "./state-registry";
import { matchObservationTarget, unmatchedTarget } from "./target-matcher";
import type { RecordingDraftStep, StepAnnotation, TargetedRecordingDraft } from "./types";
const DEFAULT_MAX_PAGE_TOKENS = 3_000;
const MIN_CAPTURE_INTERVAL_MS = 200;
export interface TabObservationCursor {
lastSettled: RegisteredObservation | null;
lastCaptureAt: number;
}
function abortableDelay(ms: number, signal?: AbortSignal): Promise<void> {
return new Promise((resolve) => {
if (signal?.aborted) return resolve();
const timer = setTimeout(resolve, ms);
signal?.addEventListener(
"abort",
() => {
clearTimeout(timer);
resolve();
},
{ once: true },
);
});
}
function isTargeted(draft: RecordingDraftStep): draft is TargetedRecordingDraft {
return (
draft.op === "click" ||
draft.op === "hover" ||
draft.op === "fill" ||
draft.op === "press" ||
draft.op === "select"
);
}
export class RecordingObservationSession {
readonly registry: RecordingStateRegistry;
readonly cursor: TabObservationCursor;
readonly annotations: StepAnnotation[];
readonly #maxTokens: number;
readonly #redactValues: boolean;
constructor(
options: {
registry?: RecordingStateRegistry;
cursor?: TabObservationCursor;
annotations?: StepAnnotation[];
maxTokens?: number;
redactValues?: boolean;
} = {},
) {
this.registry = options.registry ?? new RecordingStateRegistry();
this.cursor = options.cursor ?? { lastSettled: null, lastCaptureAt: 0 };
this.annotations = options.annotations ?? [];
this.#maxTokens = options.maxTokens ?? DEFAULT_MAX_PAGE_TOKENS;
this.#redactValues = options.redactValues ?? false;
}
async capture(
cdp: CdpRunner,
tabsApi: ChromeTabsApi,
tabId: number,
signal?: AbortSignal,
): Promise<RegisteredObservation> {
const waitMs = Math.max(0, MIN_CAPTURE_INTERVAL_MS - (Date.now() - this.cursor.lastCaptureAt));
if (waitMs > 0) await abortableDelay(waitMs, signal);
if (signal?.aborted) throw new DOMException("observation aborted", "AbortError");
const captured = await captureRecordingObservation({
cdp,
tabsApi,
tabId,
maxTokens: this.#maxTokens,
redactValues: this.#redactValues,
signal,
});
const state = this.registry.register({
url: captured.url,
title: captured.title,
vomText: captured.vomText,
truncated: captured.truncated,
});
const observation: RegisteredObservation = {
stateId: state.id,
rootFrameId: captured.rootFrameId,
index: captured.index,
url: captured.url,
};
this.cursor.lastSettled = observation;
this.cursor.lastCaptureAt = Date.now();
return observation;
}
bindDraft(draft: RecordingDraftStep, draftId: number, previousActionPending = false): void {
const observation = this.cursor.lastSettled;
if (isTargeted(draft)) {
draft.matchedTarget = observation
? matchObservationTarget({
observation,
hint: draft.targetHint,
fallback: draft.captureTarget,
})
: unmatchedTarget(draft.captureTarget);
}
if (!observation) return;
const unmatched = isTargeted(draft) && draft.matchedTarget?.unmatched === true;
if (previousActionPending && unmatched) return;
draft.preStateId = observation.stateId;
this.registry.markStep(observation.stateId, draftId);
if (!isTargeted(draft) || !draft.matchedTarget?.ref) return;
const ref = observation.index.ref(draft.matchedTarget.ref);
if (!ref) return;
this.annotations.push({
draftId,
op: draft.op,
line: ref.line,
stateId: observation.stateId,
...(draft.op === "fill" && !this.#redactValues
? { detail: JSON.stringify(draft.value) }
: {}),
});
}
}
@@ -0,0 +1,277 @@
import type { CdpRunner, ChromeTabsApi } from "@/tools/shared";
import type { StopReason, TraceV3 } from "@/transport/types";
import { type DocumentSettleScope, waitForDocumentSettled } from "./document-settle";
import type { RegisteredObservation } from "./observation-capture";
import { RecordingObservationSession } from "./observation-session";
import { inferMissingPostStates, SettleController } from "./settle-controller";
import { RecordingStateRegistry } from "./state-registry";
import { buildTraceV3 } from "./trace-builder-v3";
import type { RecordingDraftStep, StepAnnotation } from "./types";
interface TabRecordingContext {
session: RecordingObservationSession;
settle: SettleController;
pendingCapture: PendingCapture | null;
frameRefresh: Promise<void>;
frameRefreshAbort: AbortController | null;
}
interface PendingCapture {
promise: Promise<RegisteredObservation>;
abort: AbortController;
}
export class RecordingObservationRuntime {
readonly #cdp: CdpRunner;
readonly #tabsApi: ChromeTabsApi;
readonly #registry = new RecordingStateRegistry();
readonly #annotations: StepAnnotation[] = [];
readonly #contexts = new Map<number, TabRecordingContext>();
readonly #maxTokens?: number;
readonly #redactValues: boolean;
constructor(input: {
cdp: CdpRunner;
tabsApi: ChromeTabsApi;
maxTokens?: number;
redactValues?: boolean;
}) {
this.#cdp = input.cdp;
this.#tabsApi = input.tabsApi;
this.#maxTokens = input.maxTokens;
this.#redactValues = input.redactValues ?? false;
}
#context(tabId: number): TabRecordingContext {
const existing = this.#contexts.get(tabId);
if (existing) return existing;
const session = new RecordingObservationSession({
registry: this.#registry,
annotations: this.#annotations,
maxTokens: this.#maxTokens,
redactValues: this.#redactValues,
});
const context: TabRecordingContext = {
session,
settle: new SettleController({
session,
cdp: this.#cdp,
tabsApi: this.#tabsApi,
tabId,
}),
pendingCapture: null,
frameRefresh: Promise.resolve(),
frameRefreshAbort: null,
};
this.#contexts.set(tabId, context);
return context;
}
async #capture(tabId: number): Promise<RegisteredObservation> {
const context = this.#context(tabId);
if (context.pendingCapture) return context.pendingCapture.promise;
const abort = new AbortController();
let pending!: PendingCapture;
const promise = context.session.capture(this.#cdp, this.#tabsApi, tabId, abort.signal);
pending = {
abort,
promise: promise.finally(() => {
if (context.pendingCapture === pending) context.pendingCapture = null;
}),
};
context.pendingCapture = pending;
return pending.promise;
}
async captureInitial(tabId: number): Promise<void> {
const context = this.#context(tabId);
if (context.session.cursor.lastSettled) return;
await this.#capture(tabId);
}
/** Refresh the safe observation after a late child Document becomes usable. */
async refreshDocument(tabId: number, producerId: string): Promise<void> {
const context = this.#context(tabId);
context.frameRefreshAbort?.abort();
const abort = new AbortController();
context.frameRefreshAbort = abort;
const previous = context.frameRefresh;
const refresh = (async () => {
await previous;
if (abort.signal.aborted) return;
if (context.pendingCapture) await Promise.allSettled([context.pendingCapture.promise]);
if (abort.signal.aborted) return;
// The first capture discovers the safe producer -> CDP Document scope.
await this.#capture(tabId);
if (abort.signal.aborted) return;
const scope = context.session.cursor.lastSettled?.index.documentScope(producerId);
if (scope) {
await waitForDocumentSettled(this.#cdp, scope, { signal: abort.signal });
}
if (abort.signal.aborted) return;
// Capture again after the child SPA is quiet so the next action can be
// matched against refs that actually exist in its pre-action state.
await this.#capture(tabId);
})();
const tracked = refresh.catch(() => {});
context.frameRefresh = tracked;
try {
await refresh;
} finally {
if (context.frameRefresh === tracked) context.frameRefreshAbort = null;
}
}
async captureTabTransition(
fromTabId: number,
toTabId: number,
): Promise<{ preStateId?: string; postStateId?: string; targetUrl?: string }> {
const from = this.#context(fromTabId);
await this.#flushContext(from);
if (!from.session.cursor.lastSettled) {
try {
await this.captureInitial(fromTabId);
} catch {
// A target-tab observation can still preserve the forward transition.
}
}
const to = this.#context(toTabId);
let target = to.session.cursor.lastSettled;
try {
target = await this.#capture(toTabId);
} catch {
// Reuse the last settled target state when a refresh races navigation.
}
const source = from.session.cursor.lastSettled;
return {
...(source ? { preStateId: source.stateId } : {}),
...(target ? { postStateId: target.stateId, targetUrl: target.url } : {}),
};
}
bindTabTransition(
draft: Extract<RecordingDraftStep, { op: "switch_tab" }>,
draftId: number,
): void {
if (draft.preStateId) this.#registry.markStep(draft.preStateId, draftId);
}
async processDraft(
tabId: number,
drafts: RecordingDraftStep[],
draftIndex: number,
producerId?: string,
): Promise<void> {
const draft = drafts[draftIndex];
if (!draft) return;
const context = this.#context(tabId);
if (!context.session.cursor.lastSettled) {
try {
await this.captureInitial(tabId);
} catch {
// Post-action settle can still provide a usable state.
}
}
const scope = producerId
? context.session.cursor.lastSettled?.index.documentScope(producerId)
: undefined;
if (
producerId &&
(draft.op === "click" ||
draft.op === "hover" ||
draft.op === "fill" ||
draft.op === "press" ||
draft.op === "select")
) {
draft.targetHint = {
...(draft.targetHint ?? {}),
frameId: scope?.frameId ?? null,
geometrySpace: "local",
};
}
context.session.bindDraft(draft, draftIndex + 1, context.settle.hasPending);
context.settle.schedule(drafts, draftIndex, scope);
}
scheduleSettle(
tabId: number,
drafts: RecordingDraftStep[],
draftIndex: number,
scope?: DocumentSettleScope,
): void {
this.#context(tabId).settle.schedule(drafts, draftIndex, scope);
}
clearRedirect(tabId: number): void {
this.#contexts.get(tabId)?.settle.clearRedirect();
}
scheduleRedirect(tabId: number, drafts: RecordingDraftStep[], url: string): void {
this.#context(tabId).settle.scheduleRedirect(drafts, url);
}
async #flushContext(context: TabRecordingContext): Promise<void> {
await context.frameRefresh;
if (context.pendingCapture) await Promise.allSettled([context.pendingCapture.promise]);
await context.settle.flushRedirects();
await context.settle.flush();
}
async flushRedirects(tabId: number): Promise<void> {
await this.#contexts.get(tabId)?.settle.flushRedirects();
}
async flush(): Promise<void> {
for (const context of this.#contexts.values()) await this.#flushContext(context);
}
async settleTrailing(tabId: number, drafts: RecordingDraftStep[]): Promise<void> {
const context = this.#context(tabId);
if (context.pendingCapture) await Promise.allSettled([context.pendingCapture.promise]);
if (!context.session.cursor.lastSettled) {
try {
await this.captureInitial(tabId);
} catch {
// The trace may remain observation-free when CDP is unavailable.
}
}
await context.settle.settleTrailing(drafts);
inferMissingPostStates(drafts);
}
cancel(): void {
for (const context of this.#contexts.values()) {
context.frameRefreshAbort?.abort();
context.pendingCapture?.abort.abort();
context.settle.cancel();
}
}
buildTrace(input: {
drafts: RecordingDraftStep[];
startedAt: string;
purpose?: string;
startUrl?: string;
stoppedBy: StopReason;
bskVersion: string;
includeTabSwitches?: boolean;
}): TraceV3 {
return buildTraceV3({
registry: this.#registry,
drafts: input.drafts,
annotations: this.#annotations,
startedAt: input.startedAt,
purpose: input.purpose,
startUrl: input.startUrl,
stoppedBy: input.stoppedBy,
bskVersion: input.bskVersion,
redactValues: this.#redactValues,
includeTabSwitches: input.includeTabSwitches,
});
}
}
@@ -0,0 +1,233 @@
import type { CdpRunner, ChromeTabsApi } from "@/tools/shared";
import { type DocumentSettleScope, waitForDocumentSettled } from "./document-settle";
import { RecordingObservationSession } from "./observation-session";
import type { RecordingDraftStep } from "./types";
interface PendingSettle {
abort: AbortController;
scope: DocumentSettleScope;
}
interface PendingRedirect {
url: string;
abort: AbortController;
}
const CAPTURE_RETRY_DELAY_MS = 250;
function delay(ms: number, signal?: AbortSignal): Promise<void> {
return new Promise((resolve) => {
if (signal?.aborted) return resolve();
const timer = setTimeout(resolve, ms);
signal?.addEventListener(
"abort",
() => {
clearTimeout(timer);
resolve();
},
{ once: true },
);
});
}
function isAbortError(error: unknown): boolean {
return (
typeof error === "object" &&
error !== null &&
(error as { name?: string }).name === "AbortError"
);
}
export function inferMissingPostStates(drafts: RecordingDraftStep[]): void {
for (let index = 0; index < drafts.length - 1; index += 1) {
const draft = drafts[index];
const next = drafts[index + 1];
if (draft && next && !draft.postStateId && next.preStateId) {
draft.postStateId = next.preStateId;
}
}
}
export class SettleController {
readonly #session: RecordingObservationSession;
readonly #cdp: CdpRunner;
readonly #tabsApi: ChromeTabsApi;
readonly #tabId: number;
readonly #rootScope: DocumentSettleScope;
readonly #pending = new Map<number, PendingSettle>();
#queue = Promise.resolve();
#pendingRedirect: PendingRedirect | null = null;
#redirectQueue = Promise.resolve();
constructor(input: {
session: RecordingObservationSession;
cdp: CdpRunner;
tabsApi: ChromeTabsApi;
tabId: number;
}) {
this.#session = input.session;
this.#cdp = input.cdp;
this.#tabsApi = input.tabsApi;
this.#tabId = input.tabId;
this.#rootScope = { target: { tabId: input.tabId } };
}
get hasPending(): boolean {
return this.#pending.size > 0 || this.#pendingRedirect !== null;
}
async #captureWithRetry(signal?: AbortSignal) {
try {
return await this.#session.capture(this.#cdp, this.#tabsApi, this.#tabId, signal);
} catch (error) {
if (signal?.aborted) throw error;
await delay(CAPTURE_RETRY_DELAY_MS, signal);
return this.#session.capture(this.#cdp, this.#tabsApi, this.#tabId, signal);
}
}
schedule(
drafts: RecordingDraftStep[],
draftIndex: number,
scope: DocumentSettleScope = this.#rootScope,
): void {
if (scope.target.tabId !== this.#tabId) {
throw new Error(
`settle scope tab ${scope.target.tabId} does not belong to tab ${this.#tabId}`,
);
}
const landing = drafts[draftIndex]?.preStateId;
for (const [index, pending] of this.#pending) {
if (index >= draftIndex) continue;
pending.abort.abort();
this.#pending.delete(index);
if (landing && drafts[index] && !drafts[index].postStateId) {
drafts[index].postStateId = landing;
}
}
this.#pending.get(draftIndex)?.abort.abort();
const pending = { abort: new AbortController(), scope };
this.#pending.set(draftIndex, pending);
const task = async () => {
if (pending.abort.signal.aborted) return;
try {
const outcome = await waitForDocumentSettled(this.#cdp, pending.scope, {
signal: pending.abort.signal,
});
if (outcome === "cancelled") return;
const observation = await this.#captureWithRetry(pending.abort.signal);
if (!pending.abort.signal.aborted && drafts[draftIndex]) {
drafts[draftIndex].postStateId = observation.stateId;
}
} catch (error) {
if (!isAbortError(error)) {
console.warn(
`[bsk record] post-action observation failed for step ${draftIndex + 1}`,
error,
);
}
} finally {
if (this.#pending.get(draftIndex) === pending) this.#pending.delete(draftIndex);
}
};
this.#queue = this.#queue.then(task, task).catch(() => {});
}
cancel(): void {
for (const pending of this.#pending.values()) pending.abort.abort();
this.#pending.clear();
this.clearRedirect();
}
async flush(): Promise<void> {
for (let round = 0; round < 10; round += 1) {
const current = this.#queue;
await current;
if (current === this.#queue) return;
}
console.warn("[bsk record] settle queue kept growing; using completed observations");
}
async settleTrailing(drafts: RecordingDraftStep[]): Promise<void> {
const trailing: RecordingDraftStep[] = [];
for (let index = drafts.length - 1; index >= 0; index -= 1) {
const draft = drafts[index];
if (!draft || draft.postStateId) break;
trailing.push(draft);
}
if (trailing.length === 0) return;
try {
const observation = await this.#captureWithRetry();
for (const draft of trailing) draft.postStateId = observation.stateId;
} catch (error) {
console.warn("[bsk record] final observation at stop failed", error);
}
}
clearRedirect(): void {
this.#pendingRedirect?.abort.abort();
this.#pendingRedirect = null;
}
scheduleRedirect(drafts: RecordingDraftStep[], url: string): void {
this.#pendingRedirect?.abort.abort();
const pending: PendingRedirect = { url, abort: new AbortController() };
this.#pendingRedirect = pending;
const task = () => this.#settleRedirect(drafts, pending);
this.#redirectQueue = this.#redirectQueue.then(task, task).catch(() => {});
}
async #settleRedirect(drafts: RecordingDraftStep[], pending: PendingRedirect): Promise<void> {
const outcome = await waitForDocumentSettled(this.#cdp, this.#rootScope, {
signal: pending.abort.signal,
});
if (outcome === "cancelled" || this.#pendingRedirect !== pending) return;
let finalUrl = pending.url;
try {
finalUrl = (await this.#tabsApi.get(this.#tabId)).url || finalUrl;
} catch {
// The navigation event URL remains the best available destination.
}
if (this.#pendingRedirect !== pending) return;
this.#pendingRedirect = null;
if (
!finalUrl ||
finalUrl === "about:blank" ||
this.#session.cursor.lastSettled?.url === finalUrl
) {
return;
}
const last = drafts[drafts.length - 1];
if (last?.op === "navigate" && last.url === finalUrl) {
if (!last.postStateId) this.schedule(drafts, drafts.length - 1);
await this.flush();
return;
}
const draft: RecordingDraftStep = {
op: "navigate",
url: finalUrl,
pageUrl: finalUrl,
cause: "browser",
transitionQualifiers: ["server_redirect"],
preStateId: this.#session.cursor.lastSettled?.stateId,
};
drafts.push(draft);
const draftIndex = drafts.length - 1;
if (draft.preStateId) this.#session.registry.markStep(draft.preStateId, draftIndex + 1);
this.schedule(drafts, draftIndex);
await this.flush();
}
async flushRedirects(): Promise<void> {
for (let round = 0; round < 10; round += 1) {
const current = this.#redirectQueue;
await current;
if (current === this.#redirectQueue) return;
}
console.warn("[bsk record] redirect queue kept growing; using the final completed landing");
}
}
@@ -0,0 +1,57 @@
export interface RecordedStateEntry {
id: string;
url: string;
title?: string;
/** Rendered VOM text; capture indexes and source DOM/AX data never enter the registry. */
vomText: string;
truncated: boolean;
stepsHere: number[];
}
function stateIdentity(url: string, body: string): string {
return `${url}\0${body}`;
}
export class RecordingStateRegistry {
readonly #entriesById = new Map<string, RecordedStateEntry>();
readonly #idByIdentity = new Map<string, string>();
#nextId = 1;
register(input: {
url: string;
title?: string;
vomText: string;
truncated?: boolean;
}): RecordedStateEntry {
const identity = stateIdentity(input.url, input.vomText);
const existingId = this.#idByIdentity.get(identity);
if (existingId) {
const existing = this.#entriesById.get(existingId)!;
if (!existing.title && input.title) existing.title = input.title;
if (input.truncated) existing.truncated = true;
return existing;
}
const entry: RecordedStateEntry = {
id: `s${this.#nextId}`,
url: input.url,
...(input.title ? { title: input.title } : {}),
vomText: input.vomText,
truncated: input.truncated ?? false,
stepsHere: [],
};
this.#nextId += 1;
this.#entriesById.set(entry.id, entry);
this.#idByIdentity.set(identity, entry.id);
return entry;
}
values(): RecordedStateEntry[] {
return [...this.#entriesById.values()];
}
markStep(stateId: string, draftId: number): void {
const entry = this.#entriesById.get(stateId);
if (entry && !entry.stepsHere.includes(draftId)) entry.stepsHere.push(draftId);
}
}
@@ -0,0 +1,155 @@
import type { RecordStepPayload } from "../record-bridge";
import { hasRedirectQualifier } from "./navigation-policy";
import type { RecordingDraftStep, TargetMatchHint } from "./types";
export interface RecordingStepBuffer {
steps: RecordingDraftStep[];
navigation: RecordingNavigationCursor;
}
export interface RecordingNavigationCursor {
currentUrl?: string;
pendingNavigation: boolean;
pendingNavigationDeadline?: number;
}
const NAVIGATION_TRIGGER_WINDOW_MS = 3_000;
function toDraftStep(
payload: RecordStepPayload,
targetHint?: TargetMatchHint,
): RecordingDraftStep | null {
const pageUrl = payload.page_url;
const common = {
...(pageUrl ? { pageUrl } : {}),
...(targetHint ? { targetHint } : {}),
};
switch (payload.op) {
case "click":
return payload.target ? { op: "click", captureTarget: payload.target, ...common } : null;
case "hover":
return payload.target ? { op: "hover", captureTarget: payload.target, ...common } : null;
case "fill":
return payload.target
? {
op: "fill",
captureTarget: payload.target,
value: payload.value ?? "",
...(payload.commit ? { commit: payload.commit } : {}),
...(payload.redacted ? { redacted: true } : {}),
...common,
}
: null;
case "press":
return payload.key
? {
op: "press",
key: payload.key,
...(payload.target ? { captureTarget: payload.target } : {}),
...(payload.modifiers?.length ? { modifiers: payload.modifiers } : {}),
...common,
}
: null;
case "select":
return payload.target && payload.values
? {
op: "select",
captureTarget: payload.target,
values: payload.values,
...(payload.labels?.length ? { labels: payload.labels } : {}),
...common,
}
: null;
case "navigate":
return null;
}
}
function annotateLastStepNavigation(buffer: RecordingStepBuffer, url: string): number {
for (let index = buffer.steps.length - 1; index >= 0; index -= 1) {
const step = buffer.steps[index];
if (!step) continue;
if (step.op === "click" || step.op === "press" || step.op === "select" || step.op === "fill") {
step.navigatedTo = url;
return index;
}
break;
}
return -1;
}
export type NavigationObserveResult =
| { kind: "noop" }
| { kind: "annotated"; index: number }
| { kind: "appended"; index: number }
| { kind: "coalesce_redirect"; url: string };
export function observeRecordedNavigation(
buffer: RecordingStepBuffer,
url: string,
causedByAction?: boolean,
transitionType?: string,
transitionQualifiers?: string[],
): NavigationObserveResult {
const navigation = buffer.navigation;
if (!url || url === navigation.currentUrl) return { kind: "noop" };
navigation.currentUrl = url;
const pendingIsCurrent =
navigation.pendingNavigation &&
(navigation.pendingNavigationDeadline === undefined ||
navigation.pendingNavigationDeadline >= Date.now());
if (causedByAction === true || (causedByAction === undefined && pendingIsCurrent)) {
navigation.pendingNavigation = false;
navigation.pendingNavigationDeadline = undefined;
const annotatedIndex = annotateLastStepNavigation(buffer, url);
if (annotatedIndex >= 0) return { kind: "annotated", index: annotatedIndex };
} else {
navigation.pendingNavigation = false;
navigation.pendingNavigationDeadline = undefined;
}
if (hasRedirectQualifier(transitionQualifiers)) {
return { kind: "coalesce_redirect", url };
}
buffer.steps.push({
op: "navigate",
url,
pageUrl: url,
transitionType,
transitionQualifiers,
});
return { kind: "appended", index: buffer.steps.length - 1 };
}
export function appendRecordedPayload(
buffer: RecordingStepBuffer,
payload: RecordStepPayload,
targetHint?: TargetMatchHint,
): number | null {
if (payload.op === "navigate") {
if (!payload.url) return null;
const result = observeRecordedNavigation(
buffer,
payload.url,
payload.navigation_caused_by_action,
payload.transitionType,
payload.transitionQualifiers,
);
return result.kind === "appended" ? result.index : null;
}
const step = toDraftStep(
{ ...payload, page_url: payload.page_url ?? buffer.navigation.currentUrl },
targetHint,
);
if (!step) return null;
buffer.steps.push(step);
if (step.op === "click" || step.op === "press" || step.op === "select" || step.op === "fill") {
buffer.navigation.pendingNavigation = payload.expects_navigation === true;
buffer.navigation.pendingNavigationDeadline = buffer.navigation.pendingNavigation
? Date.now() + NAVIGATION_TRIGGER_WINDOW_MS
: undefined;
}
return buffer.steps.length - 1;
}
@@ -0,0 +1,61 @@
import type { RecordingNavigationCursor } from "./step-buffer";
export interface TabActivation {
tabId: number;
revision: number;
}
export class RecordingTabCoordinator {
readonly #navigationByTab = new Map<number, RecordingNavigationCursor>();
#activeTabId: number;
#currentTabId: number;
#activationRevision = 0;
constructor(initialTabId: number, initialUrl?: string) {
this.#activeTabId = initialTabId;
this.#currentTabId = initialTabId;
this.#navigationByTab.set(initialTabId, {
currentUrl: initialUrl,
pendingNavigation: false,
});
}
get activeTabId(): number {
return this.#activeTabId;
}
get currentTabId(): number {
return this.#currentTabId;
}
noteActivation(tabId: number): TabActivation {
this.#activeTabId = tabId;
this.#activationRevision += 1;
return { tabId, revision: this.#activationRevision };
}
isLatest(activation: TabActivation): boolean {
return (
activation.tabId === this.#activeTabId && activation.revision === this.#activationRevision
);
}
commit(tabId: number, currentUrl?: string): void {
if (currentUrl !== undefined) this.navigation(tabId).currentUrl = currentUrl;
this.#currentTabId = tabId;
}
navigation(tabId: number, fallbackUrl?: string): RecordingNavigationCursor {
const existing = this.#navigationByTab.get(tabId);
if (existing) {
if (!existing.currentUrl && fallbackUrl) existing.currentUrl = fallbackUrl;
return existing;
}
const created: RecordingNavigationCursor = {
currentUrl: fallbackUrl,
pendingNavigation: false,
};
this.#navigationByTab.set(tabId, created);
return created;
}
}
@@ -0,0 +1,86 @@
import type { RenderedRef } from "@browser-skill/vom";
import type { TargetDescriptorV3 } from "@/transport/types";
import type { CaptureTargetDescriptor } from "../describe-target";
import type { IndexedObservationNode, RegisteredObservation } from "./observation-capture";
import type { TargetGeometry, TargetMatchHint } from "./types";
const MATCH_TOLERANCE_PX = 2;
function close(a: number, b: number): boolean {
return Math.abs(a - b) <= MATCH_TOLERANCE_PX;
}
function rectMatches(a: TargetGeometry["rect"], b: TargetGeometry["rect"]): boolean {
return close(a.x, b.x) && close(a.y, b.y) && close(a.w, b.w) && close(a.h, b.h);
}
function candidateMatches(
candidate: IndexedObservationNode,
geometry: TargetGeometry,
geometrySpace: "top" | "local",
): boolean {
const rect = geometrySpace === "local" ? candidate.geometry.localRect : candidate.geometry.rect;
return rect != null && rectMatches(geometry.rect, rect);
}
export function unmatchedTarget(fallback?: CaptureTargetDescriptor): TargetDescriptorV3 {
return {
...(fallback?.role ? { role: fallback.role } : {}),
...(fallback?.name ? { name: fallback.name } : {}),
unmatched: true,
};
}
function normalized(value: string | undefined): string {
return value?.replace(/\s+/g, " ").trim().toLowerCase() ?? "";
}
function matchesSemantics(ref: RenderedRef, fallback?: CaptureTargetDescriptor): boolean {
const role = normalized(fallback?.role);
const name = normalized(fallback?.name);
if (!role && !name) return false;
if (role && normalized(ref.role) !== role) return false;
if (name && normalized(ref.name) !== name) return false;
return true;
}
function descriptor(ref: RenderedRef): TargetDescriptorV3 {
return {
ref: ref.ref,
...(ref.role ? { role: ref.role } : {}),
...(ref.name ? { name: ref.name } : {}),
...(ref.ctx ? { ctx: ref.ctx } : {}),
};
}
export function matchObservationTarget(input: {
observation: RegisteredObservation;
hint?: TargetMatchHint;
fallback?: CaptureTargetDescriptor;
}): TargetDescriptorV3 {
if (input.hint?.frameId === null) return unmatchedTarget(input.fallback);
const frameId = input.hint?.frameId ?? input.observation.rootFrameId;
const geometry = input.hint?.geometry;
if (geometry) {
const matches = input.observation.index
.candidates(frameId, geometry.tag)
.filter(
(candidate) =>
candidate.ref &&
candidateMatches(candidate, geometry, input.hint?.geometrySpace ?? "top"),
);
if (matches.length === 1) return descriptor(matches[0]!.ref!);
const semanticMatches = matches.filter(
(candidate) => candidate.ref && matchesSemantics(candidate.ref, input.fallback),
);
if (semanticMatches.length === 1) return descriptor(semanticMatches[0]!.ref!);
return unmatchedTarget(input.fallback);
}
const semanticMatches = input.observation.index
.refs(frameId)
.filter((ref) => matchesSemantics(ref, input.fallback));
return semanticMatches.length === 1
? descriptor(semanticMatches[0]!)
: unmatchedTarget(input.fallback);
}
@@ -0,0 +1,92 @@
import {
type StepV3,
type StopReason,
TRACE_VERSION_V3,
type TraceStateV3,
type TraceV3,
VOM_FORMAT_VERSION,
} from "@/transport/types";
import { resolveDraftStartUrl } from "./draft-policy";
import type { RecordedStateEntry, RecordingStateRegistry } from "./state-registry";
import { reduceTraceStepsV3 } from "./trace-reducer-v3";
import { formatTraceStateBody } from "./trace-state-body";
import type { RecordingDraftStep, StepAnnotation } from "./types";
function publishedEntries(registry: RecordingStateRegistry, steps: StepV3[]): RecordedStateEntry[] {
const entries = registry.values();
if (steps.length === 0) return entries.slice(0, 1);
const referenced = new Set(steps.flatMap((step) => [step.state, step.result.state]));
return entries.filter((entry) => referenced.has(entry.id));
}
function remapDraftIds(draftIds: number[], stepIdByDraftId: Map<number, number>): number[] {
return [
...new Set(
draftIds.flatMap((id) => {
const stepId = stepIdByDraftId.get(id);
return stepId === undefined ? [] : [stepId];
}),
),
].sort((a, b) => a - b);
}
export function buildTraceV3(input: {
registry: RecordingStateRegistry;
drafts: RecordingDraftStep[];
annotations?: StepAnnotation[];
startedAt: string;
purpose?: string;
startUrl?: string;
stoppedBy: StopReason;
bskVersion: string;
redactValues?: boolean;
includeTabSwitches?: boolean;
}): TraceV3 {
const reduced = reduceTraceStepsV3(input.drafts, {
includeTabSwitches: input.includeTabSwitches,
redactValues: input.redactValues,
});
const entries = publishedEntries(input.registry, reduced.steps);
const publishedId = new Map(entries.map((entry, index) => [entry.id, `s${index + 1}`]));
const annotationsByState = new Map<string, StepAnnotation[]>();
for (const annotation of input.annotations ?? []) {
const bucket = annotationsByState.get(annotation.stateId) ?? [];
bucket.push(annotation);
annotationsByState.set(annotation.stateId, bucket);
}
const steps = reduced.steps.map((step) => ({
...step,
state: publishedId.get(step.state) ?? step.state,
result: { state: publishedId.get(step.result.state) ?? step.result.state },
}));
const states: TraceStateV3[] = entries.map((entry) => {
const id = publishedId.get(entry.id) ?? entry.id;
return {
id,
url: entry.url,
...(entry.title ? { title: entry.title } : {}),
body: formatTraceStateBody({
stateId: id,
url: entry.url,
title: entry.title,
stepIds: remapDraftIds(entry.stepsHere, reduced.stepIdByDraftId),
vomText: entry.vomText,
annotations: annotationsByState.get(entry.id) ?? [],
stepIdByDraftId: reduced.stepIdByDraftId,
}),
...(entry.truncated ? { truncated: true } : {}),
};
});
return {
version: TRACE_VERSION_V3,
...(input.purpose ? { purpose: input.purpose } : {}),
recorded_at: new Date().toISOString(),
started_at: input.startedAt,
stopped_by: input.stoppedBy,
entry: { start_url: resolveDraftStartUrl(input.drafts, input.startUrl, states[0]?.url) },
recorder: { bsk: input.bskVersion, vom: VOM_FORMAT_VERSION },
states,
steps,
};
}
@@ -1,30 +1,10 @@
import type { DraftTraceStep, PageRef, SelectedOption, Step } from "@/transport/types";
const CLIPBOARD_KEYS = new Set(["a", "c", "v", "x", "A", "C", "V", "X"]);
const MODIFIER_ONLY_KEYS = new Set(["Meta", "Control", "Alt", "Shift", "OS", "Hyper", "Super"]);
export function shouldRecordPress(
key: string,
modifiers?: Array<"alt" | "ctrl" | "meta" | "shift">,
): boolean {
if (MODIFIER_ONLY_KEYS.has(key)) return false;
const mods = modifiers ?? [];
const hasCtrlOrMeta = mods.includes("ctrl") || mods.includes("meta");
if (hasCtrlOrMeta && CLIPBOARD_KEYS.has(key)) return false;
if (key === "Enter" || key === "Escape") return true;
// Drop bare character typing — FillSession already records the value.
if (key.length === 1 && !hasCtrlOrMeta && !mods.includes("alt")) return false;
return false;
}
function shouldIncludeDraft(step: DraftTraceStep): boolean {
if (step.op === "press" && !shouldRecordPress(step.key, step.modifiers)) return false;
return true;
}
import type { PageRefV2, SelectedOptionV2, StepV2, TraceV2 } from "@/transport/types";
import { resolveDraftStartUrl, shouldIncludeDraft } from "./draft-policy";
import type { RecordingDraftStep } from "./types";
/** Collapse consecutive navigations to the last hop. */
function collapseNavigations(steps: DraftTraceStep[]): DraftTraceStep[] {
const out: DraftTraceStep[] = [];
function collapseNavigations(steps: RecordingDraftStep[]): RecordingDraftStep[] {
const out: RecordingDraftStep[] = [];
for (const step of steps) {
const prev = out[out.length - 1];
if (step.op === "navigate" && prev?.op === "navigate") {
@@ -36,7 +16,7 @@ function collapseNavigations(steps: DraftTraceStep[]): DraftTraceStep[] {
return out;
}
function collectUrls(steps: DraftTraceStep[], startUrl?: string): string[] {
function collectUrls(steps: RecordingDraftStep[], startUrl?: string): string[] {
const urls: string[] = [];
if (startUrl) urls.push(startUrl);
for (const step of steps) {
@@ -44,8 +24,8 @@ function collectUrls(steps: DraftTraceStep[], startUrl?: string): string[] {
urls.push(step.url);
continue;
}
if ("page_url" in step && step.page_url) urls.push(step.page_url);
if ("navigated_to" in step && step.navigated_to) urls.push(step.navigated_to);
if (step.pageUrl) urls.push(step.pageUrl);
if ("navigatedTo" in step && step.navigatedTo) urls.push(step.navigatedTo);
}
const seen = new Set<string>();
const unique: string[] = [];
@@ -58,9 +38,9 @@ function collectUrls(steps: DraftTraceStep[], startUrl?: string): string[] {
}
function buildPageRegistry(
steps: DraftTraceStep[],
steps: RecordingDraftStep[],
startUrl?: string,
): { pages: PageRef[]; urlToId: Map<string, string> } {
): { pages: PageRefV2[]; urlToId: Map<string, string> } {
const urls = collectUrls(steps, startUrl);
const urlToId = new Map<string, string>();
const pages = urls.map((url, index) => {
@@ -81,28 +61,28 @@ function pageIdFor(
return urlToId.values().next().value ?? "p1";
}
function pageUrlForDraft(step: DraftTraceStep, fallbackUrl?: string): string | undefined {
if (step.op === "navigate") return step.page_url ?? step.url;
if ("page_url" in step && step.page_url) return step.page_url;
function pageUrlForDraft(step: RecordingDraftStep, fallbackUrl?: string): string | undefined {
if (step.op === "navigate") return step.pageUrl ?? step.url;
if (step.pageUrl) return step.pageUrl;
return fallbackUrl;
}
function effectForNavigation(
navigatedTo: string | undefined,
urlToId: Map<string, string>,
): Step["effect"] {
): StepV2["effect"] {
if (!navigatedTo) return undefined;
const pageId = urlToId.get(navigatedTo);
if (!pageId) return undefined;
return { navigated_to: pageId };
}
function withEffect(step: Step, effect: Step["effect"]): Step {
function withEffect(step: StepV2, effect: StepV2["effect"]): StepV2 {
if (!effect) return step;
return { ...step, effect };
}
function toSelection(values: string[], labels?: string[]): SelectedOption[] {
function toSelection(values: string[], labels?: string[]): SelectedOptionV2[] {
return values.map((value, index) => ({
value,
...(labels?.[index] ? { label: labels[index] } : {}),
@@ -110,11 +90,11 @@ function toSelection(values: string[], labels?: string[]): SelectedOption[] {
}
function toV2Step(
step: DraftTraceStep,
step: RecordingDraftStep,
id: number,
urlToId: Map<string, string>,
fallbackUrl?: string,
): Step | null {
): StepV2 | null {
if (!shouldIncludeDraft(step)) return null;
const pageUrl = pageUrlForDraft(step, fallbackUrl);
@@ -129,28 +109,23 @@ function toV2Step(
to: step.url,
};
case "click":
if (!step.captureTarget) return null;
return withEffect(
{
op: "click",
id,
page,
target: step.target,
target: step.captureTarget,
},
effectForNavigation(step.navigated_to, urlToId),
effectForNavigation(step.navigatedTo, urlToId),
);
case "hover":
return {
op: "hover",
id,
page,
target: step.target,
};
case "fill":
if (!step.captureTarget) return null;
return {
op: "fill",
id,
page,
target: step.target,
target: step.captureTarget,
value: step.value,
...(step.redacted ? { redacted: true } : {}),
};
@@ -161,44 +136,51 @@ function toV2Step(
id,
page,
key: step.key,
...(step.target ? { target: step.target } : {}),
...(step.captureTarget ? { target: step.captureTarget } : {}),
...(step.modifiers?.length ? { modifiers: step.modifiers } : {}),
},
effectForNavigation(step.navigated_to, urlToId),
effectForNavigation(step.navigatedTo, urlToId),
);
case "select":
if (!step.captureTarget) return null;
return withEffect(
{
op: "select",
id,
page,
target: step.target,
target: step.captureTarget,
selection: toSelection(step.values, step.labels),
},
effectForNavigation(step.navigated_to, urlToId),
effectForNavigation(step.navigatedTo, urlToId),
);
// `hover` only exists in Trace v3. Peers that negotiate v2 predate the
// step variant and fail to decode the whole result if we emit it.
case "hover":
case "scroll":
case "switch_tab":
return null;
}
}
export interface ReducedTrace {
pages: PageRef[];
steps: Step[];
interface ReducedTrace {
pages: PageRefV2[];
steps: StepV2[];
}
/**
* Compile capture drafts into record-only trace v2 steps.
* Variable inputs are NOT classified here — executing agents infer that at run time.
*/
export function reduceTraceSteps(steps: DraftTraceStep[], startUrl?: string): ReducedTrace {
function reduceTraceSteps(steps: RecordingDraftStep[], startUrl?: string): ReducedTrace {
const collapsed = collapseNavigations(steps);
const { pages, urlToId } = buildPageRegistry(collapsed, startUrl);
const out: Step[] = [];
const out: StepV2[] = [];
let id = 1;
let lastUrl = startUrl;
for (const draft of collapsed) {
if (draft.op === "navigate") lastUrl = draft.url;
else if ("navigated_to" in draft && draft.navigated_to) lastUrl = draft.navigated_to;
else if ("page_url" in draft && draft.page_url) lastUrl = draft.page_url;
else if ("navigatedTo" in draft && draft.navigatedTo) lastUrl = draft.navigatedTo;
else if (draft.pageUrl) lastUrl = draft.pageUrl;
const step = toV2Step(draft, id, urlToId, lastUrl);
if (!step) continue;
out.push(step);
@@ -207,18 +189,27 @@ export function reduceTraceSteps(steps: DraftTraceStep[], startUrl?: string): Re
return { pages, steps: out };
}
export function resolveTraceStartUrl(
drafts: DraftTraceStep[],
function resolveTraceStartUrl(
drafts: RecordingDraftStep[],
startUrl?: string,
pages?: PageRef[],
pages?: PageRefV2[],
): string {
if (startUrl) return startUrl;
const navigate = drafts.find((step): step is Extract<DraftTraceStep, { op: "navigate" }> => {
return step.op === "navigate";
});
if (navigate) return navigate.url;
for (const draft of drafts) {
if ("page_url" in draft && draft.page_url) return draft.page_url;
}
return pages?.[0]?.url ?? "about:blank";
return resolveDraftStartUrl(drafts, startUrl, pages?.[0]?.url);
}
export function buildTraceV2(input: {
steps: RecordingDraftStep[];
startedAt: string;
startUrl?: string;
purpose?: string;
}): TraceV2 {
const { pages, steps } = reduceTraceSteps(input.steps, input.startUrl);
return {
recorded_at: new Date().toISOString(),
started_at: input.startedAt,
...(input.purpose ? { purpose: input.purpose } : {}),
entry: { start_url: resolveTraceStartUrl(input.steps, input.startUrl, pages) },
pages,
steps,
};
}
@@ -0,0 +1,154 @@
import type { NavigationCause, StepV3 } from "@/transport/types";
import { shouldIncludeDraft } from "./draft-policy";
import { hasRedirectQualifier } from "./navigation-policy";
import { unmatchedTarget } from "./target-matcher";
import type { RecordingDraftStep } from "./types";
interface CollapsedDraft {
draft: RecordingDraftStep;
draftIds: number[];
}
const TRANSITION_CAUSES: Record<string, NavigationCause> = {
typed: "user_typed",
generated: "user_typed",
keyword: "user_typed",
keyword_generated: "user_typed",
link: "link",
form_submit: "form_submit",
reload: "reload",
auto_bookmark: "browser",
start_page: "browser",
};
function isRedirect(step: Extract<RecordingDraftStep, { op: "navigate" }>): boolean {
return hasRedirectQualifier(step.transitionQualifiers);
}
function collapseRedirects(steps: RecordingDraftStep[]): CollapsedDraft[] {
const output: CollapsedDraft[] = [];
steps.forEach((step, index) => {
const previous = output[output.length - 1];
if (step.op === "navigate" && previous?.draft.op === "navigate" && isRedirect(step)) {
previous.draft = {
...previous.draft,
url: step.url,
postStateId: step.postStateId ?? previous.draft.postStateId,
};
previous.draftIds.push(index + 1);
return;
}
output.push({ draft: { ...step }, draftIds: [index + 1] });
});
return output;
}
function navigationCause(step: Extract<RecordingDraftStep, { op: "navigate" }>): NavigationCause {
if (step.cause) return step.cause;
const qualifiers = step.transitionQualifiers ?? [];
if (qualifiers.includes("forward_back")) return "history";
if (qualifiers.includes("from_address_bar")) return "user_typed";
return TRANSITION_CAUSES[step.transitionType ?? ""] ?? "browser";
}
function selection(values: string[], labels?: string[]): Array<{ value: string; label?: string }> {
return values.map((value, index) => ({
value,
...(labels?.[index] ? { label: labels[index] } : {}),
}));
}
interface ReduceDraftOptions {
includeTabSwitches: boolean;
redactValues: boolean;
}
function reduceDraft(
draft: RecordingDraftStep,
id: number,
options: ReduceDraftOptions,
): StepV3 | null {
if (!shouldIncludeDraft(draft)) return null;
if (draft.op === "switch_tab") {
if (!options.includeTabSwitches || !draft.preStateId || !draft.postStateId) return null;
return {
op: "switch_tab",
id,
state: draft.preStateId,
result: { state: draft.postStateId },
};
}
const state = draft.preStateId ?? draft.postStateId;
const resultState = draft.postStateId ?? draft.preStateId;
if (!state || !resultState) return null;
const common = { id, state, result: { state: resultState } };
switch (draft.op) {
case "navigate":
return { op: "navigate", ...common, to: draft.url, cause: navigationCause(draft) };
case "click":
return {
op: "click",
...common,
target: draft.matchedTarget ?? unmatchedTarget(draft.captureTarget),
};
case "hover":
return {
op: "hover",
...common,
target: draft.matchedTarget ?? unmatchedTarget(draft.captureTarget),
};
case "fill":
const fillIsRedacted = options.redactValues || draft.redacted === true;
return {
op: "fill",
...common,
target: draft.matchedTarget ?? unmatchedTarget(draft.captureTarget),
value: fillIsRedacted ? "***" : draft.value,
commit: draft.commit ?? "blur",
...(fillIsRedacted ? { redacted: true } : {}),
};
case "press":
return {
op: "press",
...common,
key: draft.key,
...(draft.captureTarget || draft.matchedTarget
? { target: draft.matchedTarget ?? unmatchedTarget(draft.captureTarget) }
: {}),
...(draft.modifiers?.length ? { modifiers: draft.modifiers } : {}),
};
case "select":
return {
op: "select",
...common,
target: draft.matchedTarget ?? unmatchedTarget(draft.captureTarget),
...(!options.redactValues ? { selection: selection(draft.values, draft.labels) } : {}),
};
case "scroll":
return { op: "scroll", ...common };
}
}
export interface ReducedTraceV3 {
steps: StepV3[];
stepIdByDraftId: Map<number, number>;
}
export function reduceTraceStepsV3(
steps: RecordingDraftStep[],
options: { includeTabSwitches?: boolean; redactValues?: boolean } = {},
): ReducedTraceV3 {
const output: StepV3[] = [];
const stepIdByDraftId = new Map<number, number>();
for (const { draft, draftIds } of collapseRedirects(steps)) {
const step = reduceDraft(draft, output.length + 1, {
includeTabSwitches: options.includeTabSwitches === true,
redactValues: options.redactValues === true,
});
if (!step) continue;
output.push(step);
for (const draftId of draftIds) stepIdByDraftId.set(draftId, step.id);
}
return { steps: output, stepIdByDraftId };
}
@@ -0,0 +1,39 @@
import type { StepAnnotation } from "./types";
function annotationText(annotation: StepAnnotation, stepId: number): string {
return ` ⟵ step ${stepId}: ${annotation.op}${annotation.detail ? `: ${annotation.detail}` : ""}`;
}
export function formatTraceStateBody(input: {
stateId: string;
url: string;
title?: string;
stepIds: number[];
vomText: string;
annotations: StepAnnotation[];
stepIdByDraftId: Map<number, number>;
}): string {
const lines = ["# bsk-observation 1", `state: ${JSON.stringify(input.stateId)}`];
lines.push(`url: ${JSON.stringify(input.url)}`);
if (input.title) lines.push(`title: ${JSON.stringify(input.title)}`);
if (input.stepIds.length > 0) lines.push(`steps_here: [${input.stepIds.join(", ")}]`);
lines.push("---");
const byLine = new Map<number, Array<{ annotation: StepAnnotation; stepId: number }>>();
for (const annotation of input.annotations) {
const stepId = input.stepIdByDraftId.get(annotation.draftId);
if (stepId === undefined) continue;
const bucket = byLine.get(annotation.line) ?? [];
bucket.push({ annotation, stepId });
byLine.set(annotation.line, bucket);
}
input.vomText.split("\n").forEach((bodyLine, lineIndex) => {
let line = bodyLine;
for (const item of byLine.get(lineIndex) ?? []) {
line += annotationText(item.annotation, item.stepId);
}
lines.push(line);
});
return `${lines.join("\n")}\n`;
}
+85
View File
@@ -0,0 +1,85 @@
import type { CaptureTargetDescriptor } from "@/lib/describe-target";
import type {
FillCommit,
KeyModifier,
NavigationCause,
StepV3,
TargetDescriptorV3,
} from "@/transport/types";
export interface TargetGeometry {
/** Top-level viewport-relative CSS pixels, as defined by the geometry module. */
rect: { x: number; y: number; w: number; h: number };
tag: string;
}
export interface TargetMatchHint {
geometry?: TargetGeometry;
/** Missing means top frame; null means the source Document could not be resolved. */
frameId?: string | null;
geometrySpace?: "top" | "local";
}
export interface StepAnnotation {
draftId: number;
op: StepV3["op"];
line: number;
stateId: string;
detail?: string;
}
interface DraftStateLink {
pageUrl?: string;
preStateId?: string;
postStateId?: string;
}
interface DraftTarget {
captureTarget?: CaptureTargetDescriptor;
targetHint?: TargetMatchHint;
matchedTarget?: TargetDescriptorV3;
}
interface DraftNavigationEffect {
navigatedTo?: string;
}
export type RecordingDraftStep =
| ({ op: "click" } & DraftStateLink & DraftTarget & DraftNavigationEffect)
| ({ op: "hover" } & DraftStateLink & DraftTarget)
| ({
op: "fill";
value: string;
commit?: FillCommit;
redacted?: boolean;
} & DraftStateLink &
DraftTarget &
DraftNavigationEffect)
| ({
op: "press";
key: string;
modifiers?: KeyModifier[];
} & DraftStateLink &
DraftTarget &
DraftNavigationEffect)
| ({
op: "select";
values: string[];
labels?: string[];
} & DraftStateLink &
DraftTarget &
DraftNavigationEffect)
| ({ op: "scroll" } & DraftStateLink)
| ({ op: "switch_tab" } & DraftStateLink)
| ({
op: "navigate";
url: string;
cause?: NavigationCause;
transitionType?: string;
transitionQualifiers?: string[];
} & DraftStateLink);
export type TargetedRecordingDraft = Extract<
RecordingDraftStep,
{ op: "click" | "hover" | "fill" | "press" | "select" }
>;
@@ -0,0 +1,22 @@
export const RECORD_DOCUMENT_ATTRIBUTE = "data-bsk-record-document";
const RECORDING_DOCUMENT_ID_PATTERN =
/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i;
const RECORDING_DOCUMENT_MARKER_PREFIX = "bsk:";
export function recordingDocumentMarkerValue(identity: string): string {
if (!RECORDING_DOCUMENT_ID_PATTERN.test(identity)) {
throw new TypeError("recording Document identity must be a random UUID");
}
return `${RECORDING_DOCUMENT_MARKER_PREFIX}${identity}`;
}
/** Read BrowserSkill's opaque per-Document recording identity from captured attributes. */
export function readRecordingDocumentIdentity(
attrs: Readonly<Record<string, string>>,
): string | undefined {
const marker = attrs[RECORD_DOCUMENT_ATTRIBUTE];
if (!marker?.startsWith(RECORDING_DOCUMENT_MARKER_PREFIX)) return undefined;
const identity = marker.slice(RECORDING_DOCUMENT_MARKER_PREFIX.length);
return identity && RECORDING_DOCUMENT_ID_PATTERN.test(identity) ? identity : undefined;
}
@@ -127,6 +127,38 @@ describe("requestBorrowConfirmation", () => {
expect(message.requestId.length).toBeGreaterThan(0);
});
it("proactively focuses the user window before sending the overlay message", async () => {
tabs.get.mockResolvedValueOnce({ id: 42, title: "Tab" });
windows.getLastFocused.mockResolvedValueOnce(
userWindowWithActiveTab({ windowId: 77, tabId: 7, url: "https://app.example/" }),
);
tabs.sendMessage.mockResolvedValueOnce({ type: "borrow-response", allowed: true });
const pending = requestBorrowConfirmation(42, {
deps: { tabs, windows, notifications },
});
await vi.runAllTimersAsync();
await pending;
expect(windows.update).toHaveBeenCalledWith(77, { focused: true });
});
it("still proceeds when proactive window focus fails", async () => {
tabs.get.mockResolvedValueOnce({ id: 42, title: "Tab" });
windows.getLastFocused.mockResolvedValueOnce(
userWindowWithActiveTab({ windowId: 77, tabId: 7, url: "https://app.example/" }),
);
windows.update.mockRejectedValueOnce(new Error("window gone"));
tabs.sendMessage.mockResolvedValueOnce({ type: "borrow-response", allowed: true });
const pending = requestBorrowConfirmation(42, {
deps: { tabs, windows, notifications },
});
await vi.runAllTimersAsync();
expect(await pending).toBe(true);
});
it("sends confirmation to the borrow target when it is the active tab", async () => {
tabs.get.mockResolvedValueOnce({ id: 42, title: "Active Tab" });
windows.getLastFocused.mockResolvedValueOnce(
@@ -8,6 +8,7 @@ import type {
RequestFrame,
} from "@/transport/types";
import { ToolDispatcher } from "../dispatcher";
import { resetBrowserObservationForTests, setBrowserObservationAttachForTests } from "../record";
type TestDispatcherCdp = NonNullable<ConstructorParameters<typeof ToolDispatcher>[0]["cdp"]>;
@@ -44,9 +45,72 @@ function makeRequest(method: string, params: unknown): RequestFrame {
describe("ToolDispatcher", () => {
afterEach(() => {
resetBrowserObservationForTests();
vi.unstubAllGlobals();
});
it("uses the configured recording runtime for start and stop", async () => {
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
agentWindow: {
create: vi.fn(async () => 4242),
remove: vi.fn(),
ensureActiveTab: vi.fn(async () => {}),
},
});
await sessions.start("aa11");
const tab = {
id: 7,
windowId: 4242,
active: true,
status: "complete",
url: "https://example.com/start",
} as chrome.tabs.Tab;
const frameCoordinator = {
begin: vi.fn(),
armTab: vi.fn(async () => true),
sourceFor: vi.fn(() => null),
stop: vi.fn(async () => true),
cancel: vi.fn(),
};
const sendToTab = vi.fn(async () => ({ ok: true }));
setBrowserObservationAttachForTests(
() => () => {},
() => () => {},
);
const dispatcher = new ToolDispatcher({
transport,
sessions,
recording: {
tabsApi: {
get: vi.fn(async () => tab),
query: vi.fn(async () => [tab]),
},
frameCoordinator,
sendToTab,
},
});
dispatcher.start();
deliver(
makeRequest("tool.record_start", {
session_id: "aa11",
url: "https://example.com/start",
}),
);
await vi.waitFor(() => expect(sent).toHaveLength(1));
expect(sent[0]).toMatchObject({ result: { tab_id: 7, recording: true } });
expect(frameCoordinator.begin).toHaveBeenCalledOnce();
expect(frameCoordinator.armTab).toHaveBeenCalledWith(expect.any(String), 7);
deliver({ id: "r-2", method: "tool.record_stop", params: { session_id: "aa11" } });
await vi.waitFor(() => expect(sent).toHaveLength(2));
expect(frameCoordinator.stop).toHaveBeenCalledOnce();
expect(sent[1]).toMatchObject({ id: "r-2", result: { trace: { steps: [] } } });
});
it("routes tool.session_start to the SessionManager and replies with the window id", async () => {
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
@@ -168,6 +232,107 @@ describe("ToolDispatcher", () => {
});
});
it("bypasses and restores the control overlay for an upload trigger click", async () => {
const sendMessage = vi.fn(async () => undefined);
vi.stubGlobal("chrome", {
tabs: {
get: vi.fn(async () => ({ id: 7, windowId: 4242, active: true })),
query: vi.fn(async () => [{ id: 7, windowId: 4242, active: true }]),
sendMessage,
},
});
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
agentWindow: {
create: vi.fn(async () => 4242),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
},
});
const ctx = await sessions.start("aa11");
ctx.refStore.set("e1", 123, { tabId: 7 });
const send = vi.fn(async <T>(_tabId: number, method: string, params?: object) => {
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } } as T;
}
if (method === "DOM.getContentQuads") {
return { quads: [[0, 0, 20, 0, 20, 20, 0, 20]] } as T;
}
if (method === "DOM.resolveNode") {
return { object: { objectId: "trigger-object" } } as T;
}
if (method === "DOM.describeNode") return { node: { backendNodeId: 456 } } as T;
if (method === "Runtime.callFunctionOn") {
const declaration = (params as { functionDeclaration?: string }).functionDeclaration ?? "";
if (declaration.includes("count: state.inputs.length")) {
return { result: { value: { count: 1, multiple: false } } } as T;
}
if (declaration.includes("inputs[0]")) {
return { result: { objectId: "input-object" } } as T;
}
return { result: { value: true } } as T;
}
if (method === "Runtime.evaluate") {
const expression = (params as { expression?: string }).expression ?? "";
if (expression.includes("overlayDetails")) {
return { result: { value: { hitIndex: 0 } } } as T;
}
if (expression.includes("overlayHostPresent")) {
return {
result: {
value: { overlayHostPresent: true, overlayHostConnected: true },
},
} as T;
}
if (expression.includes("count:")) {
return { result: { value: { count: 1, multiple: false } } } as T;
}
if (expression.includes("?.inputs[0]")) {
return { result: { objectId: "input-object" } } as T;
}
return { result: { value: true } } as T;
}
return {} as T;
});
const cdp = {
send,
detachSession: vi.fn(async () => {}),
ensureNetworkCapture: vi.fn(async () => {}),
networkEntriesSince: vi.fn(() => ({
tab_id: 7,
entries: [],
next_since: 0,
truncated: false,
})),
setDeviceMetricsOverride: vi.fn(async () => {}),
clearDeviceMetricsOverride: vi.fn(async () => {}),
setUserAgentOverride: vi.fn(async () => {}),
setTouchEmulationEnabled: vi.fn(async () => {}),
};
const dispatcher = new ToolDispatcher({ transport, sessions, cdp: cdp as TestDispatcherCdp });
dispatcher.start();
deliver(
makeRequest("tool.upload", {
session_id: "aa11",
ref: "@e1",
files: [{ transfer_id: "tr_1", name: "test.png", staged_path: "/stage/test.png" }],
}),
);
await flushMicrotasks();
await vi.waitFor(() => expect(sent).toHaveLength(1));
expect(sent[0]).toMatchObject({ result: { tab_id: 7, file_names: ["test.png"] } });
expect(sendMessage).toHaveBeenNthCalledWith(1, 7, {
type: "bh-automation-bypass",
enabled: true,
});
expect(sendMessage).toHaveBeenNthCalledWith(2, 7, {
type: "bh-automation-bypass",
enabled: false,
});
});
it("detaches CDP state before stopping a session", async () => {
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
@@ -203,6 +368,55 @@ describe("ToolDispatcher", () => {
expect(sent[0]).toEqual({ id: "r-1", result: {} });
});
it("forwards daemon cancellation into an in-flight session stop", async () => {
const { transport, sent, deliver } = fakeTransport();
const remove = vi.fn(async () => {});
const sessions = new SessionManager({
agentWindow: {
create: vi.fn(async () => 4242),
remove,
ensureActiveTab: vi.fn(async () => {}),
},
});
await sessions.start("aa11");
let finishDetach: () => void = () => {};
const detach = new Promise<void>((resolve) => {
finishDetach = resolve;
});
const cdp = {
send: vi.fn(),
detachSession: vi.fn(() => detach),
ensureNetworkCapture: vi.fn(async () => {}),
networkEntriesSince: vi.fn(() => ({
tab_id: 7,
entries: [],
next_since: 0,
truncated: false,
})),
setDeviceMetricsOverride: vi.fn(async () => {}),
clearDeviceMetricsOverride: vi.fn(async () => {}),
setUserAgentOverride: vi.fn(async () => {}),
setTouchEmulationEnabled: vi.fn(async () => {}),
};
const dispatcher = new ToolDispatcher({ transport, sessions, cdp: cdp as TestDispatcherCdp });
dispatcher.start();
deliver(makeRequest("tool.session_stop", { session_id: "aa11" }));
await vi.waitFor(() => expect(cdp.detachSession).toHaveBeenCalledWith("aa11"));
deliver({ id: "cancel-stop", method: "cancel", params: { rpc_id: "r-1" } });
await flushMicrotasks();
finishDetach();
await flushMicrotasks();
expect(sent).toContainEqual({ id: "cancel-stop", result: { cancelled: true } });
expect(sent).toContainEqual({
id: "r-1",
error: expect.objectContaining({ code: "cancelled" }),
});
expect(sessions.has("aa11")).toBe(true);
expect(remove).not.toHaveBeenCalled();
});
it("returns not_found when stopping an unknown session", async () => {
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
@@ -0,0 +1,988 @@
import { describe, expect, it, vi } from "vitest";
import { SessionManager } from "@/session-manager/manager";
import { type DownloadsApi, handleDownload } from "../download";
import { captureBrowserDownload } from "../download-capture";
import { uploadThroughFileDrop } from "../file-drop-transaction";
import { uploadThroughActivatedFileInput } from "../file-input-transaction";
import type { ResolvedActionTarget } from "../interaction";
import type { CdpRunner } from "../shared";
import { handleUpload } from "../upload";
function sessions() {
return new SessionManager({
agentWindow: {
create: vi.fn(async () => 100),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
},
});
}
function tabsApi() {
return {
get: vi.fn(
async (tabId: number) => ({ id: tabId, windowId: 100, active: true }) as chrome.tabs.Tab,
),
query: vi.fn(async () => [{ id: 4, windowId: 100, active: true } as chrome.tabs.Tab]),
};
}
function fakeEvent<T extends (...args: never[]) => unknown>() {
const listeners = new Set<T>();
return {
addListener: (listener: T) => listeners.add(listener),
removeListener: (listener: T) => listeners.delete(listener),
emit: (...args: Parameters<T>) => {
for (const listener of listeners) listener(...args);
},
};
}
function actionTarget(frameId?: string, sessionId?: string): ResolvedActionTarget {
return {
tab: { tabId: 4, windowId: 100, active: true },
backendNodeId: 123,
cdpTarget: { tabId: 4, ...(sessionId ? { sessionId } : {}) },
...(frameId ? { frameId } : {}),
usedRef: "e3",
};
}
function withoutOverlay<T>(operation: () => Promise<T>): Promise<T> {
return operation();
}
function uploadCdp(
options: {
inputCount?: number;
multiple?: boolean;
chooser?: { frameId?: string; backendNodeId?: number; mode?: string };
pendingResolve?: boolean;
} = {},
) {
const calls: Array<{ method: string; params?: object }> = [];
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const send = vi.fn(async (_tabId: number, method: string, params?: object) => {
calls.push({ method, params });
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "Page.setInterceptFileChooserDialog") return {};
if (method === "Page.getLayoutMetrics")
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
if (method === "DOM.getContentQuads") return { quads: [[0, 0, 20, 0, 20, 20, 0, 20]] };
if (method === "DOM.resolveNode") {
if (options.pendingResolve) return new Promise<never>(() => {});
return { object: { objectId: "trigger-object" } };
}
if (method === "Runtime.callFunctionOn") {
const declaration = (params as { functionDeclaration?: string }).functionDeclaration ?? "";
if (declaration.includes("Object.defineProperty")) return { result: { value: true } };
if (declaration.includes("count: state.inputs.length")) {
return {
result: {
value: {
count: options.inputCount ?? 1,
multiple: options.multiple ?? true,
},
},
};
}
if (declaration.includes("inputs[0]")) {
return { result: { objectId: "input-object" } };
}
return { result: { value: true } };
}
if (method === "DOM.describeNode") return { node: { backendNodeId: 456 } };
if (
method === "Input.dispatchMouseEvent" &&
(params as { type?: string }).type === "mousePressed" &&
options.chooser
) {
cdpEvent?.({ tabId: 4 }, "Page.fileChooserOpened", options.chooser);
}
return {};
});
const cdp: CdpRunner = {
send: send as unknown as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
return {
calls,
emitChooser: (event: { frameId?: string; backendNodeId?: number; mode?: string }) =>
cdpEvent?.({ tabId: 4 }, "Page.fileChooserOpened", event),
cdp,
};
}
function dropCdp(
options: {
targetContainsPoint?: boolean;
dispatchError?: (type: string) => Error | Promise<never> | undefined;
onCall?: (method: string) => void;
} = {},
) {
const calls: Array<{ method: string; params?: object }> = [];
const detach = vi.fn(async () => {});
const send = vi.fn(async (_tabId: number, method: string, params?: object) => {
calls.push({ method, params });
options.onCall?.(method);
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
}
if (method === "DOM.getContentQuads") {
return { quads: [[10, 20, 210, 20, 210, 120, 10, 120]] };
}
if (method === "DOM.getNodeForLocation") return { backendNodeId: 124 };
if (method === "DOM.resolveNode") {
const id = (params as { backendNodeId?: number }).backendNodeId;
return { object: { objectId: id === 123 ? "drop-target" : "hit-target" } };
}
if (method === "Runtime.callFunctionOn") {
return { result: { value: options.targetContainsPoint ?? true } };
}
if (method === "Input.dispatchDragEvent") {
const type = (params as { type?: string }).type ?? "";
const error = options.dispatchError?.(type);
if (error instanceof Promise) return error;
if (error) throw error;
return {};
}
if (method === "Runtime.releaseObjectGroup") return {};
throw new Error(`unexpected drop CDP command ${method}`);
});
const cdp: CdpRunner = {
send: send as unknown as CdpRunner["send"],
detach,
};
return { calls, cdp, detach };
}
describe("file transfer tools", () => {
it("captures the file input activated by the requested click and injects only staged paths", async () => {
const manager = sessions();
const ctx = await manager.start("s1");
ctx.refStore.set("e3", 123, { tabId: 4 });
const { cdp, calls } = uploadCdp();
const result = await handleUpload(
manager,
{
session_id: "s1",
ref: "@e3",
files: [
{ transfer_id: "tr_1", name: "one.png", staged_path: "/private/stage/one" },
{ transfer_id: "tr_2", name: "two.png", staged_path: "/private/stage/two" },
],
},
{ cdp, tabsApi: tabsApi() },
);
expect(result).toMatchObject({ tab_id: 4, file_names: ["one.png", "two.png"] });
expect(calls[0]).toEqual({
method: "Page.setInterceptFileChooserDialog",
params: { enabled: true },
});
expect(calls).toContainEqual({
method: "Page.setInterceptFileChooserDialog",
params: { enabled: false, cancel: true },
});
expect(calls).toContainEqual({
method: "DOM.setFileInputFiles",
params: { files: ["/private/stage/one", "/private/stage/two"], backendNodeId: 456 },
});
});
it("fails immediately when the trigger does not activate a file input", async () => {
const { cdp } = uploadCdp({ inputCount: 0 });
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget(),
files: ["/private/stage/one"],
timeoutMs: 100,
trigger: async () => ({ tab_id: 4, x: 10, y: 10 }),
});
expect(result).toMatchObject({
code: "unsupported",
data: { reason: "file_input_not_activated", phase: "resolve_input" },
});
});
it("fails before clicking when chooser interception is unavailable", async () => {
const trigger = vi.fn(async () => ({ tab_id: 4, x: 10, y: 10 }));
const cdp: CdpRunner = {
send: vi.fn(async (_tabId: number, method: string) => {
if (method === "Page.setInterceptFileChooserDialog") {
throw new Error("method unavailable");
}
return {};
}) as CdpRunner["send"],
};
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget(),
files: ["/private/stage/one"],
timeoutMs: 100,
trigger,
});
expect(result).toMatchObject({
code: "cdp_failed",
data: { effect_state: "none", phase: "arm_interception" },
});
expect(trigger).not.toHaveBeenCalled();
});
it("uses an exact chooser event as an independent input-location signal", async () => {
const { cdp, calls, emitChooser } = uploadCdp({ inputCount: 0 });
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget("f1"),
files: ["/private/stage/one"],
timeoutMs: 100,
trigger: async () => {
emitChooser({ frameId: "f1", backendNodeId: 789, mode: "selectSingle" });
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(result).toMatchObject({ multiple: false });
expect(calls).toContainEqual({
method: "DOM.setFileInputFiles",
params: { files: ["/private/stage/one"], backendNodeId: 789 },
});
});
it("reports a File System Access picker without waiting for a timeout", async () => {
const { cdp, emitChooser } = uploadCdp({
inputCount: 0,
});
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget("f1"),
files: ["/private/stage/one"],
timeoutMs: 100,
trigger: async () => {
emitChooser({ frameId: "f1", mode: "selectSingle" });
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(result).toMatchObject({
code: "unsupported",
message: "upload trigger invoked a non-input file picker",
data: { reason: "file_input_not_activated", phase: "resolve_input" },
});
});
it("bounds a stuck file-input probe", async () => {
const { cdp } = uploadCdp({ pendingResolve: true });
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget(),
files: ["/private/stage/one"],
timeoutMs: 5,
trigger: vi.fn(),
});
expect(result).toMatchObject({
code: "timeout",
data: { reason: "file_input_probe_failed", phase: "arm_input_probe" },
});
});
it("marks a timed-out file assignment unknown and detaches browser state", async () => {
const fixture = uploadCdp();
const originalSend = fixture.cdp.send;
const detach = vi.fn(async () => {});
fixture.cdp.detach = detach;
fixture.cdp.send = vi.fn((tabId: number, method: string, params?: object) => {
if (method === "DOM.setFileInputFiles") return new Promise<never>(() => {});
return originalSend(tabId, method, params);
}) as CdpRunner["send"];
const result = await uploadThroughActivatedFileInput({
cdp: fixture.cdp,
actionTarget: actionTarget(),
files: ["/private/stage/one"],
timeoutMs: 10,
trigger: async () => ({ tab_id: 4, x: 10, y: 10 }),
});
expect(result).toMatchObject({
code: "timeout",
data: { effect_state: "unknown", phase: "set_files" },
});
expect(detach).toHaveBeenCalledWith(4);
});
it("delivers staged files through one explicit native drop without clicking", async () => {
const manager = sessions();
const ctx = await manager.start("s1");
ctx.refStore.set("e3", 123, { tabId: 4 });
const lifecycle: string[] = [];
const { cdp, calls } = dropCdp({ onCall: (method) => lifecycle.push(method) });
const overlayPhases: string[] = [];
const result = await handleUpload(
manager,
{
session_id: "s1",
ref: "@e3",
mode: "drop",
files: [{ transfer_id: "tr_1", name: "one.png", staged_path: "/stage/one.png" }],
},
{
cdp,
tabsApi: tabsApi(),
sendToTab: vi.fn(async (_tabId, message) => {
overlayPhases.push(message.phase);
lifecycle.push(`overlay:${message.phase}`);
}),
},
);
expect(result).toMatchObject({ tab_id: 4, used_ref: "e3", file_names: ["one.png"] });
expect(calls.filter((call) => call.method === "Input.dispatchDragEvent")).toEqual([
{
method: "Input.dispatchDragEvent",
params: {
type: "dragEnter",
x: 110,
y: 70,
data: { items: [], files: ["/stage/one.png"], dragOperationsMask: 1 },
},
},
{
method: "Input.dispatchDragEvent",
params: {
type: "dragOver",
x: 110,
y: 70,
data: { items: [], files: ["/stage/one.png"], dragOperationsMask: 1 },
},
},
{
method: "Input.dispatchDragEvent",
params: {
type: "drop",
x: 110,
y: 70,
data: { items: [], files: ["/stage/one.png"], dragOperationsMask: 1 },
},
},
]);
expect(calls.some((call) => call.method === "Input.dispatchMouseEvent")).toBe(false);
expect(overlayPhases).toEqual(["begin", "end"]);
expect(lifecycle.indexOf("DOM.getContentQuads")).toBeLessThan(
lifecycle.indexOf("overlay:begin"),
);
expect(lifecycle.indexOf("overlay:begin")).toBeLessThan(
lifecycle.indexOf("DOM.getNodeForLocation"),
);
expect(lifecycle.lastIndexOf("Input.dispatchDragEvent")).toBeLessThan(
lifecycle.indexOf("overlay:end"),
);
expect(lifecycle.indexOf("overlay:end")).toBeLessThan(
lifecycle.indexOf("Runtime.releaseObjectGroup"),
);
});
it("fails closed before dragging when another element owns the action point", async () => {
const { cdp, calls } = dropCdp({ targetContainsPoint: false });
const result = await uploadThroughFileDrop({
cdp,
actionTarget: actionTarget(),
files: ["/stage/one.png"],
timeoutMs: 100,
withOverlayHidden: withoutOverlay,
});
expect(result).toMatchObject({
code: "permission_denied",
data: {
reason: "file_drop_target_unavailable",
mechanism: "drop",
effect_state: "none",
phase: "resolve_target",
},
});
expect(calls.some((call) => call.method === "Input.dispatchDragEvent")).toBe(false);
});
it("dispatches an OOPIF file drop in the child target's local coordinates", async () => {
const rootCalls: Array<{ method: string; params?: object }> = [];
const childCalls: Array<{ method: string; params?: object }> = [];
const cdp: CdpRunner = {
send: vi.fn(async (_tabId, method, params) => {
rootCalls.push({ method, params });
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "DOM.getBoxModel") {
return { model: { content: [100, 100, 300, 100, 300, 200, 100, 200] } };
}
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 800, clientHeight: 600 } };
}
throw new Error(`unexpected root command ${method}`);
}) as CdpRunner["send"],
sendToTarget: vi.fn(async (_target, method, params) => {
childCalls.push({ method, params });
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "DOM.getContentQuads") {
return { quads: [[10, 20, 210, 20, 210, 120, 10, 120]] };
}
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 400, clientHeight: 300 } };
}
if (method === "DOM.getNodeForLocation") {
return { backendNodeId: 124, frameId: "child" };
}
if (method === "DOM.resolveNode") {
const id = (params as { backendNodeId?: number }).backendNodeId;
return { object: { objectId: id === 123 ? "drop-target" : "hit-target" } };
}
if (method === "Runtime.callFunctionOn") return { result: { value: true } };
if (method === "Input.dispatchDragEvent" || method === "Runtime.releaseObjectGroup") {
return {};
}
throw new Error(`unexpected child command ${method}`);
}) as CdpRunner["sendToTarget"],
getFrameGraph: vi.fn(async () => ({
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "child",
parentFrameId: "main",
ownerBackendNodeId: 10,
target: { tabId: 4, sessionId: "child-session" },
},
],
})),
};
const result = await uploadThroughFileDrop({
cdp,
actionTarget: actionTarget("child", "child-session"),
files: ["/stage/one.png"],
timeoutMs: 100,
withOverlayHidden: withoutOverlay,
});
expect(result).toMatchObject({ x: 110, y: 70 });
expect(
childCalls
.filter((call) => call.method === "Input.dispatchDragEvent")
.map((call) => call.params),
).toEqual([
expect.objectContaining({ type: "dragEnter", x: 110, y: 70 }),
expect.objectContaining({ type: "dragOver", x: 110, y: 70 }),
expect.objectContaining({ type: "drop", x: 110, y: 70 }),
]);
expect(rootCalls.some((call) => call.method === "Input.dispatchDragEvent")).toBe(false);
});
it("reports a missing native drag command as unsupported before commit", async () => {
const { cdp } = dropCdp({
dispatchError: (type) =>
type === "dragEnter" ? new Error("'Input.dispatchDragEvent' wasn't found") : undefined,
});
const result = await uploadThroughFileDrop({
cdp,
actionTarget: actionTarget(),
files: ["/stage/one.png"],
timeoutMs: 100,
withOverlayHidden: withoutOverlay,
});
expect(result).toMatchObject({
code: "unsupported",
data: {
reason: "upload_mechanism_unsupported",
effect_state: "none",
phase: "drag_enter",
},
});
});
it("cancels an unresolved drag before returning a pre-commit timeout", async () => {
const { cdp, calls } = dropCdp({
dispatchError: (type) => (type === "dragEnter" ? new Promise<never>(() => {}) : undefined),
});
const result = await uploadThroughFileDrop({
cdp,
actionTarget: actionTarget(),
files: ["/stage/one.png"],
timeoutMs: 10,
withOverlayHidden: withoutOverlay,
});
expect(result).toMatchObject({
code: "timeout",
data: { reason: "file_drop_failed", effect_state: "none", phase: "drag_enter" },
});
expect(calls).toContainEqual({
method: "Input.dispatchDragEvent",
params: expect.objectContaining({ type: "dragCancel" }),
});
});
it("marks an unresolved drop unknown and detaches browser state", async () => {
const { cdp, detach } = dropCdp({
dispatchError: (type) => (type === "drop" ? new Promise<never>(() => {}) : undefined),
});
const result = await uploadThroughFileDrop({
cdp,
actionTarget: actionTarget(),
files: ["/stage/one.png"],
timeoutMs: 10,
withOverlayHidden: withoutOverlay,
});
expect(result).toMatchObject({
code: "timeout",
data: { reason: "file_drop_failed", effect_state: "unknown", phase: "drop" },
});
expect(detach).toHaveBeenCalledWith(4);
});
it("routes one exact-target download through a browser-relative capability", async () => {
const manager = sessions();
const ctx = await manager.start("s1");
ctx.refStore.set("e3", 123, { tabId: 4 });
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const initial = {
id: 9,
url: "https://example.test/result.zip",
finalUrl: "https://example.test/result.zip",
filename: "result.zip",
state: "in_progress",
fileSize: -1,
totalBytes: 12,
mime: "application/zip",
danger: "safe",
} as chrome.downloads.DownloadItem;
const completed = {
...initial,
filename: "/profile/Downloads/BrowserSkill/tr_1/result.zip",
state: "complete",
fileSize: 12,
} as chrome.downloads.DownloadItem;
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn(async () => [completed]),
cancel: vi.fn(async () => {}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
let suggested: chrome.downloads.DownloadFilenameSuggestion | undefined;
const send = vi.fn(async (_tabId: number, method: string, params?: object) => {
if (method === "Page.getLayoutMetrics")
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
if (method === "DOM.getContentQuads") return { quads: [[0, 0, 20, 0, 20, 20, 0, 20]] };
if (
method === "Input.dispatchMouseEvent" &&
(params as { type?: string }).type === "mousePressed"
) {
cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", {
url: initial.url,
suggestedFilename: "result.zip",
});
await new Promise<void>((resolve) => {
onDeterminingFilename.emit(initial, (value) => {
suggested = value;
resolve();
});
});
onCreated.emit(completed);
}
return {};
});
const cdp: CdpRunner = {
send: send as unknown as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
const result = await handleDownload(
manager,
{ session_id: "s1", ref: "@e3", browser_relative_dir: "BrowserSkill/tr_1" },
{ cdp, tabsApi: tabsApi(), downloads },
);
expect(suggested).toEqual({
filename: "BrowserSkill/tr_1/result.zip",
conflictAction: "overwrite",
});
expect(result).toMatchObject({
tab_id: 4,
suggested_filename: "result.zip",
byte_size: 12,
browser_path: completed.filename,
});
});
it("does not claim a download without an intent from the exact target", async () => {
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn(async () => []),
cancel: vi.fn(async () => {}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as unknown as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
const unrelated = {
id: 17,
url: "https://example.test/unrelated.zip",
finalUrl: "https://example.test/unrelated.zip",
filename: "unrelated.zip",
state: "in_progress",
} as chrome.downloads.DownloadItem;
let defaultSuggestionCalled = false;
const result = await captureBrowserDownload({
cdp,
target: { tabId: 4, sessionId: "expected-child" },
downloads,
browserRelativeDir: "BrowserSkill/tr_1",
timeoutMs: 5,
trigger: async () => {
cdpEvent?.({ tabId: 4, sessionId: "other-child" }, "Page.downloadWillBegin", {
url: unrelated.url,
suggestedFilename: unrelated.filename,
});
onDeterminingFilename.emit(unrelated, (suggestion) => {
defaultSuggestionCalled = suggestion === undefined;
});
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(defaultSuggestionCalled).toBe(true);
expect(downloads.cancel).not.toHaveBeenCalled();
expect(result).toMatchObject({
code: "cdp_failed",
data: { reason: "download_capture_failed" },
});
});
it("correlates a filename candidate that arrives before the CDP intent", async () => {
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const initial = {
id: 21,
url: "https://example.test/candidate-first.bin",
finalUrl: "https://example.test/candidate-first.bin",
filename: "candidate-first.bin",
state: "in_progress",
fileSize: -1,
totalBytes: 4,
bytesReceived: 0,
} as chrome.downloads.DownloadItem;
const complete = {
...initial,
state: "complete",
fileSize: 4,
} as chrome.downloads.DownloadItem;
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn(async () => [complete]),
cancel: vi.fn(async () => {}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
let suggestion: chrome.downloads.DownloadFilenameSuggestion | undefined;
const result = await captureBrowserDownload({
cdp,
target: { tabId: 4 },
downloads,
browserRelativeDir: "BrowserSkill/tr_21",
timeoutMs: 1_000,
trigger: async () => {
const suggested = new Promise<void>((resolve) => {
onDeterminingFilename.emit(initial, (value) => {
suggestion = value;
resolve();
});
});
cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", {
url: initial.url,
suggestedFilename: initial.filename,
});
await suggested;
onCreated.emit(complete);
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(suggestion).toEqual({
filename: "BrowserSkill/tr_21/candidate-first.bin",
conflictAction: "overwrite",
});
expect(result).toMatchObject({ item: { id: 21, state: "complete" } });
});
it("removes a completed download when the final size exceeds the limit", async () => {
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const initial = {
id: 22,
url: "https://example.test/oversized.bin",
finalUrl: "https://example.test/oversized.bin",
filename: "oversized.bin",
state: "in_progress",
fileSize: -1,
totalBytes: -1,
bytesReceived: 0,
} as chrome.downloads.DownloadItem;
const complete = {
...initial,
filename: "/profile/Downloads/BrowserSkill/tr_22/oversized.bin",
state: "complete",
fileSize: 8,
totalBytes: 8,
bytesReceived: 8,
} as chrome.downloads.DownloadItem;
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn(async () => [complete]),
cancel: vi.fn(async () => {}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
const result = await captureBrowserDownload({
cdp,
target: { tabId: 4 },
downloads,
browserRelativeDir: "BrowserSkill/tr_22",
maxByteSize: 4,
timeoutMs: 1_000,
trigger: async () => {
cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", {
url: initial.url,
suggestedFilename: initial.filename,
});
await new Promise<void>((resolve) => {
onDeterminingFilename.emit(initial, () => resolve());
});
onCreated.emit(initial);
onChanged.emit({ id: initial.id, state: { current: "complete" } });
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(result).toMatchObject({
code: "cdp_failed",
data: { reason: "download_capture_failed", effect_state: "committed" },
});
expect(downloads.removeFile).toHaveBeenCalledWith(initial.id);
expect(downloads.cancel).not.toHaveBeenCalled();
});
it("reconciles a download that completes while cancellation is being requested", async () => {
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const initial = {
id: 23,
url: "https://example.test/racing.bin",
finalUrl: "https://example.test/racing.bin",
filename: "racing.bin",
state: "in_progress",
fileSize: -1,
totalBytes: -1,
bytesReceived: 0,
} as chrome.downloads.DownloadItem;
const complete = {
...initial,
filename: "/profile/Downloads/BrowserSkill/tr_23/racing.bin",
state: "complete",
fileSize: 4,
totalBytes: 4,
bytesReceived: 4,
} as chrome.downloads.DownloadItem;
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn().mockResolvedValueOnce([initial]).mockResolvedValueOnce([complete]),
cancel: vi.fn(async () => {
throw new Error("download already complete");
}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
const result = await captureBrowserDownload({
cdp,
target: { tabId: 4 },
downloads,
browserRelativeDir: "BrowserSkill/tr_23",
timeoutMs: 80,
trigger: async () => {
cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", {
url: initial.url,
suggestedFilename: initial.filename,
});
await new Promise<void>((resolve) => {
onDeterminingFilename.emit(initial, () => resolve());
});
onCreated.emit(initial);
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(result).toMatchObject({
code: "cdp_failed",
data: { reason: "download_capture_failed", effect_state: "committed" },
});
expect(downloads.cancel).toHaveBeenCalledWith(initial.id);
expect(downloads.removeFile).toHaveBeenCalledWith(initial.id);
expect(result).not.toMatchObject({ data: { cleanup_state: "failed" } });
});
it("rejects ambiguous attribution without cancelling either unclaimed download", async () => {
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn(async () => []),
cancel: vi.fn(async () => {}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
const defaults: number[] = [];
const candidate = (id: number) =>
({
id,
url: "https://example.test/same.bin",
finalUrl: "https://example.test/same.bin",
filename: "same.bin",
state: "in_progress",
}) as chrome.downloads.DownloadItem;
const result = await captureBrowserDownload({
cdp,
target: { tabId: 4 },
downloads,
browserRelativeDir: "BrowserSkill/tr_ambiguous",
timeoutMs: 100,
trigger: async () => {
cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", {
url: "https://example.test/same.bin",
suggestedFilename: "same.bin",
});
for (const id of [31, 32]) {
onDeterminingFilename.emit(candidate(id), (value) => {
if (value === undefined) defaults.push(id);
});
}
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(defaults.sort()).toEqual([31, 32]);
expect(downloads.cancel).not.toHaveBeenCalled();
expect(result).toMatchObject({
code: "cdp_failed",
data: { effect_state: "unknown", phase: "attribution" },
});
});
});
@@ -169,6 +169,7 @@ describe("frame geometry projection", () => {
expect(geometry).toMatchObject({
topBounds: { x: 100, y: 100, width: 200, height: 100 },
actionPoint: { x: 200, y: 150 },
targetActionPoint: { x: 100, y: 50 },
});
});
@@ -261,6 +262,7 @@ describe("frame geometry projection", () => {
expect(geometry).toMatchObject({
topBounds: { x: 100, y: 100, width: 200, height: 100 },
actionPoint: { x: 200, y: 150 },
targetActionPoint: { x: 200, y: 150 },
});
});
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -66,6 +66,52 @@ function makeApis(
}
describe("handleSessionStop with auto-return", () => {
it("leaves the session untouched when cancellation arrived before teardown", async () => {
const aw = fakeAgentWindow([100]);
const sm = new SessionManager({ agentWindow: aw });
await sm.start("aa11");
const cdp = { detachSession: vi.fn(async () => {}) };
const controller = new AbortController();
controller.abort();
const res = await handleSessionStop(
sm,
{ session_id: "aa11" },
{ cdp, signal: controller.signal },
);
expect(res).toMatchObject({ code: "cancelled" });
expect(sm.has("aa11")).toBe(true);
expect(cdp.detachSession).not.toHaveBeenCalled();
expect(aw.remove).not.toHaveBeenCalled();
});
it("does not close the Agent Window when cancellation lands during CDP detach", async () => {
const aw = fakeAgentWindow([100]);
const sm = new SessionManager({ agentWindow: aw });
await sm.start("aa11");
const controller = new AbortController();
let finishDetach: () => void = () => {};
const detach = new Promise<void>((resolve) => {
finishDetach = resolve;
});
const cdp = { detachSession: vi.fn(() => detach) };
const stopping = handleSessionStop(
sm,
{ session_id: "aa11" },
{ cdp, signal: controller.signal },
);
await vi.waitFor(() => expect(cdp.detachSession).toHaveBeenCalledWith("aa11"));
controller.abort();
finishDetach();
const res = await stopping;
expect(res).toMatchObject({ code: "cancelled" });
expect(sm.has("aa11")).toBe(true);
expect(aw.remove).not.toHaveBeenCalled();
});
it("returns every borrowed tab and closes the Agent Window in the right order", async () => {
const aw = fakeAgentWindow([100]);
const sm = new SessionManager({ agentWindow: aw });
@@ -24,7 +24,7 @@ export interface BorrowResponseMessage {
allowed: boolean;
}
export const CONFIRMATION_TIMEOUT_MS = 5000;
export const CONFIRMATION_TIMEOUT_MS = 60_000;
const EXIT_ANIMATION_MS = 150;
/** Matches BorrowConfirmationOverlay progress ring/bar transition (duration-1000). */
export const PROGRESS_TRANSITION_MS = 1000;
@@ -480,6 +480,13 @@ export async function requestBorrowConfirmation(
settle(false);
}, BACKGROUND_TIMEOUT_MS);
// Bring the user window to the front so the overlay is visible even
// when the Agent Window has stolen focus. This is fire-and-forget: if
// it fails the overlay / OS notification still give the user a path.
void windowsApi.update(notificationAnchor.windowId, { focused: true }).catch((err) => {
console.debug("[bsk borrow] proactive focus of user window failed", err);
});
// Surface the OS notification *before* messaging any candidate so the
// user has a parallel signal even if every content script is missing.
// The Allow / Deny buttons let the user authorize explicitly when the
@@ -0,0 +1,9 @@
export {
type CaptureVomObservationOptions,
captureVomObservation,
} from "./observation";
export {
type CaptureVomFrame,
type CaptureVomMatchNode,
type CaptureVomObservationResult,
} from "./vom/record-safe-observation";
+80 -38
View File
@@ -4,6 +4,7 @@ import type { Transport } from "@/transport/transport";
import type {
ClickParams,
ConsoleParams,
DownloadParams,
EmulateParams,
EvaluateParams,
FillParams,
@@ -28,10 +29,12 @@ import type {
ScreenshotParams,
SelectParams,
SnapshotParams,
UploadParams,
WaitForNavigationParams,
} from "@/transport/types";
import { isRequestFrame } from "@/transport/types";
import { handleConsole } from "./console";
import { handleDownload } from "./download";
import { type EmulateCdpRunner, handleEmulate } from "./emulate";
import { handleEvaluate } from "./evaluate";
import { handleRequestHelp } from "./human-loop";
@@ -51,7 +54,12 @@ import {
handleScreenshot,
handleSnapshot,
} from "./observation";
import { handleRecordAwait, handleRecordStart, handleRecordStop } from "./record";
import {
handleRecordAwait,
handleRecordStart,
handleRecordStop,
type RecordRuntimeDeps,
} from "./record";
import {
handleSessionStart,
handleSessionStop,
@@ -74,6 +82,7 @@ import {
type TabReturnParams,
type TabSelectParams,
} from "./tabs";
import { handleUpload } from "./upload";
import { handleWaitForNavigation } from "./waits";
import { handleWindowResize, type WindowResizeParams } from "./window";
@@ -99,6 +108,7 @@ export interface DispatcherDeps {
transport: Transport;
sessions: SessionManager;
cdp?: DispatcherCdpRunner;
recording?: RecordRuntimeDeps;
/**
* Invoked whenever a dispatched RPC may have changed the live
* session set (currently `tool.session_start` and
@@ -134,6 +144,7 @@ export class ToolDispatcher {
private readonly transport: Transport;
private readonly sessions: SessionManager;
private readonly cdp?: DispatcherCdpRunner;
private readonly recording?: RecordRuntimeDeps;
private readonly onSessionsChanged?: () => void;
private readonly onBrowserControlResumed?: (sessionId: string) => void;
private readonly approveBorrow?: BorrowConfirmationApprover;
@@ -153,6 +164,7 @@ export class ToolDispatcher {
this.transport = deps.transport;
this.sessions = deps.sessions;
this.cdp = deps.cdp;
this.recording = deps.recording;
this.onSessionsChanged = deps.onSessionsChanged;
this.onBrowserControlResumed = deps.onBrowserControlResumed;
this.approveBorrow = deps.approveBorrow;
@@ -290,6 +302,7 @@ export class ToolDispatcher {
await this.releaseHoverLatch((req.params as SessionStopParams).session_id);
return handleSessionStop(this.sessions, req.params as SessionStopParams, {
cdp: this.cdp,
signal,
});
}
case "tool.tab_list":
@@ -374,7 +387,11 @@ export class ToolDispatcher {
? {
cdp: this.cdp,
tabsApi: chromeTabsCaptureApi,
conditionalSurfaceProbe: !this.hasHoverLatchForScope(hoverScope),
// Active hover probing is opt-in. A held hover latch still
// suppresses it, because probing would move the cursor off
// the element the caller is deliberately holding.
conditionalSurfaceProbe:
params.probe_hover === true && !this.hasHoverLatchForScope(hoverScope),
hoverProbeBypassOverlay: bypassOverlay,
}
: undefined,
@@ -504,6 +521,40 @@ export class ToolDispatcher {
),
signal,
);
case "tool.upload":
return this.withHoverReleaseForRequest(
req.params as UploadParams,
() =>
this.cdp
? handleUpload(this.sessions, req.params as UploadParams, {
cdp: this.cdp,
tabsApi: chromeTabsApi,
signal,
bypassOverlay,
})
: Promise.resolve({
code: "unsupported",
message: "upload requires CDP",
} satisfies RpcError),
signal,
);
case "tool.download":
return this.withHoverReleaseForRequest(
req.params as DownloadParams,
() =>
this.cdp
? handleDownload(this.sessions, req.params as DownloadParams, {
cdp: this.cdp,
tabsApi: chromeTabsApi,
signal,
bypassOverlay,
})
: Promise.resolve({
code: "unsupported",
message: "download requires CDP",
} satisfies RpcError),
signal,
);
case "tool.evaluate":
return handleEvaluate(
this.sessions,
@@ -530,44 +581,26 @@ export class ToolDispatcher {
signal,
});
case "tool.record_start":
return handleRecordStart(this.sessions, req.params as RecordStartParams, {
tabsApi: chromeTabsApi,
sendToTab: (tabId, msg) => chrome.tabs.sendMessage(tabId, msg),
bypassOverlay: async (tabId, enabled) => {
try {
await chrome.tabs.sendMessage(tabId, {
type: OVERLAY_AUTOMATION_BYPASS,
enabled,
});
} catch {
// Content script may be unavailable on restricted pages.
}
},
...(this.cdp ? { cdp: this.cdp } : {}),
signal,
});
return this.recording
? handleRecordStart(this.sessions, req.params as RecordStartParams, {
...this.recording,
signal,
})
: recordingRuntimeUnavailable();
case "tool.record_stop":
return handleRecordStop(this.sessions, req.params as RecordStopParams, {
tabsApi: chromeTabsApi,
sendToTab: (tabId, msg) => chrome.tabs.sendMessage(tabId, msg),
bypassOverlay: async (tabId, enabled) => {
try {
await chrome.tabs.sendMessage(tabId, {
type: OVERLAY_AUTOMATION_BYPASS,
enabled,
});
} catch {
// Content script may be unavailable on restricted pages.
}
},
signal,
});
return this.recording
? handleRecordStop(this.sessions, req.params as RecordStopParams, {
...this.recording,
signal,
})
: recordingRuntimeUnavailable();
case "tool.record_await":
return handleRecordAwait(this.sessions, req.params as RecordAwaitParams, {
tabsApi: chromeTabsApi,
sendToTab: (tabId, msg) => chrome.tabs.sendMessage(tabId, msg),
signal,
});
return this.recording
? handleRecordAwait(this.sessions, req.params as RecordAwaitParams, {
...this.recording,
signal,
})
: recordingRuntimeUnavailable();
default:
return {
code: "unknown_method",
@@ -703,6 +736,13 @@ function isRpcError(v: unknown): v is RpcError {
);
}
function recordingRuntimeUnavailable(): RpcError {
return {
code: "protocol_error",
message: "recording runtime is unavailable",
};
}
function sessionIdForBrowserControlMethod(req: RequestFrame): string | null {
switch (req.method) {
case "tool.tab_create":
@@ -721,6 +761,8 @@ function sessionIdForBrowserControlMethod(req: RequestFrame): string | null {
case "tool.fill":
case "tool.press":
case "tool.select":
case "tool.upload":
case "tool.download":
case "tool.evaluate":
case "tool.observe":
case "tool.request_help":
@@ -0,0 +1,364 @@
// Order-independent coordinator for one browser download. CDP supplies the
// exact target/frame intent while chrome.downloads supplies the download id
// and filename routing hook; neither event is assumed to arrive first.
import type { CdpTarget } from "@/browser-driver/frame-graph";
import type { ClickResult, RpcError, TransferEffectState } from "@/transport/types";
import { transferError } from "./errors";
import { type CdpRunner, isRpcError } from "./shared";
const CORRELATION_GRACE_MS = 750;
const UNIQUE_SETTLE_MS = 50;
const SIZE_POLL_MS = 250;
type DeterminingFilenameListener = (
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true;
interface ListenerEvent<T> {
addListener(listener: T): void;
removeListener(listener: T): void;
}
export interface DownloadsApi {
onCreated: ListenerEvent<(item: chrome.downloads.DownloadItem) => void>;
onChanged: ListenerEvent<(delta: chrome.downloads.DownloadDelta) => void>;
onDeterminingFilename: ListenerEvent<DeterminingFilenameListener>;
search(query: chrome.downloads.DownloadQuery): Promise<chrome.downloads.DownloadItem[]>;
cancel(downloadId: number): Promise<void>;
removeFile(downloadId: number): Promise<void>;
}
export const chromeDownloadsApi: DownloadsApi = {
get onCreated() {
return chrome.downloads.onCreated;
},
get onChanged() {
return chrome.downloads.onChanged;
},
get onDeterminingFilename() {
return chrome.downloads.onDeterminingFilename;
},
search: (query) => chrome.downloads.search(query),
cancel: (id) => chrome.downloads.cancel(id),
removeFile: (id) => chrome.downloads.removeFile(id),
};
export interface DownloadCaptureOptions {
cdp: CdpRunner;
target: CdpTarget;
expectedFrameId?: string;
downloads: DownloadsApi;
browserRelativeDir: string;
maxByteSize?: number;
timeoutMs: number;
signal?: AbortSignal;
trigger(): Promise<ClickResult | RpcError>;
}
export interface DownloadCaptureResult {
click: ClickResult;
item: chrome.downloads.DownloadItem;
}
interface DownloadIntent {
url: string;
suggestedFilename: string;
frameId?: string;
}
interface DownloadCandidate {
item: chrome.downloads.DownloadItem;
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void;
suggested: boolean;
graceTimer: ReturnType<typeof setTimeout>;
}
function safeBasename(filename: string): string {
const basename = filename.split(/[\\/]/).pop()?.trim();
return basename && basename !== "." && basename !== ".." ? basename : "download";
}
function sameTarget(source: { tabId?: number; sessionId?: string }, target: CdpTarget): boolean {
return source.tabId === target.tabId && source.sessionId === target.sessionId;
}
function matchesIntent(item: chrome.downloads.DownloadItem, intent: DownloadIntent): boolean {
const urlMatches = item.url === intent.url || item.finalUrl === intent.url;
return urlMatches && safeBasename(item.filename) === safeBasename(intent.suggestedFilename);
}
function knownSize(item: chrome.downloads.DownloadItem): number | undefined {
if (item.fileSize >= 0) return item.fileSize;
if (item.totalBytes >= 0) return item.totalBytes;
return undefined;
}
function captureError(
message: string,
effectState: TransferEffectState,
phase: string,
cleanupFailed = false,
): RpcError {
return transferError("cdp_failed", "download_capture_failed", message, {
effectState,
phase,
...(cleanupFailed ? { cleanupState: "failed" } : {}),
});
}
async function cleanupClaimedDownload(downloads: DownloadsApi, downloadId: number): Promise<void> {
const lookup = async () => (await downloads.search({ id: downloadId }))[0];
const item = await lookup();
if (!item || item.state === "interrupted") return;
if (item.state === "complete") {
await downloads.removeFile(downloadId);
return;
}
try {
await downloads.cancel(downloadId);
} catch (cancelError) {
// Completion can win the race after the lookup but before cancellation.
// Reconcile against Chrome's authoritative state before declaring cleanup
// failed so every terminal state has one explicit cleanup path.
const reconciled = await lookup();
if (!reconciled || reconciled.state === "interrupted") return;
if (reconciled.state === "complete") {
await downloads.removeFile(downloadId);
return;
}
throw cancelError;
}
}
export async function captureBrowserDownload(
options: DownloadCaptureOptions,
): Promise<DownloadCaptureResult | RpcError> {
let click: ClickResult | undefined;
let intent: DownloadIntent | undefined;
let capturedId: number | undefined;
let settled = false;
let succeeded = false;
let failureResult: RpcError | undefined;
let uniquenessTimer: ReturnType<typeof setTimeout> | undefined;
let operationTimer: ReturnType<typeof setTimeout> | undefined;
let sizePoll: ReturnType<typeof setInterval> | undefined;
const candidates = new Map<number, DownloadCandidate>();
const createdItems = new Map<number, chrome.downloads.DownloadItem>();
let resolveCompletion!: (item: chrome.downloads.DownloadItem) => void;
let rejectCompletion!: (error: Error) => void;
const completion = new Promise<chrome.downloads.DownloadItem>((resolve, reject) => {
resolveCompletion = resolve;
rejectCompletion = reject;
});
const fail = (error: Error) => {
if (settled) return;
settled = true;
rejectCompletion(error);
};
const complete = (item: chrome.downloads.DownloadItem) => {
if (settled) return;
const size = knownSize(item);
if (size !== undefined && options.maxByteSize !== undefined && size > options.maxByteSize) {
fail(new Error(`download exceeds transfer limit ${options.maxByteSize}`));
return;
}
settled = true;
resolveCompletion(item);
};
const suggestDefault = (candidate: DownloadCandidate) => {
if (candidate.suggested) return;
candidate.suggested = true;
candidate.suggest();
};
const matchingCandidates = (): DownloadCandidate[] => {
const currentIntent = intent;
return currentIntent
? [...candidates.values()].filter(
(candidate) => !candidate.suggested && matchesIntent(candidate.item, currentIntent),
)
: [];
};
const claimUnique = () => {
uniquenessTimer = undefined;
if (settled || capturedId !== undefined || !intent) return;
const matches = matchingCandidates();
if (matches.length !== 1) {
if (matches.length > 1) {
for (const candidate of matches) suggestDefault(candidate);
fail(new Error("download attribution is ambiguous"));
}
return;
}
const candidate = matches[0];
candidate.suggested = true;
clearTimeout(candidate.graceTimer);
capturedId = candidate.item.id;
candidate.suggest({
filename: `${options.browserRelativeDir}/${safeBasename(intent.suggestedFilename)}`,
conflictAction: "overwrite",
});
const size = knownSize(candidate.item);
if (size !== undefined && options.maxByteSize !== undefined && size > options.maxByteSize) {
fail(new Error(`download exceeds transfer limit ${options.maxByteSize}`));
return;
}
const created = createdItems.get(candidate.item.id);
if (created?.state === "interrupted") {
fail(new Error(created.error ?? "download interrupted"));
} else if (created?.state === "complete") {
complete(created);
}
};
const reconcile = () => {
if (settled || capturedId !== undefined || !intent) return;
const matches = matchingCandidates();
if (matches.length > 1) {
for (const candidate of matches) suggestDefault(candidate);
fail(new Error("download attribution is ambiguous"));
return;
}
if (matches.length === 1 && !uniquenessTimer) {
uniquenessTimer = setTimeout(claimUnique, UNIQUE_SETTLE_MS);
}
};
const determiningListener: DeterminingFilenameListener = (item, suggest) => {
const candidate: DownloadCandidate = {
item,
suggest,
suggested: false,
graceTimer: setTimeout(() => {
suggestDefault(candidate);
candidates.delete(item.id);
if (intent && matchesIntent(item, intent) && capturedId === undefined) {
fail(new Error("download correlation grace elapsed before unique attribution"));
}
}, CORRELATION_GRACE_MS),
};
candidates.set(item.id, candidate);
reconcile();
return true;
};
const createdListener = (item: chrome.downloads.DownloadItem) => {
createdItems.set(item.id, item);
if (capturedId !== item.id) return;
if (item.state === "interrupted") {
fail(new Error(item.error ?? "download interrupted"));
} else if (item.state === "complete") {
complete(item);
}
};
const changedListener = async (delta: chrome.downloads.DownloadDelta) => {
if (capturedId === undefined || delta.id !== capturedId || settled) return;
if (delta.state?.current === "interrupted" || delta.error?.current) {
fail(new Error(delta.error?.current ?? "download interrupted"));
return;
}
if (delta.state?.current === "complete") {
try {
const [item] = await options.downloads.search({ id: delta.id });
if (item) complete(item);
else fail(new Error("completed download disappeared"));
} catch (err) {
fail(err instanceof Error ? err : new Error(String(err)));
}
}
};
const onAbort = () => fail(new DOMException("aborted", "AbortError"));
const cdpSubscription = options.cdp.onEvent?.((source, method, raw) => {
if (method !== "Page.downloadWillBegin" || !sameTarget(source, options.target)) return;
const event = raw as { url?: unknown; suggestedFilename?: unknown; frameId?: unknown };
if (typeof event.url !== "string" || typeof event.suggestedFilename !== "string") return;
if (options.expectedFrameId && event.frameId !== options.expectedFrameId) {
fail(new Error("download originated from a different frame"));
return;
}
if (intent) {
fail(new Error("download trigger produced more than one browser download intent"));
return;
}
intent = {
url: event.url,
suggestedFilename: event.suggestedFilename,
...(typeof event.frameId === "string" ? { frameId: event.frameId } : {}),
};
reconcile();
});
if (!cdpSubscription) {
return captureError("CDP download intent subscription unavailable", "none", "arm");
}
options.downloads.onDeterminingFilename.addListener(determiningListener);
options.downloads.onCreated.addListener(createdListener);
options.downloads.onChanged.addListener(changedListener);
options.signal?.addEventListener("abort", onAbort, { once: true });
operationTimer = setTimeout(
() => fail(new Error("download did not complete before timeout")),
options.timeoutMs,
);
sizePoll = setInterval(() => {
if (capturedId === undefined || settled || options.maxByteSize === undefined) return;
void options.downloads
.search({ id: capturedId })
.then(([item]) => {
if (!item || settled) return;
if (item.bytesReceived > (options.maxByteSize as number)) {
fail(new Error(`download exceeds transfer limit ${options.maxByteSize}`));
}
})
.catch((err) => fail(err instanceof Error ? err : new Error(String(err))));
}, SIZE_POLL_MS);
try {
const triggered = await options.trigger();
if (isRpcError(triggered)) {
void completion.catch(() => undefined);
const effect: TransferEffectState =
capturedId !== undefined ? "committed" : intent ? "unknown" : "none";
failureResult = {
...triggered,
data: { ...triggered.data, effect_state: effect, phase: "trigger" },
};
return failureResult;
}
click = triggered;
const item = await completion;
succeeded = true;
return { click, item };
} catch (err) {
const effect: TransferEffectState =
capturedId !== undefined ? "committed" : click ? "unknown" : "none";
failureResult = captureError(
err instanceof Error ? err.message : String(err),
effect,
capturedId !== undefined ? "download" : "attribution",
);
return failureResult;
} finally {
settled = true;
if (operationTimer) clearTimeout(operationTimer);
if (uniquenessTimer) clearTimeout(uniquenessTimer);
if (sizePoll) clearInterval(sizePoll);
options.signal?.removeEventListener("abort", onAbort);
options.downloads.onDeterminingFilename.removeListener(determiningListener);
options.downloads.onCreated.removeListener(createdListener);
options.downloads.onChanged.removeListener(changedListener);
cdpSubscription.dispose();
for (const candidate of candidates.values()) {
clearTimeout(candidate.graceTimer);
if (candidate.item.id !== capturedId) suggestDefault(candidate);
}
if (!succeeded && capturedId !== undefined) {
try {
await cleanupClaimedDownload(options.downloads, capturedId);
} catch {
if (failureResult?.data) failureResult.data.cleanup_state = "failed";
}
}
}
}
+64
View File
@@ -0,0 +1,64 @@
// Download orchestration: validate session/tab ownership, then delegate one
// browser-global chrome.downloads transaction to download-capture.ts.
import type { SessionManager } from "@/session-manager/manager";
import type { DownloadParams, DownloadResult, RpcError } from "@/transport/types";
import { captureBrowserDownload, chromeDownloadsApi, type DownloadsApi } from "./download-capture";
import { clickResolvedTarget, type InteractionDeps, resolveActionTarget } from "./interaction";
import { enforceAgentWindow, isRpcError, lookupSession, resolveTargetTab } from "./shared";
let downloadActive = false;
export type { DownloadsApi } from "./download-capture";
export interface DownloadDeps extends InteractionDeps {
downloads?: DownloadsApi;
}
export async function handleDownload(
manager: SessionManager,
params: DownloadParams,
deps: DownloadDeps,
): Promise<DownloadResult | RpcError> {
if (downloadActive) return { code: "invalid_params", message: "another bsk download is active" };
downloadActive = true;
try {
const ctx = lookupSession(manager, params, "download");
if (isRpcError(ctx)) return ctx;
const target = await resolveTargetTab(manager, ctx, params.tab_id, deps.tabsApi);
if (isRpcError(target)) return target;
const denied = enforceAgentWindow(ctx, target, "download");
if (denied) return denied;
if (!params.browser_relative_dir) {
return { code: "invalid_params", message: "download requires a daemon capability directory" };
}
const address = await resolveActionTarget(deps.cdp, ctx, target, params, "download");
if (isRpcError(address)) return address;
const capture = await captureBrowserDownload({
cdp: deps.cdp,
target: address.cdpTarget,
downloads: deps.downloads ?? chromeDownloadsApi,
browserRelativeDir: params.browser_relative_dir,
maxByteSize: params.max_byte_size,
timeoutMs: params.timeout_ms ?? 120_000,
signal: deps.signal,
expectedFrameId: address.frameId,
trigger: () => clickResolvedTarget(ctx, address, {}, deps),
});
if (isRpcError(capture)) return capture;
const { click, item } = capture;
return {
tab_id: target.tabId,
used_ref: click.used_ref,
used_selector: click.used_selector,
suggested_filename: item.filename.split(/[\\/]/).pop() ?? "download",
byte_size: item.fileSize >= 0 ? item.fileSize : item.totalBytes,
mime: item.mime || undefined,
danger: item.danger,
browser_path: item.filename,
};
} finally {
downloadActive = false;
}
}
+27 -1
View File
@@ -2,10 +2,23 @@
// rendering. Extension handlers attach reasons here; human-facing copy
// lives in bsk-cli `render_error.rs`.
import type { ErrorCode, RpcError, RpcErrorData, RpcErrorReason } from "@/transport/types";
import type {
ErrorCode,
RpcError,
RpcErrorData,
RpcErrorReason,
TransferCleanupState,
TransferEffectState,
} from "@/transport/types";
export type { RpcErrorData, RpcErrorReason };
export interface TransferErrorOptions {
effectState: TransferEffectState;
phase: string;
cleanupState?: TransferCleanupState;
}
export function rpcError(
code: ErrorCode,
reason: RpcErrorReason,
@@ -15,3 +28,16 @@ export function rpcError(
const data: RpcErrorData = { reason, ...extra };
return { code, message, data };
}
export function transferError(
code: ErrorCode,
reason: RpcErrorReason,
message: string,
options: TransferErrorOptions,
): RpcError {
return rpcError(code, reason, message, {
effect_state: options.effectState,
phase: options.phase,
...(options.cleanupState ? { cleanup_state: options.cleanupState } : {}),
});
}
@@ -0,0 +1,319 @@
// One native file-drop transaction bound to an explicit page target. CDP drag
// events are sent to the target that owns the node: root targets use top-level
// coordinates, while OOPIF targets require coordinates local to their session.
import type { RpcError, TransferEffectState } from "@/transport/types";
import { transferError } from "./errors";
import { resolveNodeGeometry } from "./frame-geometry";
import type { ResolvedActionTarget } from "./interaction";
import { type CdpRunner, isRpcError, sendToCdpTarget } from "./shared";
import { BoundedWaitError, waitBounded } from "./transfer-transaction";
const CLEANUP_TIMEOUT_MS = 1_000;
type FileDropPhase = "resolve_target" | "drag_enter" | "drag_over" | "drop" | "cleanup";
interface RuntimeReply {
result?: { value?: unknown; objectId?: string };
exceptionDetails?: { text?: string; exception?: { description?: string } };
}
interface DragData {
items: [];
files: string[];
dragOperationsMask: number;
}
export interface FileDropTransactionOptions {
cdp: CdpRunner;
actionTarget: ResolvedActionTarget;
files: string[];
timeoutMs: number;
signal?: AbortSignal;
withOverlayHidden: <T>(operation: () => Promise<T>) => Promise<T>;
}
export interface FileDropTransactionResult {
x: number;
y: number;
}
function failure(
code: RpcError["code"],
reason: "upload_mechanism_unsupported" | "file_drop_target_unavailable" | "file_drop_failed",
message: string,
effectState: TransferEffectState,
phase: FileDropPhase,
): RpcError {
const error = transferError(code, reason, message, { effectState, phase });
error.data = { ...error.data, mechanism: "drop" };
return error;
}
function enrichFailure(
error: RpcError,
effectState: TransferEffectState,
phase: FileDropPhase,
): RpcError {
return {
...error,
data: {
...error.data,
mechanism: "drop",
effect_state: effectState,
phase,
},
};
}
function messageOf(error: unknown): string {
return error instanceof Error ? error.message : String(error);
}
function methodUnavailable(error: unknown): boolean {
return /(?:method|command).*(?:not found|wasn't found|unsupported)|wasn't found|-32601/i.test(
messageOf(error),
);
}
function runtimeError(reply: RuntimeReply, fallback: string): Error | null {
if (!reply.exceptionDetails) return null;
return new Error(
reply.exceptionDetails.exception?.description ?? reply.exceptionDetails.text ?? fallback,
);
}
async function pointBelongsToTarget(
options: FileDropTransactionOptions,
x: number,
y: number,
objectGroup: string,
): Promise<boolean> {
const target = options.actionTarget.cdpTarget;
const hit = await sendToCdpTarget<{ backendNodeId?: number; frameId?: string }>(
options.cdp,
target,
"DOM.getNodeForLocation",
{ x: Math.round(x), y: Math.round(y), includeUserAgentShadowDOM: true },
);
if (typeof hit.backendNodeId !== "number") return false;
if (options.actionTarget.frameId && hit.frameId !== options.actionTarget.frameId) return false;
const [resolvedTarget, resolvedHit] = await Promise.all([
sendToCdpTarget<{ object?: { objectId?: string } }>(options.cdp, target, "DOM.resolveNode", {
backendNodeId: options.actionTarget.backendNodeId,
objectGroup,
}),
sendToCdpTarget<{ object?: { objectId?: string } }>(options.cdp, target, "DOM.resolveNode", {
backendNodeId: hit.backendNodeId,
objectGroup,
}),
]);
const targetObjectId = resolvedTarget.object?.objectId;
const hitObjectId = resolvedHit.object?.objectId;
if (!targetObjectId || !hitObjectId) return false;
const reply = await sendToCdpTarget<RuntimeReply>(options.cdp, target, "Runtime.callFunctionOn", {
objectId: targetObjectId,
functionDeclaration: `function(hit) {
for (let node = hit; node; ) {
if (node === this) return true;
if (this.contains?.(node)) return true;
const root = node.getRootNode?.();
node = node.parentNode || (root && root !== node ? root.host : null);
}
return false;
}`,
arguments: [{ objectId: hitObjectId }],
returnByValue: true,
});
const error = runtimeError(reply, "failed to verify file-drop target");
if (error) throw error;
return reply.result?.value === true;
}
export async function uploadThroughFileDrop(
options: FileDropTransactionOptions,
): Promise<FileDropTransactionResult | RpcError> {
const deadline = Date.now() + options.timeoutMs;
const target = options.actionTarget.cdpTarget;
const objectGroup = `bsk-file-drop-${crypto.randomUUID()}`;
const dragData: DragData = {
items: [],
files: options.files,
dragOperationsMask: 1,
};
let dragStarted = false;
let dropSent = false;
let dragPoint = { x: 0, y: 0 };
let cleanupFailed = false;
let outcome: FileDropTransactionResult | RpcError = failure(
"protocol_error",
"file_drop_failed",
"file-drop transaction ended without an outcome",
"none",
"cleanup",
);
try {
const geometry = await waitBounded(
resolveNodeGeometry(
options.cdp,
options.actionTarget.tab.tabId,
{
target,
backendNodeId: options.actionTarget.backendNodeId,
...(options.actionTarget.frameId ? { frameId: options.actionTarget.frameId } : {}),
},
{ scrollIntoView: true },
),
deadline,
options.signal,
"resolving file-drop target timed out",
);
if (isRpcError(geometry)) {
outcome = enrichFailure(geometry, "none", "resolve_target");
return outcome;
}
const point = geometry.targetActionPoint;
dragPoint = point;
outcome = await options.withOverlayHidden(async () => {
try {
try {
const belongs = await waitBounded(
pointBelongsToTarget(options, point.x, point.y, objectGroup),
deadline,
options.signal,
"verifying file-drop target timed out",
);
if (!belongs) {
return failure(
"permission_denied",
"file_drop_target_unavailable",
"file-drop target is not the topmost element at its action point",
"none",
"resolve_target",
);
}
} catch (error) {
return failure(
error instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"file_drop_target_unavailable",
messageOf(error),
"none",
"resolve_target",
);
}
for (const [type, phase] of [
["dragEnter", "drag_enter"],
["dragOver", "drag_over"],
] as const) {
try {
if (type === "dragEnter") dragStarted = true;
await waitBounded(
sendToCdpTarget(options.cdp, target, "Input.dispatchDragEvent", {
type,
x: point.x,
y: point.y,
data: dragData,
}),
deadline,
options.signal,
`${type} timed out`,
);
} catch (error) {
const unsupported = methodUnavailable(error);
if (type === "dragEnter" && unsupported) dragStarted = false;
return failure(
unsupported
? "unsupported"
: error instanceof BoundedWaitError
? "timeout"
: "cdp_failed",
unsupported ? "upload_mechanism_unsupported" : "file_drop_failed",
messageOf(error),
"none",
phase,
);
}
}
try {
dropSent = true;
await waitBounded(
sendToCdpTarget(options.cdp, target, "Input.dispatchDragEvent", {
type: "drop",
x: point.x,
y: point.y,
data: dragData,
}),
deadline,
options.signal,
"file drop timed out",
);
return { x: point.x, y: point.y };
} catch (error) {
outcome = failure(
error instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"file_drop_failed",
messageOf(error),
"unknown",
"drop",
);
return outcome;
}
} finally {
if (dragStarted && !dropSent) {
try {
await waitBounded(
sendToCdpTarget(options.cdp, target, "Input.dispatchDragEvent", {
type: "dragCancel",
x: dragPoint.x,
y: dragPoint.y,
data: dragData,
}),
Date.now() + CLEANUP_TIMEOUT_MS,
undefined,
"cancelling file drag timed out",
);
} catch {
cleanupFailed = true;
}
}
}
});
return outcome;
} catch (error) {
outcome = failure(
error instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"file_drop_target_unavailable",
messageOf(error),
"none",
"resolve_target",
);
return outcome;
} finally {
try {
await waitBounded(
sendToCdpTarget(options.cdp, target, "Runtime.releaseObjectGroup", { objectGroup }),
Date.now() + CLEANUP_TIMEOUT_MS,
undefined,
"releasing file-drop object group timed out",
);
} catch {
cleanupFailed = true;
}
const effect = isRpcError(outcome)
? (outcome.data?.effect_state as TransferEffectState | undefined)
: "committed";
if (effect === "unknown" || cleanupFailed) {
await options.cdp.detach?.(target.tabId);
}
if (cleanupFailed && isRpcError(outcome)) {
outcome.data = { ...outcome.data, cleanup_state: "failed" };
}
}
}
@@ -0,0 +1,445 @@
// One upload transaction with an explicit browser-side commit boundary.
// Chrome interception prevents a native chooser from escaping automation;
// chooser events and a frame-scoped DOM probe are independent input-location
// signals, so event delivery is not required for standard file inputs.
import type { ClickResult, RpcError, TransferEffectState } from "@/transport/types";
import { transferError } from "./errors";
import type { ResolvedActionTarget } from "./interaction";
import { type CdpRunner, isRpcError, sendToCdpTarget } from "./shared";
import { BoundedWaitError, remainingMs, waitBounded } from "./transfer-transaction";
const CLEANUP_TIMEOUT_MS = 1_000;
const ACTIVATION_GRACE_MS = 1_000;
const PROBE_INTERVAL_MS = 20;
type UploadPhase =
| "arm_interception"
| "arm_input_probe"
| "trigger"
| "resolve_input"
| "set_files"
| "cleanup";
interface RuntimeReply {
result?: { value?: unknown; objectId?: string };
exceptionDetails?: { text?: string; exception?: { description?: string } };
}
interface ChooserEvent {
frameId?: string;
backendNodeId?: number;
mode?: "selectSingle" | "selectMultiple";
}
export interface FileInputTransactionOptions {
cdp: CdpRunner;
actionTarget: ResolvedActionTarget;
files: string[];
timeoutMs: number;
signal?: AbortSignal;
trigger(): Promise<ClickResult | RpcError>;
}
export interface FileInputTransactionResult {
click: ClickResult;
multiple: boolean;
}
function runtimeError(reply: RuntimeReply, fallback: string): Error | null {
if (!reply.exceptionDetails) return null;
return new Error(
reply.exceptionDetails.exception?.description ?? reply.exceptionDetails.text ?? fallback,
);
}
function transferFailure(
code: RpcError["code"],
reason: "file_input_probe_failed" | "file_input_not_activated" | "set_file_input_failed",
message: string,
effectState: TransferEffectState,
phase: UploadPhase,
): RpcError {
return transferError(code, reason, message, { effectState, phase });
}
function enrichFailure(
error: RpcError,
effectState: TransferEffectState,
phase: UploadPhase,
): RpcError {
return {
...error,
data: { ...error.data, effect_state: effectState, phase },
};
}
function sameTarget(
source: { tabId?: number; sessionId?: string },
target: { tabId: number; sessionId?: string },
): boolean {
return source.tabId === target.tabId && source.sessionId === target.sessionId;
}
async function delay(ms: number, signal?: AbortSignal): Promise<void> {
await waitBounded(
new Promise((resolve) => setTimeout(resolve, ms)),
Date.now() + ms + 1,
signal,
"upload activation probe timed out",
);
}
async function callOnTrigger<T = RuntimeReply>(
options: FileInputTransactionOptions,
objectId: string,
functionDeclaration: string,
args: unknown[],
returnByValue: boolean,
): Promise<T> {
return sendToCdpTarget<T>(options.cdp, options.actionTarget.cdpTarget, "Runtime.callFunctionOn", {
objectId,
functionDeclaration,
arguments: args.map((value) => ({ value })),
returnByValue,
awaitPromise: false,
});
}
export async function uploadThroughActivatedFileInput(
options: FileInputTransactionOptions,
): Promise<FileInputTransactionResult | RpcError> {
const deadline = Date.now() + options.timeoutMs;
const target = options.actionTarget.cdpTarget;
const objectGroup = `bsk-upload-${crypto.randomUUID()}`;
const stateKey = `__bskUpload_${crypto.randomUUID().replaceAll("-", "")}`;
const chooserEvents: ChooserEvent[] = [];
let interceptionArmed = false;
let probeArmed = false;
let triggerObjectId: string | undefined;
let outcome: FileInputTransactionResult | RpcError = transferFailure(
"protocol_error",
"file_input_probe_failed",
"upload transaction ended without an outcome",
"none",
"cleanup",
);
const chooserSubscription = options.cdp.onEvent?.((source, method, raw) => {
if (method !== "Page.fileChooserOpened" || !sameTarget(source, target)) return;
const event = raw as ChooserEvent;
chooserEvents.push(event);
});
try {
try {
await waitBounded(
sendToCdpTarget(options.cdp, target, "Page.setInterceptFileChooserDialog", {
enabled: true,
}),
deadline,
options.signal,
"arming native file chooser interception timed out",
);
interceptionArmed = true;
} catch (err) {
outcome = transferFailure(
err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"file_input_probe_failed",
err instanceof Error ? err.message : String(err),
"none",
"arm_interception",
);
return outcome;
}
try {
const resolved = await waitBounded(
sendToCdpTarget<{ object?: { objectId?: string } }>(
options.cdp,
target,
"DOM.resolveNode",
{
backendNodeId: options.actionTarget.backendNodeId,
objectGroup,
},
),
deadline,
options.signal,
"resolving upload trigger timed out",
);
triggerObjectId = resolved.object?.objectId;
if (!triggerObjectId) throw new Error("DOM.resolveNode returned no trigger objectId");
const armed = await waitBounded(
callOnTrigger<RuntimeReply>(
options,
triggerObjectId,
`function(key) {
const doc = this.ownerDocument;
const owner = doc.defaultView;
if (!owner) return false;
const state = { inputs: [], listener: null };
Object.defineProperty(owner, key, { value: state, configurable: true });
state.listener = event => {
const path = typeof event.composedPath === "function" ? event.composedPath() : [];
const candidate = path[0] || event.target;
if (candidate && candidate.nodeType === 1 &&
candidate.localName === "input" && candidate.type === "file") {
if (!state.inputs.includes(candidate)) state.inputs.push(candidate);
event.preventDefault();
}
};
doc.addEventListener("click", state.listener, true);
return true;
}`,
[stateKey],
true,
),
deadline,
options.signal,
"arming frame-scoped file input probe timed out",
);
const armError = runtimeError(armed, "failed to arm file input probe");
if (armError || armed.result?.value !== true) {
throw armError ?? new Error("file input probe did not arm");
}
probeArmed = true;
} catch (err) {
outcome = transferFailure(
err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"file_input_probe_failed",
err instanceof Error ? err.message : String(err),
"none",
"arm_input_probe",
);
return outcome;
}
let click: ClickResult | RpcError;
try {
click = await waitBounded(
options.trigger(),
deadline,
options.signal,
"upload trigger timed out",
);
} catch (err) {
outcome = transferFailure(
err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"file_input_probe_failed",
err instanceof Error ? err.message : String(err),
"none",
"trigger",
);
return outcome;
}
if (isRpcError(click)) {
outcome = enrichFailure(click, "none", "trigger");
return outcome;
}
try {
const activationDeadline = Math.min(deadline, Date.now() + ACTIVATION_GRACE_MS);
let summary: { count: number; multiple: boolean } = { count: 0, multiple: false };
while (Date.now() < activationDeadline) {
if (chooserEvents.length > 0) break;
const reply = await callOnTrigger<RuntimeReply>(
options,
triggerObjectId,
`function(key) {
const state = this.ownerDocument.defaultView?.[key];
return state
? { count: state.inputs.length, multiple: state.inputs[0]?.multiple === true }
: { count: 0, multiple: false };
}`,
[stateKey],
true,
);
const summaryError = runtimeError(reply, "failed to inspect activated file input");
if (summaryError) throw summaryError;
const value = reply.result?.value as { count?: unknown; multiple?: unknown } | undefined;
summary = {
count: typeof value?.count === "number" ? value.count : 0,
multiple: value?.multiple === true,
};
if (summary.count > 0) break;
await delay(Math.min(PROBE_INTERVAL_MS, remainingMs(activationDeadline)), options.signal);
}
if (chooserEvents.length > 1) {
outcome = transferFailure(
"unsupported",
"file_input_not_activated",
"upload trigger activated more than one file chooser",
"none",
"resolve_input",
);
return outcome;
}
const chooser = chooserEvents[0];
let backendNodeId: number | undefined;
let multiple = summary.multiple;
if (chooser) {
if (options.actionTarget.frameId && chooser.frameId !== options.actionTarget.frameId) {
outcome = transferFailure(
"unsupported",
"file_input_not_activated",
"upload trigger activated a file chooser in a different frame",
"none",
"resolve_input",
);
return outcome;
}
if (typeof chooser.backendNodeId !== "number") {
outcome = transferFailure(
"unsupported",
"file_input_not_activated",
"upload trigger invoked a non-input file picker",
"none",
"resolve_input",
);
return outcome;
}
backendNodeId = chooser.backendNodeId;
multiple = chooser.mode === "selectMultiple";
} else {
if (summary.count !== 1) {
outcome = transferFailure(
"unsupported",
"file_input_not_activated",
summary.count === 0
? "upload trigger did not activate an input[type=file]"
: "upload trigger activated more than one input[type=file]",
"none",
"resolve_input",
);
return outcome;
}
const input = await callOnTrigger<RuntimeReply>(
options,
triggerObjectId,
`function(key) { return this.ownerDocument.defaultView?.[key]?.inputs[0]; }`,
[stateKey],
false,
);
const inputError = runtimeError(input, "failed to resolve activated file input object");
if (inputError) throw inputError;
if (!input.result?.objectId) throw new Error("activated file input returned no objectId");
const described = await sendToCdpTarget<{ node?: { backendNodeId?: number } }>(
options.cdp,
target,
"DOM.describeNode",
{ objectId: input.result.objectId },
);
backendNodeId = described.node?.backendNodeId;
if (typeof backendNodeId !== "number") {
throw new Error("DOM.describeNode returned no file input backendNodeId");
}
}
if (!multiple && options.files.length !== 1) {
outcome = enrichFailure(
{ code: "invalid_params", message: "file input accepts exactly one file" },
"none",
"resolve_input",
);
return outcome;
}
try {
await waitBounded(
sendToCdpTarget(options.cdp, target, "DOM.setFileInputFiles", {
files: options.files,
backendNodeId,
}),
deadline,
options.signal,
"setting file input files timed out",
);
} catch (err) {
outcome = transferFailure(
err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"set_file_input_failed",
err instanceof Error ? err.message : String(err),
"unknown",
"set_files",
);
return outcome;
}
outcome = { click, multiple };
return outcome;
} catch (err) {
outcome = transferFailure(
err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"file_input_probe_failed",
err instanceof Error ? err.message : String(err),
"none",
"resolve_input",
);
return outcome;
}
} finally {
chooserSubscription?.dispose();
let cleanupFailed = false;
if (probeArmed && triggerObjectId) {
try {
await waitBounded(
callOnTrigger(
options,
triggerObjectId,
`function(key) {
const owner = this.ownerDocument.defaultView;
const state = owner?.[key];
if (state?.listener) this.ownerDocument.removeEventListener("click", state.listener, true);
if (owner) delete owner[key];
}`,
[stateKey],
true,
),
Date.now() + CLEANUP_TIMEOUT_MS,
undefined,
"cleaning file input probe timed out",
);
} catch {
cleanupFailed = true;
}
}
if (interceptionArmed) {
try {
await waitBounded(
sendToCdpTarget(options.cdp, target, "Page.setInterceptFileChooserDialog", {
enabled: false,
cancel: true,
}),
Date.now() + CLEANUP_TIMEOUT_MS,
undefined,
"disabling file chooser interception timed out",
);
} catch {
cleanupFailed = true;
}
}
try {
await waitBounded(
sendToCdpTarget(options.cdp, target, "Runtime.releaseObjectGroup", { objectGroup }),
Date.now() + CLEANUP_TIMEOUT_MS,
undefined,
"releasing upload object group timed out",
);
} catch {
cleanupFailed = true;
}
const effect = isRpcError(outcome)
? (outcome.data?.effect_state as TransferEffectState | undefined)
: "committed";
if (effect === "unknown" || cleanupFailed) {
await options.cdp.detach?.(target.tabId);
}
if (cleanupFailed && isRpcError(outcome)) {
outcome.data = { ...outcome.data, cleanup_state: "failed" };
}
}
}
+27 -1
View File
@@ -29,7 +29,10 @@ export interface NodeAddress {
export interface ResolvedNodeGeometry {
topVisibleRegions: Region;
topBounds: ViewportRect;
/** Point in the top-level tab viewport, used by root-target input events. */
actionPoint: Point;
/** Point in the addressed CDP target's viewport, used by OOPIF-local input events. */
targetActionPoint: Point;
}
function geometryError(message: string): RpcError {
@@ -262,11 +265,30 @@ export async function resolveNodeGeometry(
if (isRpcError(localRegion)) return localRegion;
let topVisibleRegions: Region;
let targetActionPoint: Point | null = null;
if (address.frameId && graph) {
const projection = await resolveFrameProjection(cdp, graph, address.frameId);
if (!projection)
return geometryError(`could not resolve frame geometry for ${address.frameId}`);
topVisibleRegions = projectRegionToViewport(localRegion, projection);
if (address.target.sessionId) {
const localViewport = projection.edges[0]?.sourceViewport ?? projection.topViewport;
const localVisibleRegions = localRegion
.map((polygon) =>
clipPolygon(
polygon,
rectPolygon({
x: 0,
y: 0,
w: localViewport.width,
h: localViewport.height,
}),
),
)
.filter((polygon) => polygon.length >= 3);
const localActionRegion = largestRegion(localVisibleRegions);
targetActionPoint = localActionRegion ? polygonCentroid(localActionRegion) : null;
}
} else {
const viewport = await targetViewport(cdp, address.target);
if (!viewport) return geometryError("could not resolve top viewport geometry");
@@ -283,7 +305,11 @@ export async function resolveNodeGeometry(
if (!topBounds || !actionPoint) {
return { code: "permission_denied", message: "element not visible" };
}
return { topVisibleRegions, topBounds, actionPoint };
if (address.target.sessionId && !targetActionPoint) {
return { code: "permission_denied", message: "element not visible in its target" };
}
targetActionPoint ??= actionPoint;
return { topVisibleRegions, topBounds, actionPoint, targetActionPoint };
} catch (error) {
return geometryError(error instanceof Error ? error.message : String(error));
}
+39 -9
View File
@@ -40,6 +40,7 @@ import {
enforceAgentWindow,
isRpcError,
lookupSession,
type ResolvedTargetTab,
resolveTargetTab,
} from "./shared";
import { resolveSnapshotRef } from "./snapshot-ref";
@@ -56,6 +57,15 @@ export interface InteractionDeps {
keepOverlayBypassAfterHover?: boolean;
}
export interface ResolvedActionTarget {
tab: ResolvedTargetTab;
backendNodeId: number;
cdpTarget: CdpTarget;
frameId?: string;
usedRef?: string;
usedSelector?: string;
}
const DEFAULT_TIMEOUT_MS = 30_000;
const DEFAULT_HOVER_SETTLE_MS = 200;
@@ -128,7 +138,7 @@ async function wait(ms: number, signal?: AbortSignal): Promise<void> {
* `RpcError` if the caller supplied neither (or both), or if neither
* lookup matched.
*/
async function resolveBackendNode(
export async function resolveBackendNode(
cdp: CdpRunner,
ctx: SessionContext,
target: { tabId: number },
@@ -215,6 +225,17 @@ async function resolveBackendNode(
}
}
export async function resolveActionTarget(
cdp: CdpRunner,
ctx: SessionContext,
target: ResolvedTargetTab,
params: { ref?: string; selector?: string },
toolName: string,
): Promise<ResolvedActionTarget | RpcError> {
const node = await resolveBackendNode(cdp, ctx, target, params, toolName);
return isRpcError(node) ? node : { tab: target, ...node };
}
// ---------------------------------------------------------------------------
// tool.click
// ---------------------------------------------------------------------------
@@ -233,10 +254,19 @@ export async function handleClick(
if (isRpcError(target)) return target;
const denied = enforceAgentWindow(ctx, target, "click");
if (denied) return denied;
const dialogCursor = markDialogCursor(deps.cdp, target.tabId);
const resolved = await resolveActionTarget(deps.cdp, ctx, target, params, "click");
if (isRpcError(resolved)) return resolved;
return clickResolvedTarget(ctx, resolved, params, deps);
}
const node = await resolveBackendNode(deps.cdp, ctx, target, params, "click");
if (isRpcError(node)) return node;
export async function clickResolvedTarget(
ctx: SessionContext,
resolved: ResolvedActionTarget,
params: Pick<ClickParams, "button" | "click_count" | "modifiers">,
deps: InteractionDeps,
): Promise<ClickResult | RpcError> {
const { tab: target } = resolved;
const dialogCursor = markDialogCursor(deps.cdp, target.tabId);
if (throwIfAborted(deps.signal)) {
return { code: "cancelled", message: "click aborted" };
@@ -247,9 +277,9 @@ export async function handleClick(
deps.cdp,
target.tabId,
{
target: node.cdpTarget,
backendNodeId: node.backendNodeId,
...(node.frameId ? { frameId: node.frameId } : {}),
target: resolved.cdpTarget,
backendNodeId: resolved.backendNodeId,
...(resolved.frameId ? { frameId: resolved.frameId } : {}),
},
{ scrollIntoView: true },
);
@@ -337,8 +367,8 @@ export async function handleClick(
return attachDialogs(deps.cdp, target.tabId, dialogCursor, {
tab_id: target.tabId,
used_ref: node.usedRef,
used_selector: node.usedSelector,
used_ref: resolved.usedRef,
used_selector: resolved.usedSelector,
x: centre.x,
y: centre.y,
});
File diff suppressed because it is too large Load Diff
+391 -74
View File
@@ -15,22 +15,37 @@ import {
type RecordQueryResponse,
type RecordStartAck,
type RecordStartMessage,
type RecordStepAck,
type RecordStopMessage,
} from "@/lib/record-bridge";
import { appendRecordedPayload, observeRecordedNavigation } from "@/lib/recording-step-buffer";
import { reduceTraceSteps, resolveTraceStartUrl } from "@/lib/trace-reducer";
import {
type RecordFrameCoordinator,
type RecordingCaptureScope,
recordFrameCoordinator,
} from "@/lib/recording/frame-coordinator";
import { RecordingObservationRuntime } from "@/lib/recording/recording-runtime";
import {
appendRecordedPayload,
observeRecordedNavigation,
type RecordingStepBuffer,
} from "@/lib/recording/step-buffer";
import { RecordingTabCoordinator, type TabActivation } from "@/lib/recording/tab-coordinator";
import { buildTraceV2 } from "@/lib/recording/trace-reducer-v2";
import type { RecordingDraftStep } from "@/lib/recording/types";
import type { SessionManager } from "@/session-manager/manager";
import { EXTENSION_VERSION } from "@/transport/handshake";
import type {
DraftTraceStep,
RecordAwaitParams,
RecordAwaitResult,
RecordedTrace,
RecordStartParams,
RecordStartResult,
RecordStopParams,
RecordStopResult,
RpcError,
Trace,
StopReason,
} from "@/transport/types";
import { TRACE_VERSION_V3 } from "@/transport/types";
import { handleNavigate } from "./navigation";
import {
type CdpRunner,
@@ -43,21 +58,46 @@ import {
interface ActiveRecording {
requestId: string;
tabId: number;
tabs: RecordingTabCoordinator;
agentWindowId: number;
startUrl?: string;
purpose?: string;
steps: DraftTraceStep[];
steps: RecordingDraftStep[];
startedAt: string;
startedAtMs: number;
finishPromise: Promise<Trace>;
resolveFinish: (trace: Trace) => void;
traceVersion: 2 | 3;
supportsTabSwitchSteps: boolean;
finishPromise: Promise<RecordedTrace>;
resolveFinish: (trace: RecordedTrace) => void;
rejectFinish: (err: Error) => void;
settled: boolean;
finishing: boolean;
currentUrl?: string;
pendingNavigation: boolean;
pendingNavigationDeadline?: number;
finishAttempt: Promise<RecordedTrace | null> | null;
observation: RecordingObservationRuntime | null;
stoppedBy: StopReason;
/** Navigation callbacks tracked from event receipt through action enqueue. */
navigationCallbacks: Set<Promise<void>>;
/** Synchronous intake gate closed only after finish drains to stability. */
acceptingNavigation: boolean;
/**
* Serializes step appends with navigation observation so a click is always
* in `steps` before a same-turn `webNavigation` tries to annotate it.
*/
actionQueue: Promise<void>;
/** Last accepted sequence for each content-script document producer. */
lastStepSequenceByProducer: Map<string, number>;
}
function enqueueRecordingAction(
recording: ActiveRecording,
task: () => Promise<void>,
): Promise<void> {
const queued = recording.actionQueue.then(task, task);
recording.actionQueue = queued.catch(() => {});
return queued;
}
function isRecordingFinishing(recording: ActiveRecording): boolean {
return recording.settled || recording.finishAttempt !== null;
}
const recordings = new Map<string, ActiveRecording>();
@@ -78,6 +118,9 @@ function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
/** Recording producer version mirrored into trace.recorder.bsk. */
export const BSK_TRACE_VERSION = EXTENSION_VERSION;
/** Injectable http(s) landing page when `tool.record_start` omits `url`. */
export const RECORD_DEFAULT_START_URL = "https://example.com/";
@@ -137,9 +180,11 @@ async function sendRecordStartWithAck(
tabId: number,
msg: RecordStartMessage,
sendToTab: RecordDeps["sendToTab"],
cancelled?: () => boolean,
): Promise<void> {
let lastError: unknown;
for (let attempt = 0; attempt < RECORD_START_RETRIES; attempt += 1) {
if (cancelled?.()) throw new Error("record start cancelled");
try {
const response = await sendToTab(tabId, msg);
if (isRecordStartAck(response)) return;
@@ -147,6 +192,7 @@ async function sendRecordStartWithAck(
} catch (err) {
lastError = err;
}
if (cancelled?.()) throw new Error("record start cancelled");
if (attempt + 1 < RECORD_START_RETRIES) {
await sleep(RECORD_START_RETRY_DELAY_MS);
}
@@ -154,19 +200,83 @@ async function sendRecordStartWithAck(
throw lastError ?? new Error("failed to start recording in content script");
}
function buildTrace(recording: ActiveRecording): Trace {
const { pages, steps } = reduceTraceSteps(recording.steps, recording.startUrl);
const startUrl = resolveTraceStartUrl(recording.steps, recording.startUrl, pages);
function negotiatedTraceVersion(params: RecordStartParams): 2 | 3 | RpcError {
if (params.trace_version === undefined) return 2;
if (params.trace_version === TRACE_VERSION_V3) return 3;
return {
...(recording.purpose ? { purpose: recording.purpose } : {}),
recorded_at: new Date().toISOString(),
started_at: recording.startedAt,
entry: { start_url: startUrl },
pages,
steps,
code: "invalid_params",
message: `unsupported trace_version ${params.trace_version}; supported values are omitted (v2) or ${TRACE_VERSION_V3} (v3)`,
};
}
function buildTrace(recording: ActiveRecording): RecordedTrace {
if (recording.traceVersion === 3 && recording.observation) {
return recording.observation.buildTrace({
drafts: recording.steps,
startedAt: recording.startedAt,
purpose: recording.purpose,
startUrl: recording.startUrl,
stoppedBy: recording.stoppedBy,
bskVersion: BSK_TRACE_VERSION,
includeTabSwitches: recording.supportsTabSwitchSteps,
});
}
if (recording.traceVersion === 3) {
throw new Error("trace v3 observation runtime is unavailable");
}
return buildTraceV2({
steps: recording.steps,
startedAt: recording.startedAt,
...(recording.startUrl ? { startUrl: recording.startUrl } : {}),
...(recording.purpose ? { purpose: recording.purpose } : {}),
});
}
function stepBufferFor(
recording: ActiveRecording,
tabId: number,
fallbackUrl?: string,
): RecordingStepBuffer {
return {
steps: recording.steps,
navigation: recording.tabs.navigation(tabId, fallbackUrl),
};
}
async function activateRecordingTab(
recording: ActiveRecording,
targetTabId: number,
): Promise<void> {
const previousTabId = recording.tabs.currentTabId;
if (targetTabId === previousTabId) return;
const transition = recording.observation
? await recording.observation.captureTabTransition(previousTabId, targetTabId)
: {};
const { targetUrl, ...stateLinks } = transition;
const draft: Extract<RecordingDraftStep, { op: "switch_tab" }> = {
op: "switch_tab",
...stateLinks,
};
recording.steps.push(draft);
recording.observation?.bindTabTransition(draft, recording.steps.length);
recording.tabs.commit(targetTabId, targetUrl);
}
async function processRecordedStep(
recording: ActiveRecording,
draftIndex: number,
tabId: number,
producerId?: string,
): Promise<void> {
if (!recording.observation) return;
try {
await recording.observation.processDraft(tabId, recording.steps, draftIndex, producerId);
} catch (err) {
console.warn(`[bsk record] observation failed for step ${draftIndex + 1}`, err);
}
}
export interface RecordDeps {
tabsApi: ChromeTabsApi;
sendToTab(
@@ -174,16 +284,25 @@ export interface RecordDeps {
msg: RecordStartMessage | RecordStopMessage | RecordCancelMessage,
): Promise<unknown>;
bypassOverlay?: (tabId: number, enabled: boolean) => Promise<void>;
frameCoordinator?: Pick<
RecordFrameCoordinator,
"begin" | "armTab" | "sourceFor" | "stop" | "cancel"
>;
cdp?: CdpRunner;
signal?: AbortSignal;
}
export type RecordRuntimeDeps = Omit<RecordDeps, "frameCoordinator" | "signal"> & {
frameCoordinator: NonNullable<RecordDeps["frameCoordinator"]>;
};
let defaultDeps: RecordDeps | null = null;
function getDefaultDeps(): RecordDeps {
if (!defaultDeps) {
defaultDeps = {
tabsApi: chromeTabsApi,
sendToTab: (tabId, msg) => chrome.tabs.sendMessage(tabId, msg),
frameCoordinator: recordFrameCoordinator,
};
}
return defaultDeps;
@@ -241,16 +360,58 @@ export function releaseBrowserObservationListenersIfIdle(): void {
detachBrowserObservation = null;
}
export function attachRecordStepListener(): () => void {
export function attachRecordStepListener(deps: RecordDeps = getDefaultDeps()): () => void {
const listener = (
message: unknown,
_sender: chrome.runtime.MessageSender,
_sendResponse: () => void,
sender: chrome.runtime.MessageSender,
sendResponse: (response: RecordStepAck) => void,
) => {
if (!isRecordStepMessage(message)) return false;
for (const recording of recordings.values()) {
if (recording.requestId !== message.requestId) continue;
appendRecordedPayload(recording, message.step);
const source: RecordingCaptureScope | null | undefined = deps.frameCoordinator
? deps.frameCoordinator.sourceFor(message.requestId, message.producerId, sender)
: undefined;
if (deps.frameCoordinator && !source) return false;
const sourceTabId = source?.tabId ?? sender.tab?.id ?? recording.tabs.currentTabId;
const sourceWasActive = sender.tab?.active ?? sourceTabId === recording.tabs.activeTabId;
const producerKey = source
? `${source.tabId}:${source.documentId}:${source.producerId}`
: `${sourceTabId}:${message.producerId}`;
const expectedSequence = (recording.lastStepSequenceByProducer.get(producerKey) ?? 0) + 1;
if (message.sequence < expectedSequence) {
sendResponse({ ok: true, sequence: message.sequence });
return false;
}
if (message.sequence > expectedSequence) {
sendResponse({
ok: false,
expectedSequence,
error: `out-of-order recorded step ${message.sequence}`,
});
return false;
}
recording.lastStepSequenceByProducer.set(producerKey, message.sequence);
enqueueRecordingAction(recording, async () => {
if (sourceTabId !== recording.tabs.currentTabId) {
if (!sourceWasActive) return;
if (sourceTabId !== recording.tabs.activeTabId)
recording.tabs.noteActivation(sourceTabId);
await activateRecordingTab(recording, sourceTabId);
}
await recording.observation?.flushRedirects(sourceTabId);
const targetHint = message.step.geometry ? { geometry: message.step.geometry } : undefined;
const draftIndex = appendRecordedPayload(
stepBufferFor(recording, sourceTabId, message.step.page_url),
message.step,
targetHint,
);
if (draftIndex !== null) {
await processRecordedStep(recording, draftIndex, sourceTabId, source?.producerId);
}
});
sendResponse({ ok: true, sequence: message.sequence });
return false;
}
return false;
@@ -277,7 +438,7 @@ export function attachRecordFinishListener(deps: RecordDeps = getDefaultDeps()):
function findRecordingByTabId(tabId: number): ActiveRecording | null {
for (const recording of recordings.values()) {
if (recording.tabId === tabId && !recording.settled) return recording;
if (recording.tabs.currentTabId === tabId && !recording.settled) return recording;
}
return null;
}
@@ -314,7 +475,7 @@ async function clearRearmTimersForRecording(
recording: ActiveRecording,
deps: RecordDeps,
): Promise<void> {
clearRearmTimer(recording.tabId);
clearRearmTimer(recording.tabs.currentTabId);
try {
const tabs = await deps.tabsApi.query({ windowId: recording.agentWindowId });
for (const tab of tabs) {
@@ -329,13 +490,16 @@ async function stopRecordingOnAllAgentTabs(
recording: ActiveRecording,
deps: RecordDeps,
): Promise<void> {
if (deps.frameCoordinator && !(await deps.frameCoordinator.stop(recording.requestId))) {
throw new Error("failed to flush one or more recording documents");
}
const stopMsg: RecordStopMessage = { type: RECORD_STOP, requestId: recording.requestId };
let tabIds = [recording.tabId];
let tabIds = [recording.tabs.currentTabId];
try {
const tabs = await deps.tabsApi.query({ windowId: recording.agentWindowId });
tabIds = [
...new Set([
recording.tabId,
recording.tabs.currentTabId,
...tabs.flatMap((tab) => (typeof tab.id === "number" ? [tab.id] : [])),
]),
];
@@ -346,11 +510,11 @@ async function stopRecordingOnAllAgentTabs(
for (const tabId of tabIds) {
try {
const response = await deps.sendToTab(tabId, stopMsg);
if (tabId === recording.tabId && !isRecordStartAck(response)) {
if (tabId === recording.tabs.currentTabId && !isRecordStartAck(response)) {
throw new Error("content script did not confirm recorded steps");
}
} catch {
if (tabId === recording.tabId) {
if (tabId === recording.tabs.currentTabId) {
throw new Error("failed to flush recorded steps");
}
}
@@ -368,22 +532,39 @@ async function rearmRecording(
recording: ActiveRecording,
targetTabId: number,
deps: RecordDeps,
activation?: TabActivation,
): Promise<boolean> {
// Do NOT toggle automation-bypass here: each retry used to increment the
// content-script counter, and a single stop decrement left the ControlOverlay
// stuck with pointer-events:none (page usable, Interrupt dead). RecordOverlay
// already hides the control chrome while activeRecord is set.
const isFinishing = () => isRecordingFinishing(recording);
for (let attempt = 0; attempt < RECORD_REARM_MAX_ATTEMPTS; attempt += 1) {
if (isFinishing()) return false;
const startMsg: RecordStartMessage = {
type: RECORD_START,
requestId: recording.requestId,
startedAtMs: recording.startedAtMs,
};
try {
await sendRecordStartWithAck(targetTabId, startMsg, deps.sendToTab);
recording.tabId = targetTabId;
if (deps.frameCoordinator) {
const frameStarted = await deps.frameCoordinator.armTab(recording.requestId, targetTabId);
if (!frameStarted) throw new Error("recording document did not start");
}
await sendRecordStartWithAck(targetTabId, startMsg, deps.sendToTab, isFinishing);
if (isFinishing()) return false;
if (activation) {
if (!recording.tabs.isLatest(activation)) return true;
await enqueueRecordingAction(recording, async () => {
if (isFinishing() || !recording.tabs.isLatest(activation)) {
return;
}
await activateRecordingTab(recording, targetTabId);
});
}
return true;
} catch {
if (isFinishing()) return false;
if (attempt + 1 < RECORD_REARM_MAX_ATTEMPTS) {
await sleep(RECORD_REARM_RETRY_DELAY_MS);
}
@@ -392,7 +573,7 @@ async function rearmRecording(
return false;
}
function scheduleRearmForTab(tabId: number, deps: RecordDeps): void {
function scheduleRearmForTab(tabId: number, deps: RecordDeps, activation?: TabActivation): void {
const existing = rearmTimers.get(tabId);
if (existing) clearTimeout(existing);
rearmTimers.set(
@@ -401,7 +582,7 @@ function scheduleRearmForTab(tabId: number, deps: RecordDeps): void {
rearmTimers.delete(tabId);
void (async () => {
const current = await findRecordingForTab(tabId, deps);
if (current) await rearmRecording(current, tabId, deps);
if (current) await rearmRecording(current, tabId, deps, activation);
})();
}, RECORD_REARM_DEBOUNCE_MS),
);
@@ -413,14 +594,20 @@ export function attachRecordTabListener(deps: RecordDeps = getDefaultDeps()): ()
const windowId = tab.windowId;
if (tabId === undefined || windowId === undefined) return;
for (const recording of recordings.values()) {
if (recording.settled || recording.agentWindowId !== windowId) continue;
if (isRecordingFinishing(recording) || recording.agentWindowId !== windowId) continue;
scheduleRearmForTab(tabId, deps);
return;
}
};
const onActivated = (activeInfo: chrome.tabs.TabActiveInfo) => {
scheduleRearmForTab(activeInfo.tabId, deps);
for (const recording of recordings.values()) {
if (isRecordingFinishing(recording) || recording.agentWindowId !== activeInfo.windowId)
continue;
const activation = recording.tabs.noteActivation(activeInfo.tabId);
scheduleRearmForTab(activeInfo.tabId, deps, activation);
return;
}
};
chrome.tabs.onCreated.addListener(onCreated);
@@ -432,13 +619,47 @@ export function attachRecordTabListener(deps: RecordDeps = getDefaultDeps()): ()
}
export function attachRecordNavigationListener(deps: RecordDeps = getDefaultDeps()): () => void {
const observeMainFrame = (tabId: number, url?: string, causedByAction?: boolean) => {
void (async () => {
const recording = await findRecordingForTab(tabId, deps);
if (recording && url) {
observeRecordedNavigation(recording, url, causedByAction);
}
})();
const observeMainFrame = (
tabId: number,
url?: string,
causedByAction?: boolean,
transitionType?: string,
transitionQualifiers?: string[],
) => {
if (!url) return;
const candidates = [...recordings.values()].filter(
(recording) =>
!recording.settled && recording.acceptingNavigation && recording.tabs.activeTabId === tabId,
);
for (const recording of candidates) {
const queued = enqueueRecordingAction(recording, async () => {
if (tabId !== recording.tabs.currentTabId) {
await activateRecordingTab(recording, tabId);
}
const result = observeRecordedNavigation(
stepBufferFor(recording, tabId, url),
url,
causedByAction,
transitionType,
transitionQualifiers,
);
if (result.kind === "coalesce_redirect") {
recording.observation?.scheduleRedirect(tabId, recording.steps, result.url);
return;
}
if (result.kind === "noop") return;
recording.observation?.clearRedirect(tabId);
if (result.kind === "appended") {
await processRecordedStep(recording, result.index, tabId);
} else {
recording.observation?.scheduleSettle(tabId, recording.steps, result.index);
}
});
const tracked = queued.catch(() => {});
recording.navigationCallbacks.add(tracked);
void tracked.finally(() => recording.navigationCallbacks.delete(tracked));
}
};
const onMainFrameComplete = (tabId: number, url?: string) => {
void (async () => {
@@ -458,18 +679,13 @@ export function attachRecordNavigationListener(deps: RecordDeps = getDefaultDeps
details: chrome.webNavigation.WebNavigationTransitionCallbackDetails,
) => {
if (details.frameId !== 0) return;
const causedByAction =
details.transitionType === "link" || details.transitionType === "form_submit"
? true
: details.transitionType === "typed" ||
details.transitionType === "auto_bookmark" ||
details.transitionType === "generated" ||
details.transitionType === "keyword" ||
details.transitionType === "keyword_generated" ||
details.transitionType === "reload"
? false
: undefined;
observeMainFrame(details.tabId, details.url, causedByAction);
observeMainFrame(
details.tabId,
details.url,
undefined,
details.transitionType,
details.transitionQualifiers,
);
};
chrome.webNavigation.onCompleted.addListener(completedListener);
chrome.webNavigation.onCommitted?.addListener(committedListener);
@@ -487,6 +703,26 @@ export function attachRecordNavigationListener(deps: RecordDeps = getDefaultDeps
return () => chrome.tabs.onUpdated.removeListener(listener);
}
const MAX_FINISH_DRAIN_ROUNDS = 10;
async function drainRecordingToStability(recording: ActiveRecording): Promise<boolean> {
for (let round = 0; round < MAX_FINISH_DRAIN_ROUNDS; round += 1) {
await Promise.all([...recording.navigationCallbacks]);
const actionTail = recording.actionQueue;
await actionTail;
await recording.observation?.flush();
if (recording.navigationCallbacks.size === 0 && recording.actionQueue === actionTail) {
// No event or promise continuation can interleave with this synchronous
// check-and-close, so work accepted before the cutoff is fully drained.
recording.acceptingNavigation = false;
return true;
}
}
console.warn("[bsk record] navigation/action queues did not stabilize at stop");
return false;
}
export function attachRecordQueryListener(deps: RecordDeps = getDefaultDeps()): () => void {
const listener = (
message: unknown,
@@ -505,7 +741,8 @@ export function attachRecordQueryListener(deps: RecordDeps = getDefaultDeps()):
sendResponse({ active: false });
return;
}
await rearmRecording(recording, tabId, deps);
const activation = sender.tab?.active ? recording.tabs.noteActivation(tabId) : undefined;
await rearmRecording(recording, tabId, deps, activation);
sendResponse({
active: true,
requestId: recording.requestId,
@@ -527,23 +764,50 @@ async function finishRecordingByRequest(
if (recording.requestId !== requestId || recording.settled) continue;
const match = await findRecordingForTab(tabId, deps);
if (match !== recording) continue;
await finishRecording(sessionId, deps);
await finishRecording(sessionId, deps, "user_finish");
return;
}
}
async function finishRecording(sessionId: string, deps: RecordDeps): Promise<Trace | null> {
async function finishRecording(
sessionId: string,
deps: RecordDeps,
stoppedBy: StopReason,
): Promise<RecordedTrace | null> {
const recording = recordings.get(sessionId);
if (!recording || recording.settled || recording.finishing) return null;
recording.finishing = true;
if (!recording || recording.settled) return null;
if (recording.finishAttempt) return recording.finishAttempt;
recording.stoppedBy = stoppedBy;
const attempt = finishRecordingAttempt(sessionId, recording, deps);
recording.finishAttempt = attempt;
try {
return await attempt;
} finally {
if (!recording.settled) {
recording.finishAttempt = null;
}
}
}
async function finishRecordingAttempt(
sessionId: string,
recording: ActiveRecording,
deps: RecordDeps,
): Promise<RecordedTrace | null> {
await clearRearmTimersForRecording(recording, deps);
try {
// Disposing content capture may commit a final dirty fill. Flush content
// first so all resulting RECORD_STEP messages enter actionQueue before it
// and the observation queues are drained.
await stopRecordingOnAllAgentTabs(recording, deps);
} catch {
recording.finishing = false;
return null;
}
if (!(await drainRecordingToStability(recording))) {
return null;
}
await recording.observation?.settleTrailing(recording.tabs.currentTabId, recording.steps);
recording.settled = true;
recordings.delete(sessionId);
@@ -558,6 +822,15 @@ export async function handleRecordStart(
params: RecordStartParams,
deps: RecordDeps = getDefaultDeps(),
): Promise<RecordStartResult | RpcError> {
const traceVersionOrErr = negotiatedTraceVersion(params);
if (typeof traceVersionOrErr !== "number") return traceVersionOrErr;
if (traceVersionOrErr === 3 && !deps.cdp) {
return {
code: "protocol_error",
message: "trace v3 recording requires an active CDP connection",
};
}
const ctxOrErr = lookupSession(manager, params, "record_start");
if (isRpcError(ctxOrErr)) return ctxOrErr;
const ctx = ctxOrErr;
@@ -574,37 +847,64 @@ export async function handleRecordStart(
// on the destination page can RECORD_QUERY → rearm → show RecordOverlay
// instead of flashing ControlOverlay ("Agent 正在控制").
const requestId = makeRequestId(target.tabId);
let resolveFinish!: (trace: Trace) => void;
let resolveFinish!: (trace: RecordedTrace) => void;
let rejectFinish!: (err: Error) => void;
const finishPromise = new Promise<Trace>((resolve, reject) => {
const finishPromise = new Promise<RecordedTrace>((resolve, reject) => {
resolveFinish = resolve;
rejectFinish = reject;
});
const navigateUrl = params.url ?? RECORD_DEFAULT_START_URL;
const startedAtMs = Date.now();
const maxPageTokens = params.max_page_tokens;
const redactValues = params.redact_values ?? false;
recordings.set(params.session_id, {
requestId,
tabId: target.tabId,
tabs: new RecordingTabCoordinator(target.tabId, navigateUrl),
agentWindowId: ctx.agentWindowId,
startUrl: navigateUrl,
...(params.purpose ? { purpose: params.purpose } : {}),
steps: [],
startedAt: new Date(startedAtMs).toISOString(),
startedAtMs,
traceVersion: traceVersionOrErr,
supportsTabSwitchSteps: params.supports_tab_switch_steps === true,
finishPromise,
resolveFinish,
rejectFinish,
settled: false,
finishing: false,
currentUrl: navigateUrl,
pendingNavigation: false,
pendingNavigationDeadline: undefined,
finishAttempt: null,
observation:
traceVersionOrErr === 3 && deps.cdp
? new RecordingObservationRuntime({
cdp: deps.cdp,
tabsApi: deps.tabsApi,
maxTokens: maxPageTokens,
redactValues,
})
: null,
stoppedBy: "user_finish",
navigationCallbacks: new Set(),
acceptingNavigation: true,
actionQueue: Promise.resolve(),
lastStepSequenceByProducer: new Map(),
});
deps.frameCoordinator?.begin(requestId, startedAtMs, target.tabId, async (scope) => {
const recording = recordings.get(params.session_id);
if (!recording || recording.requestId !== requestId || recording.settled) return;
try {
await recording.observation?.refreshDocument(scope.tabId, scope.producerId);
} catch {
// The normal action-time capture remains available if a child Document
// is replaced while its readiness refresh is in flight.
}
});
// Observe navigations for the whole recording lifetime; attach before
// optional navigate so the destination load can rearm capture.
ensureBrowserObservationListeners(deps);
const abortPending = async (notifyContent: boolean) => {
recordings.get(params.session_id)?.observation?.cancel();
deps.frameCoordinator?.cancel(requestId);
recordings.delete(params.session_id);
releaseBrowserObservationListenersIfIdle();
if (notifyContent) {
@@ -686,7 +986,7 @@ export async function handleRecordStart(
return cancelledError();
}
active.startUrl = startUrl;
active.currentUrl = startUrl;
active.tabs.navigation(target.tabId, startUrl).currentUrl = startUrl;
if (isContentScriptRestrictedUrl(startUrl)) {
await abortPending(false);
@@ -694,7 +994,7 @@ export async function handleRecordStart(
code: "invalid_params",
message: params.url
? `cannot record on restricted URL (${startUrl}); use an http(s) page`
: `cannot record on restricted URL (${startUrl}); default start page must be injectable http(s)`,
: `cannot record on restricted URL (${startUrl}); default start page https://example.com/ did not load — pass --url with a page you can open`,
};
}
@@ -721,6 +1021,10 @@ export async function handleRecordStart(
const startMsg: RecordStartMessage = { type: RECORD_START, requestId, startedAtMs };
try {
if (deps.frameCoordinator) {
const frameStarted = await deps.frameCoordinator.armTab(requestId, target.tabId);
if (!frameStarted) throw new Error("top recording document did not start");
}
await sendRecordStartWithAck(target.tabId, startMsg, deps.sendToTab);
} catch {
await abortPending(true);
@@ -731,6 +1035,17 @@ export async function handleRecordStart(
};
}
if (active.observation) {
const activeRecording = recordings.get(params.session_id);
if (activeRecording) {
try {
await activeRecording.observation?.captureInitial(target.tabId);
} catch {
// Proceed without initial observation; steps may be dropped by reducer.
}
}
}
{
const cancelled = await abortIfCancelled(true);
if (cancelled) return cancelled;
@@ -755,7 +1070,7 @@ export async function handleRecordStop(
};
}
const trace = await finishRecording(params.session_id, deps);
const trace = await finishRecording(params.session_id, deps, "cli_stop");
if (!trace) {
return {
code: "protocol_error",
@@ -785,9 +1100,9 @@ export async function handleRecordAwait(
return { code: "cancelled", message: "record_await aborted" };
}
const outcome = await new Promise<{ trace: Trace } | { error: RpcError }>((resolve) => {
const outcome = await new Promise<{ trace: RecordedTrace } | { error: RpcError }>((resolve) => {
let settled = false;
const finish = (result: { trace: Trace } | { error: RpcError }) => {
const finish = (result: { trace: RecordedTrace } | { error: RpcError }) => {
if (settled) return;
settled = true;
if (timer) clearTimeout(timer);
@@ -829,6 +1144,8 @@ export function clearRecordingForSession(sessionId: string): void {
}
void clearRearmTimersForRecording(recording, getDefaultDeps());
if (!recording.settled) {
getDefaultDeps().frameCoordinator?.cancel(recording.requestId);
recording.observation?.cancel();
recording.settled = true;
recording.rejectFinish(new Error("recording cleared"));
}
+20
View File
@@ -76,6 +76,7 @@ export interface SessionStopResult {
}
export interface SessionStopDeps {
signal?: AbortSignal;
cdp?: {
detachSession(sessionId: string): Promise<void>;
};
@@ -177,6 +178,9 @@ export async function handleSessionStop(
message: `session ${params.session_id} unknown`,
};
}
if (deps.signal?.aborted) {
return { code: "cancelled", message: "session_stop aborted before teardown" };
}
// Step 1: auto-return borrowed tabs. Iterate over a snapshot of the
// ids so deletions during iteration do not break the Map iterator.
@@ -187,6 +191,7 @@ export async function handleSessionStop(
try {
const tabManagement = {
...(deps.tabManagement ?? {}),
signal: deps.signal,
isAgentWindowId:
deps.tabManagement?.isAgentWindowId ??
((windowId: number) => manager.findByWindowId(windowId) !== null),
@@ -214,6 +219,17 @@ export async function handleSessionStop(
}
}
if (deps.signal?.aborted) {
const nonCancelFailures = returnFailures?.filter((failure) => failure.code !== "cancelled");
if (nonCancelFailures && nonCancelFailures.length > 0) {
return {
...(returnedTabIds.length > 0 ? { returned_tab_ids: returnedTabIds } : {}),
return_failures: nonCancelFailures,
};
}
return { code: "cancelled", message: "session_stop aborted during tab return" };
}
const result: SessionStopResult = {};
if (returnedTabIds.length > 0) result.returned_tab_ids = returnedTabIds;
if (returnFailures && returnFailures.length > 0) {
@@ -233,6 +249,10 @@ export async function handleSessionStop(
// Step 3: detach CDP sessions this session opened (no-op if none).
await deps.cdp?.detachSession(params.session_id);
if (deps.signal?.aborted) {
return { code: "cancelled", message: "session_stop aborted before window close" };
}
// Step 4: close the Agent Window and drop the context.
await manager.stop(params.session_id);
+1
View File
@@ -53,6 +53,7 @@ export type { DialogCursor };
export interface CdpRunner {
send<T = unknown>(tabId: number, method: string, params?: object): Promise<T>;
sendToTarget?<T = unknown>(target: CdpTarget, method: string, params?: object): Promise<T>;
detach?(tabId: number): Promise<void>;
getFrameGraph?(tabId: number): Promise<CdpFrameGraph>;
ensureAttachedToUrl?(tabId: number, expectedUrl: string | undefined): Promise<void>;
trackSessionTab?(sessionId: string, tabId: number): void;
@@ -0,0 +1,42 @@
// Shared time boundary for browser-side file-transfer transactions. Concrete
// upload/download mechanisms own their state machines and effect semantics.
export class BoundedWaitError extends Error {
constructor(
readonly kind: "timeout" | "aborted",
message: string,
) {
super(message);
}
}
export function remainingMs(deadline: number): number {
return Math.max(0, deadline - Date.now());
}
export async function waitBounded<T>(
promise: Promise<T>,
deadline: number,
signal: AbortSignal | undefined,
timeoutMessage: string,
): Promise<T> {
const remaining = remainingMs(deadline);
if (signal?.aborted) throw new BoundedWaitError("aborted", "file transfer aborted");
if (remaining === 0) throw new BoundedWaitError("timeout", timeoutMessage);
let timer: ReturnType<typeof setTimeout> | undefined;
let onAbort: (() => void) | undefined;
const boundary = new Promise<never>((_resolve, reject) => {
timer = setTimeout(() => reject(new BoundedWaitError("timeout", timeoutMessage)), remaining);
if (signal) {
onAbort = () => reject(new BoundedWaitError("aborted", "file transfer aborted"));
signal.addEventListener("abort", onAbort, { once: true });
}
});
try {
return await Promise.race([promise, boundary]);
} finally {
if (timer) clearTimeout(timer);
if (signal && onAbort) signal.removeEventListener("abort", onAbort);
}
}
+85
View File
@@ -0,0 +1,85 @@
// Upload orchestration: validate the session-scoped request, resolve its click
// target, then delegate the browser protocol transaction to the file-input
// transaction module.
import {
type CaptureSuppressSendToTab,
withExtensionOverlayHidden,
} from "@/lib/capture-suppress-bridge";
import type { SessionManager } from "@/session-manager/manager";
import type { RpcError, UploadParams, UploadResult } from "@/transport/types";
import { uploadThroughFileDrop } from "./file-drop-transaction";
import { uploadThroughActivatedFileInput } from "./file-input-transaction";
import { clickResolvedTarget, type InteractionDeps, resolveActionTarget } from "./interaction";
import {
type CdpRunner,
enforceAgentWindow,
isRpcError,
lookupSession,
resolveTargetTab,
} from "./shared";
const DEFAULT_TIMEOUT_MS = 120_000;
export interface UploadDeps extends InteractionDeps {
cdp: CdpRunner;
sendToTab?: CaptureSuppressSendToTab;
}
export async function handleUpload(
manager: SessionManager,
params: UploadParams,
deps: UploadDeps,
): Promise<UploadResult | RpcError> {
const ctx = lookupSession(manager, params, "upload");
if (isRpcError(ctx)) return ctx;
const target = await resolveTargetTab(manager, ctx, params.tab_id, deps.tabsApi);
if (isRpcError(target)) return target;
const denied = enforceAgentWindow(ctx, target, "upload");
if (denied) return denied;
if (
params.files.length === 0 ||
params.files.length > 20 ||
params.files.some((file) => !file.staged_path)
) {
return { code: "invalid_params", message: "upload requires daemon-staged files" };
}
const address = await resolveActionTarget(deps.cdp, ctx, target, params, "upload");
if (isRpcError(address)) return address;
const timeoutMs = params.timeout_ms ?? DEFAULT_TIMEOUT_MS;
const files = params.files.map((file) => file.staged_path as string);
const mode = params.mode ?? "input";
let usedRef = address.usedRef;
let usedSelector = address.usedSelector;
if (mode === "drop") {
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
const transaction = await uploadThroughFileDrop({
cdp: deps.cdp,
actionTarget: address,
files,
timeoutMs,
signal: deps.signal,
withOverlayHidden: (operation) =>
withExtensionOverlayHidden(target.tabId, operation, deps.sendToTab),
});
if (isRpcError(transaction)) return transaction;
} else {
const transaction = await uploadThroughActivatedFileInput({
cdp: deps.cdp,
actionTarget: address,
files,
timeoutMs,
signal: deps.signal,
trigger: () => clickResolvedTarget(ctx, address, {}, deps),
});
if (isRpcError(transaction)) return transaction;
usedRef = transaction.click.used_ref;
usedSelector = transaction.click.used_selector;
}
return {
tab_id: target.tabId,
used_ref: usedRef,
used_selector: usedSelector,
file_names: params.files.map((file) => file.name),
};
}
@@ -1,6 +1,6 @@
import { describe, expect, it, vi } from "vitest";
import { OVERLAY_HOST_MARKER_ATTR, OVERLAY_HOST_NAME } from "../../../lib/overlay-bridge";
import { captureViewModel, collectOverlayExcludedBackendIds } from "../capture";
import { captureViewModel, collectOverlayExcludedBackendIds, probeHoverSurfaces } from "../capture";
// Minimal but format-accurate captureSnapshot reply: a body with one
// fixed full-screen overlay div carrying a password input.
@@ -36,7 +36,7 @@ function fakeSnapshotReply() {
},
layout: {
nodeIndex: [1, 2, 3],
// styles columns follow requested computedStyles order: [position, pointer-events]
// Legacy fixture: omitted style columns fall back to visible defaults.
styles: [
[i("static"), i("auto")],
[i("fixed"), i("auto")],
@@ -251,11 +251,10 @@ describe("captureViewModel", () => {
expect(input?.attrs).toEqual({ type: "password" });
});
it("does not run conditional surface probes by default", async () => {
it("never hovers the page while capturing", async () => {
const cdp = makeCdp(fakeSnapshotReply());
const result = await captureViewModel(cdp, 4);
await captureViewModel(cdp, 4);
expect(result.surfaceProbes).toEqual([]);
expect(cdp.send).not.toHaveBeenCalledWith(4, "Input.dispatchMouseEvent", expect.anything());
});
@@ -319,9 +318,10 @@ describe("captureViewModel", () => {
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
const result = await captureViewModel(cdp, 4, { conditionalSurfaceProbe: true });
const captured = await captureViewModel(cdp, 4);
const surfaceProbes = await probeHoverSurfaces(cdp, 4, captured.nodes);
expect(result.surfaceProbes).toEqual([
expect(surfaceProbes).toEqual([
{
triggerBackendNodeId: 12,
triggerPoint: { x: 956, y: 32 },
@@ -378,9 +378,10 @@ describe("captureViewModel", () => {
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
const result = await captureViewModel(cdp, 4, { conditionalSurfaceProbe: true });
const captured = await captureViewModel(cdp, 4);
const surfaceProbes = await probeHoverSurfaces(cdp, 4, captured.nodes);
expect(result.surfaceProbes).toEqual([
expect(surfaceProbes).toEqual([
expect.objectContaining({
triggerBackendNodeId: 12,
subItems: ["My profile", "Sign out"],
@@ -422,7 +423,8 @@ describe("captureViewModel", () => {
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
await captureViewModel(cdp, 4, { conditionalSurfaceProbe: true });
const captured = await captureViewModel(cdp, 4);
await probeHoverSurfaces(cdp, 4, captured.nodes);
expect(hoverMoves).toBe(1);
});
@@ -682,6 +684,46 @@ describe("captureViewModel", () => {
expect(nodes.find((n) => n.backendNodeId === 11)?.cursor).toBe("auto");
});
it("captures painted visibility as DOM fallback evidence", async () => {
const S = ["html", "body", "button", "static", "auto", "visible", "hidden", "1", "0"];
const i = (value: string) => S.indexOf(value);
const snapshot = {
strings: S,
documents: [
{
nodes: {
parentIndex: [-1, 0, 1, 1, 1],
nodeName: [i("html"), i("body"), i("button"), i("button"), i("button")],
backendNodeId: [10, 11, 12, 13, 14],
attributes: [[], [], [], [], []],
},
layout: {
nodeIndex: [1, 2, 3, 4],
styles: [
[i("static"), i("auto"), i("auto"), i("visible"), i("1")],
[i("static"), i("auto"), i("auto"), i("visible"), i("1")],
[i("static"), i("auto"), i("auto"), i("hidden"), i("1")],
[i("static"), i("auto"), i("auto"), i("visible"), i("0")],
],
bounds: [
[0, 0, 1000, 800],
[10, 10, 100, 30],
[10, 50, 100, 30],
[10, 90, 100, 30],
],
paintOrders: [0, 1, 2, 3],
},
},
],
};
const { nodes } = await captureViewModel(makeCdp(snapshot), 4);
expect(nodes.find((node) => node.backendNodeId === 12)?.rendered).toBe(true);
expect(nodes.find((node) => node.backendNodeId === 13)?.rendered).toBe(false);
expect(nodes.find((node) => node.backendNodeId === 14)?.rendered).toBe(false);
});
it("subtracts scroll offset to make rects viewport-relative", async () => {
const cdp = {
send: vi.fn(async (_t: number, method: string) => {
@@ -132,4 +132,67 @@ describe("captureFrameData", () => {
]),
);
});
it("keeps OOPIF semantics when viewport projection is unavailable", async () => {
const owner = { ...ownerNode(10, 50), rect: null, localRect: null };
const captured: CapturedViewModel = {
nodes: [owner],
viewport: { width: 1000, height: 800 },
iframeNodes: new Map(),
frameNodes: new Map([["main", [owner]]]),
rootFrameId: "main",
excludedBackendNodeIds: new Set(),
};
const cdp: CdpRunner = {
send: vi.fn(async (_tabId, method) => {
if (method === "Accessibility.enable") return {};
if (method === "Accessibility.getFullAXTree") return { nodes: [] };
if (method === "DOM.getBoxModel") throw new Error("owner geometry unavailable");
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1000, clientHeight: 800 } };
}
throw new Error(`unexpected root ${method}`);
}) as CdpRunner["send"],
sendToTarget: vi.fn(async (_target, method) => {
if (method === "Page.getLayoutMetrics") throw new Error("viewport unavailable");
if (method === "DOMSnapshot.enable" || method === "Accessibility.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") return childSnapshot("child", 101);
if (method === "Accessibility.getFullAXTree") {
return {
nodes: [
{
nodeId: "button",
backendDOMNodeId: 101,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Continue" },
},
],
};
}
throw new Error(`unexpected ${method}`);
}) as unknown as NonNullable<CdpRunner["sendToTarget"]>,
getFrameGraph: vi.fn(async () => ({
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "child",
parentFrameId: "main",
ownerBackendNodeId: 10,
target: { tabId: 4, sessionId: "child-session" },
},
],
})),
};
const documents = await captureFrameData(cdp, 4, captured);
const childDocument = documents.find((document) => document.frameId === "child");
expect(childDocument?.axNodes).toEqual([
expect.objectContaining({ backendDOMNodeId: 101, frameId: "child" }),
]);
expect(childDocument?.domNodes.find((node) => node.backendNodeId === 101)).toEqual(
expect.objectContaining({ rect: null, localRect: { x: 10, y: 20, w: 100, h: 40 } }),
);
});
});
@@ -0,0 +1,84 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { ProbeBudget, resetOverlayBypassDepth, withOverlayBypass } from "../hover-perception";
afterEach(() => {
resetOverlayBypassDepth();
});
describe("ProbeBudget", () => {
it("refuses a candidate that cannot finish inside the budget", () => {
const budget = new ProbeBudget(500);
expect(budget.canAfford(400)).toBe(true);
expect(budget.canAfford(600)).toBe(false);
});
it("stops admitting candidates once the remaining budget is smaller than one", async () => {
const budget = new ProbeBudget(60);
await new Promise((resolve) => setTimeout(resolve, 40));
// A top-of-loop `elapsed > total` check would still admit this candidate
// and then overshoot by its full cost.
expect(budget.canAfford(50)).toBe(false);
});
});
describe("withOverlayBypass", () => {
it("toggles the overlay once around nested probe phases", async () => {
const bypass = vi.fn(async () => {});
await withOverlayBypass(bypass, 4, async () => {
await withOverlayBypass(bypass, 4, async () => {
expect(bypass).toHaveBeenCalledTimes(1);
});
expect(bypass).toHaveBeenCalledTimes(1);
});
expect(bypass.mock.calls).toEqual([
[4, true],
[4, false],
]);
});
it("restores the overlay when the probe throws", async () => {
const bypass = vi.fn(async () => {});
await expect(
withOverlayBypass(bypass, 4, async () => {
throw new Error("probe exploded");
}),
).rejects.toThrow("probe exploded");
expect(bypass).toHaveBeenLastCalledWith(4, false);
});
it("reports a failed restore instead of swallowing it", async () => {
const onRestoreFailure = vi.fn();
const bypass = vi.fn(async (_tabId: number, enabled: boolean) => {
if (!enabled) throw new Error("tab is gone");
});
await withOverlayBypass(bypass, 4, async () => undefined, { onRestoreFailure });
expect(onRestoreFailure).toHaveBeenCalledTimes(1);
});
it("does not pin the overlay off for the tab when a restore fails", async () => {
const failing = vi.fn(async (_tabId: number, enabled: boolean) => {
if (!enabled) throw new Error("tab is gone");
});
await withOverlayBypass(failing, 4, async () => undefined, { onRestoreFailure: () => {} });
const bypass = vi.fn(async () => {});
await withOverlayBypass(bypass, 4, async () => undefined);
expect(bypass.mock.calls).toEqual([
[4, true],
[4, false],
]);
});
it("runs the probe untouched when no bypass is wired up", async () => {
await expect(withOverlayBypass(undefined, 4, async () => "done")).resolves.toBe("done");
});
});
@@ -0,0 +1,214 @@
import { describe, expect, it, vi } from "vitest";
import type { CdpTarget } from "@/browser-driver/frame-graph";
import type { CdpRunner } from "../../shared";
import type { CapturedNode } from "../capture";
import type { FrameDocument, FrameOwnedAxNode } from "../frame-document";
import { probeTooltipNames } from "../name-enrichment";
import { buildSemanticGraph, resolveSemanticGraph } from "../semantic-graph";
import { stableIdentifierName } from "../semantic-graph/name-evidence";
import { frameBackendKey } from "../semantic-graph/types";
function document(
target: CdpTarget,
backendNodeId: number,
accessibleName?: string,
frameId = "child",
x = 10,
): FrameDocument<FrameOwnedAxNode> {
const node: CapturedNode = {
backendNodeId,
parentBackendNodeId: null,
frameId,
tag: "button",
attrs: {
id: "toolInsertRecord",
"data-test-id": "toolInsertRecord",
...(accessibleName ? { "aria-label": accessibleName } : {}),
},
rect: { x, y: 20, w: 40, h: 30 },
localRect: { x: 2, y: 3, w: 40, h: 30 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
cursor: "pointer",
};
return {
frameId,
parentFrameId: "main",
contextScopeId: "child",
target,
axNodes: [],
domNodes: [node],
url: "https://example.com/frame",
};
}
function semantics(documents: FrameDocument<FrameOwnedAxNode>[]) {
return resolveSemanticGraph(
buildSemanticGraph({
documents,
viewport: { width: 800, height: 600 },
rootFrameId: documents[0]?.frameId,
}),
{ identifierFallback: false },
);
}
describe("tooltip name enrichment", () => {
it("uses the owning frame target and upgrades an identifier name from a tooltip", async () => {
const target = { tabId: 7, sessionId: "oopif-session" };
let hovered = false;
const cdp: CdpRunner = {
send: vi.fn(async (_tabId, method, params) => {
if (method === "Input.dispatchMouseEvent") {
hovered = (params as { x?: number }).x === 30;
}
return {};
}) as CdpRunner["send"],
sendToTarget: vi.fn(async (_target, method) => {
if (method === "Page.createIsolatedWorld") return { executionContextId: 11 };
if (method === "DOM.resolveNode") return { object: { objectId: "button-501" } };
if (method === "Runtime.callFunctionOn") {
return { result: { value: hovered ? ["插入行"] : [] } };
}
return {};
}) as CdpRunner["sendToTarget"],
};
const documents = [document(target, 501)];
const graph = semantics(documents);
const names = await probeTooltipNames(cdp, 7, documents, graph);
expect(names.get(frameBackendKey("child", 501))).toBe("插入行");
expect(cdp.sendToTarget).toHaveBeenCalledWith(
target,
"Page.createIsolatedWorld",
expect.objectContaining({ frameId: "child" }),
);
const sendsBeforeCache = vi.mocked(cdp.send).mock.calls.length;
const targetSendsBeforeCache = vi.mocked(
cdp.sendToTarget as NonNullable<CdpRunner["sendToTarget"]>,
).mock.calls.length;
const cached = await probeTooltipNames(cdp, 7, documents, graph);
expect(cached.get(frameBackendKey("child", 501))).toBe("插入行");
expect(cdp.send).toHaveBeenCalledTimes(sendsBeforeCache);
expect(cdp.sendToTarget).toHaveBeenCalledTimes(targetSendsBeforeCache);
});
it("does not probe controls that already have a higher-confidence name", async () => {
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
sendToTarget: vi.fn(async () => ({ executionContextId: 12 })) as CdpRunner["sendToTarget"],
};
const documents = [document({ tabId: 8 }, 601, "Add a new record")];
const names = await probeTooltipNames(cdp, 8, documents, semantics(documents));
expect(names.size).toBe(0);
expect(cdp.send).not.toHaveBeenCalled();
expect(cdp.sendToTarget).not.toHaveBeenCalled();
});
it("does not collect supplemental names from form controls", async () => {
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
sendToTarget: vi.fn(async () => ({ executionContextId: 12 })) as CdpRunner["sendToTarget"],
};
const inputDocument = document({ tabId: 8 }, 602);
inputDocument.domNodes[0] = {
...inputDocument.domNodes[0],
tag: "input",
attrs: { type: "email" },
formValue: "user@example.com",
formDefaultValue: "",
formState: "filled",
};
const names = await probeTooltipNames(cdp, 8, [inputDocument], semantics([inputDocument]));
expect(names.size).toBe(0);
expect(cdp.send).not.toHaveBeenCalled();
expect(cdp.sendToTarget).not.toHaveBeenCalled();
});
it("remembers controls that revealed no tooltip so the miss is paid once", async () => {
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
sendToTarget: vi.fn(async (_target, method) => {
if (method === "Page.createIsolatedWorld") return { executionContextId: 31 };
if (method === "DOM.resolveNode") return { object: { objectId: "button-801" } };
if (method === "Runtime.callFunctionOn") return { result: { value: [] } };
return {};
}) as CdpRunner["sendToTarget"],
};
const documents = [
document({ tabId: 11, sessionId: "miss-session" }, 801, undefined, "miss-frame"),
];
const graph = semantics(documents);
const first = await probeTooltipNames(cdp, 11, documents, graph);
expect(first.size).toBe(0);
const hoversAfterFirst = vi
.mocked(cdp.send)
.mock.calls.filter(([, method]) => method === "Input.dispatchMouseEvent").length;
expect(hoversAfterFirst).toBeGreaterThan(0);
const second = await probeTooltipNames(cdp, 11, documents, graph);
expect(second.size).toBe(0);
expect(
vi.mocked(cdp.send).mock.calls.filter(([, method]) => method === "Input.dispatchMouseEvent")
.length,
).toBe(hoversAfterFirst);
});
it("keeps tooltip evidence isolated across sibling frame sessions", async () => {
let hoveredX = -10;
const cdp: CdpRunner = {
send: vi.fn(async (_tabId, method, params) => {
if (method === "Input.dispatchMouseEvent") {
hoveredX = (params as { x: number }).x;
}
return {};
}) as CdpRunner["send"],
sendToTarget: vi.fn(async (target, method, params) => {
if (method === "Page.createIsolatedWorld") {
return { executionContextId: target.sessionId === "left-session" ? 21 : 22 };
}
if (method === "DOM.resolveNode") {
return { object: { objectId: target.sessionId === "left-session" ? "left" : "right" } };
}
if (method === "Runtime.callFunctionOn") {
const objectId = (params as { objectId: string }).objectId;
const value =
objectId === "left" && hoveredX === 30
? ["Left action"]
: objectId === "right" && hoveredX === 130
? ["Right action"]
: [];
return { result: { value } };
}
return {};
}) as CdpRunner["sendToTarget"],
};
const documents = [
document({ tabId: 9, sessionId: "left-session" }, 701, undefined, "left", 10),
document({ tabId: 9, sessionId: "right-session" }, 701, undefined, "right", 110),
];
const names = await probeTooltipNames(cdp, 9, documents, semantics(documents));
expect(names.get(frameBackendKey("left", 701))).toBe("Left action");
expect(names.get(frameBackendKey("right", 701))).toBe("Right action");
});
});
describe("stable identifier evidence", () => {
it("accepts readable action identifiers and rejects framework-generated ids", () => {
expect(stableIdentifierName({ id: "toolInsertRecord" })).toBe("Insert record");
expect(stableIdentifierName({ id: "reactivateAccount" })).toBe("Reactivate account");
expect(stableIdentifierName({ id: "headlessui-menu-button-r1" })).toBeUndefined();
expect(stableIdentifierName({ id: "7f15a3bc91" })).toBeUndefined();
});
});
@@ -0,0 +1,82 @@
import { describe, expect, it } from "vitest";
import {
RECORD_DOCUMENT_ATTRIBUTE,
recordingDocumentMarkerValue,
} from "@/shared/recording-document-identity";
import { projectRecordSafeObservation } from "../record-safe-observation";
describe("projectRecordSafeObservation", () => {
it("projects the opaque recording identity without exposing captured attributes", () => {
const recordingDocumentId = "123e4567-e89b-42d3-a456-426614174000";
const result = projectRecordSafeObservation({
rootFrameId: "child",
frameDocuments: [
{
frameId: "child",
target: { tabId: 4, sessionId: "oopif-session" },
contextScopeId: "scope-1",
axNodes: [],
domNodes: [
{
backendNodeId: 1,
parentBackendNodeId: null,
tag: "html",
attrs: {
[RECORD_DOCUMENT_ATTRIBUTE]: recordingDocumentMarkerValue(recordingDocumentId),
value: "user@example.com",
},
rect: null,
paintOrder: 0,
position: "static",
pointerEvents: "auto",
},
],
},
],
rendered: { text: '@vom 1\nRootWebArea "Example"', refs: [], truncated: false },
});
expect(result.frames).toEqual([
{
frameId: "child",
target: { tabId: 4, sessionId: "oopif-session" },
recordingDocumentId,
},
]);
const dumped = JSON.stringify(result);
expect(dumped).not.toContain("user@example.com");
expect(dumped).not.toContain("attrs");
expect(dumped).not.toContain("domNodes");
expect(dumped).not.toContain("axNodes");
});
it("does not project a page-controlled marker value as recording identity", () => {
const result = projectRecordSafeObservation({
rootFrameId: "root",
frameDocuments: [
{
frameId: "root",
target: { tabId: 4 },
contextScopeId: "scope-1",
axNodes: [],
domNodes: [
{
backendNodeId: 1,
parentBackendNodeId: null,
tag: "html",
attrs: { [RECORD_DOCUMENT_ATTRIBUTE]: "bsk:user@example.com" },
rect: null,
paintOrder: 0,
position: "static",
pointerEvents: "auto",
},
],
},
],
rendered: { text: '@vom 1\nRootWebArea "Example"', refs: [], truncated: false },
});
expect(result.frames).toEqual([{ frameId: "root", target: { tabId: 4 } }]);
expect(JSON.stringify(result)).not.toContain("user@example.com");
});
});
@@ -0,0 +1,782 @@
import { isVomReferenceNode, renderVom } from "@browser-skill/vom";
import { describe, expect, it } from "vitest";
import type { CapturedNode } from "../capture";
import type { FrameDocument } from "../frame-document";
import { buildSemanticVomScene, type SemanticAxNode } from "../semantic-graph";
function dom(
backendNodeId: number,
parentBackendNodeId: number | null,
tag: string,
attrs: Record<string, string> = {},
textContent?: string,
): CapturedNode {
return {
backendNodeId,
parentBackendNodeId,
tag,
attrs,
rect: { x: 0, y: 0, w: 100, h: 30 },
paintOrder: backendNodeId,
position: "static",
pointerEvents: "auto",
...(textContent ? { textContent } : {}),
};
}
function document(
frameId: string,
axNodes: SemanticAxNode[],
domNodes: CapturedNode[],
frame?: Pick<FrameDocument<SemanticAxNode>, "parentFrameId" | "ownerBackendNodeId">,
): FrameDocument<SemanticAxNode> {
return {
frameId,
contextScopeId: frameId,
target: { tabId: 1 },
axNodes,
domNodes,
...frame,
};
}
describe("semantic VOM graph", () => {
it("preserves backend-less AX structure without making it referenceable", () => {
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["toolbar"],
},
{
nodeId: "toolbar",
parentId: "root",
role: { type: "role", value: "toolbar" },
name: { type: "computedString", value: "Formatting" },
childIds: ["bold", "italic"],
},
{
nodeId: "bold",
parentId: "toolbar",
backendDOMNodeId: 2,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Bold" },
},
{
nodeId: "italic",
parentId: "toolbar",
backendDOMNodeId: 3,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Italic" },
},
],
[dom(1, null, "body"), dom(2, 1, "button"), dom(3, 1, "button")],
),
],
});
const toolbar = scene.nodes.find((node) => node.role === "toolbar");
const buttons = scene.nodes.filter((node) => node.role === "button");
expect(toolbar).toEqual(expect.objectContaining({ name: "Formatting", referenceable: false }));
expect(buttons.map((node) => node.parentId)).toEqual([toolbar?.id, toolbar?.id]);
expect(toolbar && isVomReferenceNode(toolbar)).toBe(false);
expect(renderVom(scene).refs.map((ref) => ref.backendNodeId)).toEqual([2, 3]);
});
it("resolves missing AX names through one DOM naming pipeline", () => {
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["labelled", "titled", "text"],
},
{
nodeId: "labelled",
parentId: "root",
backendDOMNodeId: 2,
role: { type: "role", value: "button" },
},
{
nodeId: "titled",
parentId: "root",
backendDOMNodeId: 3,
role: { type: "role", value: "button" },
},
{
nodeId: "text",
parentId: "root",
backendDOMNodeId: 4,
role: { type: "role", value: "button" },
},
],
[
dom(1, null, "body"),
dom(2, 1, "button", { "aria-label": "Add row" }),
dom(3, 1, "button", { title: "Settings" }),
dom(4, 1, "button"),
dom(5, 4, "span", {}, "Save"),
],
),
],
});
expect(scene.nodes.filter((node) => node.role === "button").map((node) => node.name)).toEqual([
"Add row",
"Settings",
"Save",
]);
});
it("aggregates AX text at the semantic text frontier", () => {
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["covered", "inline-only", "repeated", "labelled"],
},
{
nodeId: "covered",
parentId: "root",
backendDOMNodeId: 2,
role: { type: "role", value: "paragraph" },
childIds: ["covered-static"],
},
{
nodeId: "covered-static",
parentId: "covered",
role: { type: "role", value: "StaticText" },
name: { type: "computedString", value: "Covered text" },
childIds: ["covered-inline"],
},
{
nodeId: "covered-inline",
parentId: "covered-static",
role: { type: "role", value: "InlineTextBox" },
name: { type: "computedString", value: "Covered text" },
},
{
nodeId: "inline-only",
parentId: "root",
backendDOMNodeId: 3,
role: { type: "role", value: "paragraph" },
childIds: ["inline-leaf"],
},
{
nodeId: "inline-leaf",
parentId: "inline-only",
role: { type: "role", value: "InlineTextBox" },
name: { type: "computedString", value: "Inline fallback" },
},
{
nodeId: "repeated",
parentId: "root",
backendDOMNodeId: 4,
role: { type: "role", value: "paragraph" },
childIds: ["first-go", "second-go"],
},
{
nodeId: "first-go",
parentId: "repeated",
role: { type: "role", value: "StaticText" },
name: { type: "computedString", value: "Go" },
},
{
nodeId: "second-go",
parentId: "repeated",
role: { type: "role", value: "StaticText" },
name: { type: "computedString", value: "Go" },
},
{
nodeId: "labelled",
parentId: "root",
backendDOMNodeId: 5,
role: { type: "role", value: "button" },
childIds: ["visible-label"],
},
{
nodeId: "visible-label",
parentId: "labelled",
role: { type: "role", value: "StaticText" },
name: { type: "computedString", value: "Visible text" },
},
],
[
dom(1, null, "body"),
dom(2, 1, "p"),
dom(3, 1, "p"),
dom(4, 1, "p"),
dom(5, 1, "button", { "aria-label": "Explicit action" }),
],
),
],
});
expect(scene.nodes.find((node) => node.backendNodeId === 2)?.name).toBe("Covered text");
expect(scene.nodes.find((node) => node.backendNodeId === 3)?.name).toBe("Inline fallback");
expect(scene.nodes.find((node) => node.backendNodeId === 4)?.name).toBe("Go Go");
expect(scene.nodes.find((node) => node.backendNodeId === 5)?.name).toBe("Explicit action");
expect(renderVom(scene).text).not.toContain("Covered text Covered text");
});
it("uses stable identifiers only as a final name fallback", () => {
const base = document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["fallback", "labelled", "container"],
},
{
nodeId: "fallback",
parentId: "root",
backendDOMNodeId: 2,
role: { type: "role", value: "button" },
},
{
nodeId: "labelled",
parentId: "root",
backendDOMNodeId: 3,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Localized name" },
},
{
nodeId: "container",
parentId: "root",
backendDOMNodeId: 4,
role: { type: "role", value: "generic" },
},
],
[
dom(1, null, "body"),
dom(2, 1, "button", { "data-test-id": "toolInsertRecord" }),
dom(3, 1, "button", { id: "toolIgnoredFallback" }),
dom(4, 1, "div", { id: "toolbarContainer" }),
],
);
const staticScene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [base],
});
const enrichedScene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [base],
supplementalNames: new Map([["main\u00002", "插入行"]]),
});
expect(staticScene.nodes.find((node) => node.backendNodeId === 2)?.name).toBe("Insert record");
expect(staticScene.nodes.find((node) => node.backendNodeId === 3)?.name).toBe("Localized name");
expect(staticScene.nodes.find((node) => node.backendNodeId === 4)?.name).toBeUndefined();
expect(enrichedScene.nodes.find((node) => node.backendNodeId === 2)?.name).toBe("插入行");
expect(enrichedScene.nodes.find((node) => node.backendNodeId === 3)?.name).toBe(
"Localized name",
);
});
it("isolates equal backend node ids in sibling frames", () => {
const scene = buildSemanticVomScene({
viewport: { width: 1000, height: 800 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["owner-a", "owner-b"],
},
{
nodeId: "owner-a",
parentId: "root",
backendDOMNodeId: 10,
role: { type: "role", value: "Iframe" },
},
{
nodeId: "owner-b",
parentId: "root",
backendDOMNodeId: 20,
role: { type: "role", value: "Iframe" },
},
],
[dom(1, null, "body"), dom(10, 1, "iframe"), dom(20, 1, "iframe")],
),
document(
"frame-a",
[
{
nodeId: "button-a",
backendDOMNodeId: 5,
role: { type: "role", value: "button" },
childIds: ["text-a"],
},
{
nodeId: "text-a",
parentId: "button-a",
role: { type: "role", value: "StaticText" },
name: { type: "computedString", value: "Accept" },
childIds: ["inline-a"],
},
{
nodeId: "inline-a",
parentId: "text-a",
role: { type: "role", value: "InlineTextBox" },
name: { type: "computedString", value: "Accept" },
},
],
[dom(5, null, "button")],
{ parentFrameId: "main", ownerBackendNodeId: 10 },
),
document(
"frame-b",
[
{
nodeId: "button-b",
backendDOMNodeId: 5,
role: { type: "role", value: "button" },
childIds: ["text-b"],
},
{
nodeId: "text-b",
parentId: "button-b",
role: { type: "role", value: "StaticText" },
name: { type: "computedString", value: "Decline" },
childIds: ["inline-b"],
},
{
nodeId: "inline-b",
parentId: "text-b",
role: { type: "role", value: "InlineTextBox" },
name: { type: "computedString", value: "Decline" },
},
],
[dom(5, null, "button")],
{ parentFrameId: "main", ownerBackendNodeId: 20 },
),
],
});
const frameButtons = scene.nodes.filter((node) => node.backendNodeId === 5);
const owners = new Map(
scene.nodes
.filter((node) => node.backendNodeId === 10 || node.backendNodeId === 20)
.map((node) => [node.backendNodeId, node.id]),
);
expect(frameButtons.map((node) => node.frameId)).toEqual(["frame-a", "frame-b"]);
expect(frameButtons.map((node) => node.name)).toEqual(["Accept", "Decline"]);
expect(frameButtons.map((node) => node.parentId)).toEqual([owners.get(10), owners.get(20)]);
expect(renderVom(scene).refs.map((ref) => [ref.frameId, ref.backendNodeId])).toEqual([
["frame-a", 5],
["frame-b", 5],
]);
});
it("requires render evidence before promoting DOM fallback semantics", () => {
const hiddenOwner = {
...dom(10, 1, "iframe", { hidden: "" }),
rect: null,
localRect: null,
rendered: false,
};
const hiddenButton = {
...dom(2, 1, "button", { "aria-label": "Hidden action" }),
rendered: false,
};
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
},
],
[dom(1, null, "body"), hiddenButton, hiddenOwner],
),
document("hidden-frame", [], [dom(20, null, "button", {}, "Frame action")], {
parentFrameId: "main",
ownerBackendNodeId: 10,
}),
],
});
expect(scene.nodes.some((node) => node.backendNodeId === 2)).toBe(false);
expect(scene.nodes.some((node) => node.backendNodeId === 10)).toBe(false);
expect(scene.nodes.some((node) => node.frameId === "hidden-frame")).toBe(false);
expect(renderVom(scene).refs).toEqual([]);
});
it("retains rendered DOM fallback controls when AX semantics are unavailable", () => {
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[],
[dom(1, null, "body"), { ...dom(2, 1, "button", {}, "Save"), rendered: true }],
),
],
});
expect(renderVom(scene).refs).toEqual([
expect.objectContaining({ backendNodeId: 2, role: "button", name: "Save" }),
]);
});
it("lets explicit DOM structure override ignored AX metadata", () => {
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["toolbar"],
},
{
nodeId: "toolbar",
parentId: "root",
backendDOMNodeId: 2,
ignored: true,
role: { type: "role", value: "generic" },
childIds: ["wrapper"],
},
{
nodeId: "wrapper",
parentId: "toolbar",
backendDOMNodeId: 5,
role: { type: "role", value: "generic" },
childIds: ["bold", "italic"],
},
{
nodeId: "bold",
parentId: "wrapper",
backendDOMNodeId: 3,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Bold" },
},
{
nodeId: "italic",
parentId: "wrapper",
backendDOMNodeId: 4,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Italic" },
},
],
[
dom(1, null, "body"),
dom(2, 1, "div", { role: "toolbar" }),
dom(5, 2, "div"),
dom(3, 5, "button"),
dom(4, 5, "button"),
],
),
],
});
const toolbar = scene.nodes.find((node) => node.backendNodeId === 2);
const buttons = scene.nodes.filter((node) => [3, 4].includes(node.backendNodeId ?? -1));
expect(toolbar).toEqual(expect.objectContaining({ role: "toolbar" }));
expect(toolbar && isVomReferenceNode(toolbar)).toBe(false);
expect(scene.nodes.some((node) => node.role === "group")).toBe(false);
expect(buttons.map((node) => node.parentId)).toEqual([toolbar?.id, toolbar?.id]);
});
it("infers an unnamed group only from independent interaction branches", () => {
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["wrapper"],
},
{
nodeId: "wrapper",
parentId: "root",
backendDOMNodeId: 2,
role: { type: "role", value: "generic" },
childIds: ["first", "second"],
},
{
nodeId: "first",
parentId: "wrapper",
backendDOMNodeId: 3,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "First" },
},
{
nodeId: "second",
parentId: "wrapper",
backendDOMNodeId: 4,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Second" },
},
],
[dom(1, null, "body"), dom(2, 1, "div"), dom(3, 2, "button"), dom(4, 2, "button")],
),
],
});
const group = scene.nodes.find((node) => node.backendNodeId === 2);
const buttons = scene.nodes.filter((node) => [3, 4].includes(node.backendNodeId ?? -1));
expect(group).toEqual(expect.objectContaining({ role: "group" }));
expect(group?.name).toBeUndefined();
expect(group && isVomReferenceNode(group)).toBe(false);
expect(buttons.map((node) => node.parentId)).toEqual([group?.id, group?.id]);
const rendered = renderVom(scene);
const lines = rendered.text.split("\n");
const groupLine = lines.findIndex((line) => line.trim() === "group");
expect(groupLine).toBeGreaterThanOrEqual(0);
for (const ref of rendered.refs) {
expect(ref.line).toBeGreaterThan(groupLine);
expect(lines[ref.line]).toMatch(/^\s+@e\d+ button/);
}
});
it("never promotes a form value into its semantic name through supplemental evidence", () => {
const secret = "user@example.com";
const input = {
...dom(2, 1, "input", { type: "email" }),
formValue: secret,
formDefaultValue: "",
formState: "filled" as const,
};
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
supplementalNames: new Map([["main\u00002", secret]]),
identifierFallback: false,
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["email"],
},
{
nodeId: "email",
parentId: "root",
backendDOMNodeId: 2,
role: { type: "role", value: "textbox" },
value: { value: secret },
},
],
[dom(1, null, "body"), input],
),
],
});
const field = scene.nodes.find((node) => node.backendNodeId === 2);
expect(field?.name).toBeUndefined();
const rendered = renderVom(scene, { redactValues: true });
expect(rendered.text).toContain("•••");
expect(JSON.stringify(rendered)).not.toContain(secret);
});
it("keeps uncertain single-branch wrappers flat", () => {
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["wrapper"],
},
{
nodeId: "wrapper",
parentId: "root",
backendDOMNodeId: 2,
role: { type: "role", value: "generic" },
childIds: ["button"],
},
{
nodeId: "button",
parentId: "wrapper",
backendDOMNodeId: 3,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Only action" },
},
],
[dom(1, null, "body"), dom(2, 1, "div"), dom(3, 2, "button")],
),
],
});
expect(scene.nodes.some((node) => node.role === "group")).toBe(false);
expect(scene.nodes.find((node) => node.backendNodeId === 3)?.parentId).toBe(
scene.nodes.find((node) => node.backendNodeId === 1)?.id,
);
});
it("does not infer across mixed content boundaries", () => {
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["wrapper"],
},
{
nodeId: "wrapper",
parentId: "root",
backendDOMNodeId: 2,
role: { type: "role", value: "generic" },
childIds: ["first", "second", "paragraph"],
},
{
nodeId: "first",
parentId: "wrapper",
backendDOMNodeId: 3,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "First" },
},
{
nodeId: "second",
parentId: "wrapper",
backendDOMNodeId: 4,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Second" },
},
{
nodeId: "paragraph",
parentId: "wrapper",
backendDOMNodeId: 5,
role: { type: "role", value: "paragraph" },
},
],
[
dom(1, null, "body"),
dom(2, 1, "div"),
dom(3, 2, "button"),
dom(4, 2, "button"),
dom(5, 2, "p", {}, "Description"),
],
),
],
});
expect(scene.nodes.some((node) => node.role === "group")).toBe(false);
});
it("keeps only the innermost reliable inferred group", () => {
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["outer"],
},
{
nodeId: "outer",
parentId: "root",
backendDOMNodeId: 2,
role: { type: "role", value: "generic" },
childIds: ["inner", "third"],
},
{
nodeId: "inner",
parentId: "outer",
backendDOMNodeId: 3,
role: { type: "role", value: "generic" },
childIds: ["first", "second"],
},
{
nodeId: "first",
parentId: "inner",
backendDOMNodeId: 4,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "First" },
},
{
nodeId: "second",
parentId: "inner",
backendDOMNodeId: 5,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Second" },
},
{
nodeId: "third",
parentId: "outer",
backendDOMNodeId: 6,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Third" },
},
],
[
dom(1, null, "body"),
dom(2, 1, "div"),
dom(3, 2, "div"),
dom(4, 3, "button"),
dom(5, 3, "button"),
dom(6, 2, "button"),
],
),
],
});
const groups = scene.nodes.filter((node) => node.role === "group");
expect(groups.map((node) => node.backendNodeId)).toEqual([3]);
expect(scene.nodes.find((node) => node.backendNodeId === 6)?.parentId).not.toBe(groups[0]?.id);
});
});
+107 -98
View File
@@ -1,16 +1,17 @@
// CDP capture adapter: DOMSnapshot.captureSnapshot + Page.getLayoutMetrics
// → CapturedNode[] + Viewport. This is the ONLY VOM module that touches
// raw CDP. The captureSnapshot reply is columnar (parallel arrays + a
// shared string table); we request exactly three computed styles so the
// `styles` columns are [position, pointer-events, cursor] in that order.
// shared string table); requested computed styles are decoded through
// STYLE_COL so capture and visibility policy share one explicit contract.
import type { Rect, Viewport } from "@browser-skill/vom";
import { evaluateHoverTrigger } from "@/lib/hover-trigger-policy";
import { isOverlayHostNode, OVERLAY_HOST_SELECTOR } from "../../lib/overlay-bridge";
import { childFrameProjection, type GeometryProjection, projectRectToViewport } from "../geometry";
import type { CdpRunner } from "../shared";
import { clearHover, ProbeBudget, waitForHover } from "./hover-perception";
const REQUESTED_STYLES = ["position", "pointer-events", "cursor"] as const;
const REQUESTED_STYLES = ["position", "pointer-events", "cursor", "visibility", "opacity"] as const;
const STYLE_COL = Object.fromEntries(
REQUESTED_STYLES.map((name, index) => [name, index]),
) as Record<(typeof REQUESTED_STYLES)[number], number>;
@@ -38,6 +39,12 @@ export interface CapturedNode {
* `textContent`: the live parser always sets it, hand-built fixtures may not.
*/
cursor?: string;
/**
* Whether the live DOM snapshot provides a painted, non-hidden box for this
* node. Semantic resolution uses this only for DOM fallback nodes; AX-backed
* nodes remain authoritative even when they are outside the viewport.
*/
rendered?: boolean;
textContent?: string;
formState?: "empty" | "filled" | "default";
formValue?: string;
@@ -64,14 +71,11 @@ export interface CapturedViewModel {
/** Explicit DOMSnapshot frame ancestry; never inferred from backend node ids. */
frameParentIds?: Map<string, string>;
rootFrameId?: string;
surfaceProbes?: CapturedSurfaceProbe[];
/** Backend node ids belonging to the agent overlay host + its shadow subtree. */
excludedBackendNodeIds: Set<number>;
}
export interface CaptureViewModelOptions {
conditionalSurfaceProbe?: boolean;
hoverProbeBypassOverlay?: (tabId: number, enabled: boolean) => Promise<void>;
signal?: AbortSignal;
}
@@ -361,7 +365,16 @@ function collectBackendIdsFromDomNode(node: CdpDomNode | undefined, out: Set<num
}
}
const MAX_HOVER_PROBE_MS = 2_000;
/**
* Ceiling for the whole hover-surface phase.
*
* The previous 2000 was only a floor: the loop checked elapsed time at the top,
* so a candidate could start with 1ms left and still run its two settle
* windows, pushing real cost to ~2.6s. This is that true ceiling, now actually
* enforced by an up-front affordability check, so the same number of candidates
* get probed under a bound that no longer lies.
*/
const MAX_HOVER_PROBE_MS = 2_600;
const MAX_HOVER_TRIGGERS = 6;
const MAX_HOVER_SURFACES = 3;
const HOVER_SETTLE_MS = 300;
@@ -460,36 +473,6 @@ function hoverStateExpression(): string {
})()`;
}
async function wait(ms: number, signal?: AbortSignal): Promise<void> {
throwIfAborted(signal);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => {
signal?.removeEventListener("abort", onAbort);
resolve();
}, ms);
const onAbort = () => {
clearTimeout(timer);
signal?.removeEventListener("abort", onAbort);
reject(captureAbortError());
};
signal?.addEventListener("abort", onAbort, { once: true });
});
}
async function clearHover(cdp: CdpRunner, tabId: number): Promise<void> {
await cdp
.send(tabId, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: -10,
y: -10,
})
.catch(() =>
cdp.send(tabId, "Input.dispatchMouseEvent", { type: "mouseMoved", x: 0, y: 0 }).catch(() => {
// best effort
}),
);
}
function capturedText(node: CapturedNode): string | undefined {
const value =
node.attrs["aria-label"] ?? node.attrs.title ?? node.attrs.alt ?? node.textContent ?? "";
@@ -637,13 +620,35 @@ function confidenceForHover(
return "low";
}
async function probeHoverSurfaces(
/**
* One candidate costs two settle windows (baseline + post-hover) plus a few
* CDP round trips. Used to decide whether the next candidate still fits the
* budget before paying for it.
*/
const HOVER_CANDIDATE_COST_MS = HOVER_SETTLE_MS * 2;
export interface HoverSurfaceProbeOptions {
signal?: AbortSignal;
}
/**
* Hovers a bounded set of likely menu triggers and reports the sub-items each
* one reveals.
*
* Runs after DOM *and* accessibility capture. Hovering can open menus and
* change layout, so probing between the two captures would leave the DOM half
* of an observation describing the page before the change and the AX half
* describing it after.
*
* The caller owns the overlay bypass span (see `withOverlayBypass`).
*/
export async function probeHoverSurfaces(
cdp: CdpRunner,
tabId: number,
nodes: CapturedNode[],
options: CaptureViewModelOptions,
options: HoverSurfaceProbeOptions = {},
): Promise<CapturedSurfaceProbe[]> {
const started = Date.now();
const budget = new ProbeBudget(MAX_HOVER_PROBE_MS);
try {
throwIfAborted(options.signal);
const cssScan = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
@@ -658,60 +663,54 @@ async function probeHoverSurfaces(
const results: CapturedSurfaceProbe[] = [];
const seen = new Set<number>();
throwIfAborted(options.signal);
await options.hoverProbeBypassOverlay?.(tabId, true).catch(() => undefined);
try {
for (const candidate of candidates.slice(0, MAX_HOVER_TRIGGERS)) {
throwIfAborted(options.signal);
for (const candidate of candidates.slice(0, MAX_HOVER_TRIGGERS)) {
if (!budget.canAfford(HOVER_CANDIDATE_COST_MS)) break;
if (results.length >= MAX_HOVER_SURFACES) break;
if (seen.has(candidate.backendNodeId)) continue;
try {
await clearHover(cdp, tabId);
throwIfAborted(options.signal);
if (Date.now() - started > MAX_HOVER_PROBE_MS) break;
if (results.length >= MAX_HOVER_SURFACES) break;
if (seen.has(candidate.backendNodeId)) continue;
try {
await clearHover(cdp, tabId);
throwIfAborted(options.signal);
await wait(HOVER_SETTLE_MS, options.signal);
const baselineReply = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: hoverStateExpression(),
returnByValue: true,
});
throwIfAborted(options.signal);
const baselineItems = runtimeValue<HoverRuntimeItem[]>(baselineReply) ?? [];
await waitForHover(HOVER_SETTLE_MS, options.signal);
const baselineReply = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: hoverStateExpression(),
returnByValue: true,
});
throwIfAborted(options.signal);
const baselineItems = runtimeValue<HoverRuntimeItem[]>(baselineReply) ?? [];
throwIfAborted(options.signal);
await cdp.send(tabId, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: candidate.x,
y: candidate.y,
});
throwIfAborted(options.signal);
await wait(HOVER_SETTLE_MS, options.signal);
const collected = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: hoverStateExpression(),
returnByValue: true,
});
throwIfAborted(options.signal);
const subItems = diffHoverItems(
baselineItems,
runtimeValue<HoverRuntimeItem[]>(collected) ?? [],
);
if (subItems.length === 0) continue;
seen.add(candidate.backendNodeId);
results.push({
triggerBackendNodeId: candidate.backendNodeId,
triggerPoint: { x: candidate.x, y: candidate.y },
triggerAction: "hover",
subItems,
confidence: confidenceForHover(candidate, subItems),
});
} catch (error) {
if (isAbortError(error)) throw error;
continue;
} finally {
await clearHover(cdp, tabId);
}
throwIfAborted(options.signal);
await cdp.send(tabId, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: candidate.x,
y: candidate.y,
});
throwIfAborted(options.signal);
await waitForHover(HOVER_SETTLE_MS, options.signal);
const collected = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: hoverStateExpression(),
returnByValue: true,
});
throwIfAborted(options.signal);
const subItems = diffHoverItems(
baselineItems,
runtimeValue<HoverRuntimeItem[]>(collected) ?? [],
);
if (subItems.length === 0) continue;
seen.add(candidate.backendNodeId);
results.push({
triggerBackendNodeId: candidate.backendNodeId,
triggerPoint: { x: candidate.x, y: candidate.y },
triggerAction: "hover",
subItems,
confidence: confidenceForHover(candidate, subItems),
});
} catch (error) {
if (isAbortError(error)) throw error;
continue;
} finally {
await clearHover(cdp, tabId);
}
} finally {
await options.hoverProbeBypassOverlay?.(tabId, false).catch(() => undefined);
}
return results;
} catch (err) {
@@ -817,6 +816,8 @@ function parseDocumentNodes(
const posCol = STYLE_COL.position;
const peCol = STYLE_COL["pointer-events"];
const cursorCol = STYLE_COL.cursor;
const visibilityCol = STYLE_COL.visibility;
const opacityCol = STYLE_COL.opacity;
const inputValues = sparseIndexMap(dn.inputValue);
const textValues = sparseIndexMap(dn.textValue);
const checkedInputs = new Set(dn.inputChecked?.index ?? []);
@@ -861,6 +862,7 @@ function parseDocumentNodes(
let position = "static";
let pointerEvents = "auto";
let cursor = "auto";
let rendered = false;
const li = layoutByNode.get(n);
if (li !== undefined) {
const b = dl?.bounds?.[li];
@@ -881,6 +883,13 @@ function parseDocumentNodes(
position = str(strings, styleRow[posCol]) || "static";
pointerEvents = str(strings, styleRow[peCol]) || "auto";
cursor = str(strings, styleRow[cursorCol]) || "auto";
const visibility = str(strings, styleRow[visibilityCol]) || "visible";
const opacity = str(strings, styleRow[opacityCol]) || "1";
rendered =
localRect !== null &&
visibility !== "hidden" &&
visibility !== "collapse" &&
(Number.parseFloat(opacity) || 0) > 0;
}
// Skip non-element nodes (#text, #cdata-section, etc.) — they carry no
@@ -916,6 +925,7 @@ function parseDocumentNodes(
position,
pointerEvents,
cursor,
rendered,
textContent,
...(formValue !== undefined ? { formValue } : {}),
...(tag === "input" || tag === "textarea"
@@ -1020,7 +1030,13 @@ export async function captureViewModel(
options: CaptureViewModelOptions = {},
): Promise<CapturedViewModel> {
throwIfAborted(options.signal);
const metrics = await cdp.send<LayoutMetricsReply>(tabId, "Page.getLayoutMetrics", {});
let metrics: LayoutMetricsReply = {};
try {
metrics = await cdp.send<LayoutMetricsReply>(tabId, "Page.getLayoutMetrics", {});
} catch (error) {
if (isAbortError(error)) throw error;
console.debug("[bsk capture] layout metrics unavailable", error);
}
throwIfAborted(options.signal);
const dpr = devicePixelRatio(metrics);
const vpSrc = metrics.cssLayoutViewport ?? metrics.layoutViewport ?? {};
@@ -1048,7 +1064,6 @@ export async function captureViewModel(
nodes: [],
viewport,
iframeNodes: new Map(),
surfaceProbes: [],
excludedBackendNodeIds: new Set(),
};
}
@@ -1077,11 +1092,6 @@ export async function captureViewModel(
await enrichFormControlStates(cdp, tabId, [nodes, ...iframeNodes.values()], options.signal);
throwIfAborted(options.signal);
const surfaceProbes = options.conditionalSurfaceProbe
? await probeHoverSurfaces(cdp, tabId, nodes, options)
: [];
throwIfAborted(options.signal);
const frameNodes = new Map<string, CapturedNode[]>();
if (topContext.frameId) frameNodes.set(topContext.frameId, nodes);
for (const iframeFrameNodes of iframeNodes.values()) {
@@ -1102,7 +1112,6 @@ export async function captureViewModel(
frameOwnerBackendNodeIds: frameParsed.frameOwnerBackendNodeIds,
frameParentIds: frameParsed.frameParentIds,
...(topContext.frameId ? { rootFrameId: topContext.frameId } : {}),
surfaceProbes,
excludedBackendNodeIds,
};
}
+24 -22
View File
@@ -33,25 +33,19 @@ async function discoverFrameGraph(cdp: CdpRunner, tabId: number): Promise<CdpFra
function transformFrameNodes(
nodes: CapturedNode[],
projection: GeometryProjection,
projection: GeometryProjection | null,
): CapturedNode[] {
return nodes.map((node) => ({
...node,
rect: (() => {
const bounds = projectRectToViewport(node.rect, projection);
return bounds ? { x: bounds.x, y: bounds.y, w: bounds.width, h: bounds.height } : null;
})(),
rect: projection
? (() => {
const bounds = projectRectToViewport(node.rect, projection);
return bounds ? { x: bounds.x, y: bounds.y, w: bounds.width, h: bounds.height } : null;
})()
: null,
}));
}
function frameOwnerNode(captured: CapturedViewModel, frame: CdpFrame): CapturedNode | undefined {
if (frame.ownerBackendNodeId === undefined) return undefined;
const parentNodes = frame.parentFrameId
? captured.frameNodes?.get(frame.parentFrameId)
: captured.nodes;
return parentNodes?.find((node) => node.backendNodeId === frame.ownerBackendNodeId);
}
async function captureMissingFrameDocuments(
cdp: CdpRunner,
tabId: number,
@@ -65,14 +59,20 @@ async function captureMissingFrameDocuments(
for (const frame of graph.frames) {
if (!frame.target.sessionId || captured.frameNodes.has(frame.frameId)) continue;
const owner = frameOwnerNode(captured, frame);
if (!owner?.rect || frame.ownerBackendNodeId === undefined) continue;
try {
const child = await captureViewModel(cdpRunnerForTarget(cdp, frame.target), tabId, {
signal,
});
const projection = await resolveFrameProjection(cdp, graph, frame.frameId);
if (!projection) continue;
let projection: GeometryProjection | null = null;
try {
projection = await resolveFrameProjection(cdp, graph, frame.frameId);
} catch (err) {
if (isAbortError(err)) throw err;
console.debug("[bsk observation] child frame geometry projection failed", {
frameId: frame.frameId,
err,
});
}
const rootFrameId = child.rootFrameId ?? frame.frameId;
const childFrames = child.frameNodes ?? new Map<string, CapturedNode[]>();
if (!childFrames.has(rootFrameId)) childFrames.set(rootFrameId, child.nodes);
@@ -85,11 +85,13 @@ async function captureMissingFrameDocuments(
for (const [childFrameId, nodes] of childFrames) {
captured.frameNodes.set(childFrameId, transformFrameNodes(nodes, projection));
}
captured.iframeNodes.set(
frame.ownerBackendNodeId,
captured.frameNodes.get(frame.frameId) ?? [],
);
captured.frameOwnerBackendNodeIds.set(frame.frameId, frame.ownerBackendNodeId);
if (frame.ownerBackendNodeId !== undefined) {
captured.iframeNodes.set(
frame.ownerBackendNodeId,
captured.frameNodes.get(frame.frameId) ?? [],
);
captured.frameOwnerBackendNodeIds.set(frame.frameId, frame.ownerBackendNodeId);
}
if (frame.parentFrameId) captured.frameParentIds.set(frame.frameId, frame.parentFrameId);
for (const [childFrameId, ownerBackendNodeId] of child.frameOwnerBackendNodeIds ?? []) {
captured.frameOwnerBackendNodeIds.set(childFrameId, ownerBackendNodeId);
@@ -77,9 +77,12 @@ export function buildFrameDocuments<T extends FrameOwnedAxNode>(
): FrameDocument<T>[] {
const frames = frameList(graph, batches, captured);
const frameById = new Map(frames.map((frame) => [frame.frameId, frame]));
const rootFrameId = graph?.rootFrameId ?? captured.rootFrameId ?? frames[0]?.frameId;
const domNodesForFrame = (frameId: string): CapturedNode[] =>
captured.frameNodes?.get(frameId) ?? (frameId === rootFrameId ? captured.nodes : []);
const backendOwner = new Map<string, string>();
for (const frame of frames) {
for (const node of captured.frameNodes?.get(frame.frameId) ?? []) {
for (const node of domNodesForFrame(frame.frameId)) {
backendOwner.set(targetBackendKey(frame.target, node.backendNodeId), frame.frameId);
}
}
@@ -165,6 +168,6 @@ export function buildFrameDocuments<T extends FrameOwnedAxNode>(
...frame,
contextScopeId: frame.frameId,
axNodes: axNodesByFrame.get(frame.frameId) ?? [],
domNodes: captured.frameNodes?.get(frame.frameId) ?? [],
domNodes: domNodesForFrame(frame.frameId),
}));
}
@@ -0,0 +1,125 @@
import type { CdpRunner } from "../shared";
function abortError(): Error {
const error = new Error("hover perception aborted");
error.name = "AbortError";
return error;
}
export async function waitForHover(ms: number, signal?: AbortSignal): Promise<void> {
if (signal?.aborted) throw abortError();
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => {
signal?.removeEventListener("abort", onAbort);
resolve();
}, ms);
const onAbort = () => {
clearTimeout(timer);
signal?.removeEventListener("abort", onAbort);
reject(abortError());
};
signal?.addEventListener("abort", onAbort, { once: true });
});
}
export async function clearHover(cdp: CdpRunner, tabId: number): Promise<void> {
await cdp
.send(tabId, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: -10,
y: -10,
})
.catch(() =>
cdp.send(tabId, "Input.dispatchMouseEvent", { type: "mouseMoved", x: 0, y: 0 }).catch(() => {
// Best effort cleanup.
}),
);
}
/**
* Wall-clock budget for a probe phase.
*
* Callers ask whether the *next* candidate still fits before starting it,
* rather than checking elapsed time at the top of the loop. A top-of-loop
* check lets a candidate start with 1ms of budget left and then run to
* completion, overshooting by a full settle window.
*/
export class ProbeBudget {
private readonly startedAt = Date.now();
constructor(private readonly totalMs: number) {}
get elapsedMs(): number {
return Date.now() - this.startedAt;
}
/** True when a candidate costing `estimatedMs` still fits in the budget. */
canAfford(estimatedMs: number): boolean {
return this.elapsedMs + estimatedMs <= this.totalMs;
}
}
export type OverlayBypass = (tabId: number, enabled: boolean) => Promise<void>;
export interface OverlayBypassScopeOptions {
/**
* Called when the overlay could not be restored. Defaults to `console.error`
* so a stuck-bypassed overlay is never silent; tests inject a spy.
*/
onRestoreFailure?: (error: unknown) => void;
}
const overlayBypassDepth = new Map<number, number>();
/**
* Runs `body` with the agent overlay bypassed, restoring it afterwards.
*
* Reference counted per tab so nested probe phases share a single bypass span
* instead of toggling the overlay once per phase. Restore failures are
* reported rather than swallowed, and the depth is released even when restore
* throws so a failure cannot pin the overlay off for the tab's lifetime.
*/
export async function withOverlayBypass<T>(
bypass: OverlayBypass | undefined,
tabId: number,
body: () => Promise<T>,
options: OverlayBypassScopeOptions = {},
): Promise<T> {
if (!bypass) return body();
const depth = overlayBypassDepth.get(tabId) ?? 0;
overlayBypassDepth.set(tabId, depth + 1);
if (depth === 0) {
try {
await bypass(tabId, true);
} catch (error) {
// Failing to hide the overlay only risks probing our own UI, so continue.
console.debug("[bsk hover] overlay bypass could not be enabled", error);
}
}
try {
return await body();
} finally {
const current = overlayBypassDepth.get(tabId) ?? 1;
if (current <= 1) {
overlayBypassDepth.delete(tabId);
try {
await bypass(tabId, false);
} catch (error) {
(options.onRestoreFailure ?? defaultRestoreFailure)(error);
}
} else {
overlayBypassDepth.set(tabId, current - 1);
}
}
}
function defaultRestoreFailure(error: unknown): void {
console.error("[bsk hover] agent overlay could not be restored after probing", error);
}
/** Test seam: drops any bypass depth left behind by a crashed probe. */
export function resetOverlayBypassDepth(): void {
overlayBypassDepth.clear();
}
@@ -0,0 +1,307 @@
import { isVomReferenceNode } from "@browser-skill/vom";
import { type CdpRunner, sendToCdpTarget } from "../shared";
import type { FrameDocument, FrameOwnedAxNode } from "./frame-document";
import { clearHover, ProbeBudget, waitForHover } from "./hover-perception";
import { stableIdentifierName } from "./semantic-graph/name-evidence";
import { frameBackendKey, type ResolvedSemanticGraph } from "./semantic-graph/types";
const MAX_TOOLTIP_CANDIDATES = 8;
const MAX_TOOLTIP_PROBE_MS = 2_400;
const TOOLTIP_SETTLE_MS = 250;
const CACHE_TTL_MS = 5 * 60_000;
/**
* Misses expire sooner than hits: a control that gains a tooltip once the app
* finishes booting should still be discoverable, but not at the cost of
* re-probing every unlabelled control on every observation.
*/
const MISS_CACHE_TTL_MS = 60_000;
const MAX_CACHE_ENTRIES = 500;
const FORM_CONTROL_TAGS = new Set(["input", "select", "textarea"]);
interface TooltipCacheEntry {
/** `null` records "probed, revealed no tooltip" so the miss is not re-paid. */
name: string | null;
expiresAt: number;
}
interface TooltipCandidate<T extends FrameOwnedAxNode> {
document: FrameDocument<T>;
backendNodeId: number;
x: number;
y: number;
stableIdentifier: boolean;
}
export interface TooltipNameProbeOptions {
signal?: AbortSignal;
}
const tooltipCache = new Map<string, TooltipCacheEntry>();
function cacheKey<T extends FrameOwnedAxNode>(
document: FrameDocument<T>,
backendNodeId: number,
): string {
return `${document.target.tabId}\u0000${document.frameId}\u0000${document.url ?? ""}\u0000${backendNodeId}`;
}
/** Returns the cache entry, or `undefined` when this control was never probed. */
function cachedEntry<T extends FrameOwnedAxNode>(
document: FrameDocument<T>,
backendNodeId: number,
): TooltipCacheEntry | undefined {
const key = cacheKey(document, backendNodeId);
const entry = tooltipCache.get(key);
if (!entry) return undefined;
if (entry.expiresAt <= Date.now()) {
tooltipCache.delete(key);
return undefined;
}
return entry;
}
function storeName<T extends FrameOwnedAxNode>(
document: FrameDocument<T>,
backendNodeId: number,
name: string | null,
): void {
if (tooltipCache.size >= MAX_CACHE_ENTRIES) {
const oldest = tooltipCache.keys().next().value;
if (oldest !== undefined) tooltipCache.delete(oldest);
}
tooltipCache.set(cacheKey(document, backendNodeId), {
name,
expiresAt: Date.now() + (name === null ? MISS_CACHE_TTL_MS : CACHE_TTL_MS),
});
}
function tooltipCollectorFunction(): string {
return `function() {
const ownRoot = this && typeof this.getRootNode === "function" ? this.getRootNode() : null;
const roots = ownRoot && ownRoot !== document ? [ownRoot, document] : [document];
const items = [];
const seen = new Set();
for (const root of roots) {
for (const el of Array.from(root.querySelectorAll("[role='tooltip']"))) {
const style = getComputedStyle(el);
const rect = el.getBoundingClientRect();
if (rect.width <= 0 || rect.height <= 0) continue;
if (style.display === "none" || style.visibility === "hidden" || style.opacity === "0") continue;
const text = String(el.getAttribute("aria-label") || el.getAttribute("title") || el.textContent || "")
.replace(/\\s+/g, " ")
.trim();
if (!text || text.length > 160 || seen.has(text)) continue;
seen.add(text);
items.push(text);
}
}
return items.slice(0, 20);
}`;
}
function cleanTooltipDiff(before: string[], after: string[]): string | undefined {
const existing = new Set(before.map((value) => value.toLowerCase()));
const added = [...new Set(after.map((value) => value.replace(/\s+/g, " ").trim()))].filter(
(value) => value && !existing.has(value.toLowerCase()),
);
return added.length === 1 ? added[0] : undefined;
}
function candidates<T extends FrameOwnedAxNode>(
documents: FrameDocument<T>[],
graph: ResolvedSemanticGraph,
): TooltipCandidate<T>[] {
const documentByFrame = new Map(documents.map((document) => [document.frameId, document]));
const byFrame = new Map<string, TooltipCandidate<T>[]>();
for (const node of graph.nodes.values()) {
if (
node.vom.name ||
FORM_CONTROL_TAGS.has(node.dom?.tag.toLowerCase() ?? "") ||
!node.referenceable ||
node.backendNodeId === undefined ||
!isVomReferenceNode({
...node.vom,
id: 0,
parentId: null,
backendNodeId: node.backendNodeId,
frameId: node.frameId,
referenceable: true,
})
) {
continue;
}
const document = documentByFrame.get(node.frameId);
const rect = node.dom?.rect;
if (!document || !rect || rect.w <= 0 || rect.h <= 0) continue;
if (
rect.x + rect.w <= 0 ||
rect.y + rect.h <= 0 ||
rect.x >= graph.viewport.width ||
rect.y >= graph.viewport.height
) {
continue;
}
if (
node.vom.disabled ||
node.vom.inert ||
node.vom.pointerEvents === "none" ||
node.vom.sensitive
) {
continue;
}
const frameCandidates = byFrame.get(node.frameId) ?? [];
frameCandidates.push({
document,
backendNodeId: node.backendNodeId,
x: (Math.max(0, rect.x) + Math.min(graph.viewport.width, rect.x + rect.w)) / 2,
y: (Math.max(0, rect.y) + Math.min(graph.viewport.height, rect.y + rect.h)) / 2,
stableIdentifier: stableIdentifierName(node.dom?.attrs ?? {}) !== undefined,
});
byFrame.set(node.frameId, frameCandidates);
}
const out: TooltipCandidate<T>[] = [];
for (const stableIdentifier of [true, false]) {
const queues = documents.map((document) =>
(byFrame.get(document.frameId) ?? []).filter(
(candidate) => candidate.stableIdentifier === stableIdentifier,
),
);
for (let index = 0; queues.some((queue) => index < queue.length); index += 1) {
for (const queue of queues) {
const candidate = queue[index];
if (candidate) out.push(candidate);
}
}
}
return out;
}
async function executionContextId<T extends FrameOwnedAxNode>(
cdp: CdpRunner,
document: FrameDocument<T>,
): Promise<number | undefined> {
const result = await sendToCdpTarget<{ executionContextId?: number }>(
cdp,
document.target,
"Page.createIsolatedWorld",
{
frameId: document.frameId,
worldName: "bsk-vom-tooltip",
grantUniveralAccess: false,
},
);
return result.executionContextId;
}
async function resolvedObjectId<T extends FrameOwnedAxNode>(
cdp: CdpRunner,
document: FrameDocument<T>,
contextId: number,
backendNodeId: number,
): Promise<string | undefined> {
const result = await sendToCdpTarget<{
object?: { objectId?: string };
}>(cdp, document.target, "DOM.resolveNode", {
backendNodeId,
executionContextId: contextId,
});
return result.object?.objectId;
}
async function tooltipTexts<T extends FrameOwnedAxNode>(
cdp: CdpRunner,
document: FrameDocument<T>,
objectId: string,
): Promise<string[]> {
const result = await sendToCdpTarget<{
result?: { value?: string[] };
}>(cdp, document.target, "Runtime.callFunctionOn", {
functionDeclaration: tooltipCollectorFunction(),
objectId,
returnByValue: true,
});
return result.result?.value ?? [];
}
async function releaseObject<T extends FrameOwnedAxNode>(
cdp: CdpRunner,
document: FrameDocument<T>,
objectId: string,
): Promise<void> {
await sendToCdpTarget(cdp, document.target, "Runtime.releaseObject", { objectId }).catch(
() => undefined,
);
}
export async function probeTooltipNames<T extends FrameOwnedAxNode>(
cdp: CdpRunner,
tabId: number,
documents: FrameDocument<T>[],
graph: ResolvedSemanticGraph,
options: TooltipNameProbeOptions = {},
): Promise<Map<string, string>> {
const names = new Map<string, string>();
const pending: TooltipCandidate<T>[] = [];
for (const candidate of candidates(documents, graph)) {
const cached = cachedEntry(candidate.document, candidate.backendNodeId);
if (cached) {
// A cached miss is still a decision: skip the control instead of re-probing.
if (cached.name !== null) {
names.set(
frameBackendKey(candidate.document.frameId, candidate.backendNodeId),
cached.name,
);
}
} else if (pending.length < MAX_TOOLTIP_CANDIDATES) {
pending.push(candidate);
}
}
if (pending.length === 0) return names;
const budget = new ProbeBudget(MAX_TOOLTIP_PROBE_MS);
const contexts = new Map<string, number>();
try {
for (const candidate of pending) {
if (options.signal?.aborted) throw new DOMException("observation aborted", "AbortError");
if (!budget.canAfford(TOOLTIP_SETTLE_MS)) break;
let objectId: string | undefined;
try {
let contextId = contexts.get(candidate.document.frameId);
if (contextId === undefined) {
contextId = await executionContextId(cdp, candidate.document);
if (contextId === undefined) continue;
contexts.set(candidate.document.frameId, contextId);
}
objectId = await resolvedObjectId(
cdp,
candidate.document,
contextId,
candidate.backendNodeId,
);
if (!objectId) continue;
await clearHover(cdp, tabId);
const before = await tooltipTexts(cdp, candidate.document, objectId);
await cdp.send(tabId, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: candidate.x,
y: candidate.y,
});
await waitForHover(TOOLTIP_SETTLE_MS, options.signal);
const after = await tooltipTexts(cdp, candidate.document, objectId);
const name = cleanTooltipDiff(before, after);
storeName(candidate.document, candidate.backendNodeId, name ?? null);
if (!name) continue;
names.set(frameBackendKey(candidate.document.frameId, candidate.backendNodeId), name);
} catch (error) {
if (error instanceof Error && error.name === "AbortError") throw error;
} finally {
if (objectId) await releaseObject(cdp, candidate.document, objectId);
await clearHover(cdp, tabId);
}
}
} finally {
await clearHover(cdp, tabId);
}
return names;
}
@@ -0,0 +1,101 @@
import type { Rect, RenderedRef, VomResult } from "@browser-skill/vom";
import type { CdpTarget } from "@/browser-driver/frame-graph";
import { readRecordingDocumentIdentity } from "@/shared/recording-document-identity";
import type { CapturedSurfaceProbe } from "./capture";
import type { CapturedFrameDocument, FrameAxNode } from "./frame-capture";
/** Frame identity needed to resolve an `@eN` that lives in an iframe. */
export interface CaptureVomFrame {
frameId: string;
target: CdpTarget;
parentFrameId?: string;
ownerBackendNodeId?: number;
/** BrowserSkill-generated identity for the recording agent in this Document. */
recordingDocumentId?: string;
}
/** Allowlisted geometry used to match a recorded action to a rendered ref. */
export interface CaptureVomMatchNode {
frameId: string;
backendNodeId: number;
tag: string;
/** Top-level viewport-relative geometry. */
rect: Rect | null;
/** Frame-local viewport-relative geometry, when available. */
localRect?: Rect | null;
}
/**
* Raw AX/DOM trees are never returned. When `redactValues` is enabled by the
* caller, form values in `text` are masked as well.
*/
export interface CaptureVomObservationResult {
text: string;
refs: RenderedRef[];
truncated: boolean;
rootFrameId: string;
frames: CaptureVomFrame[];
matchNodes: CaptureVomMatchNode[];
surfaceProbes?: CapturedSurfaceProbe[];
/**
* Whether this observation actively hovered the page, and whether that
* revealed content the static tree does not contain. Lets callers judge how
* far the page may have drifted from the returned snapshot.
*/
hoverProbe?: HoverProbeReport;
}
export interface HoverProbeReport {
performed: boolean;
revealedContent: boolean;
}
function projectFrames(documents: CapturedFrameDocument<FrameAxNode>[]): CaptureVomFrame[] {
return documents.map((document) => {
let recordingDocumentId: string | undefined;
for (const node of document.domNodes) {
recordingDocumentId = readRecordingDocumentIdentity(node.attrs);
if (recordingDocumentId) break;
}
return {
frameId: document.frameId,
target: document.target,
...(document.parentFrameId ? { parentFrameId: document.parentFrameId } : {}),
...(document.ownerBackendNodeId !== undefined
? { ownerBackendNodeId: document.ownerBackendNodeId }
: {}),
...(recordingDocumentId ? { recordingDocumentId } : {}),
};
});
}
function projectMatchNodes(documents: CapturedFrameDocument<FrameAxNode>[]): CaptureVomMatchNode[] {
return documents.flatMap((document) =>
document.domNodes.map((node) => ({
frameId: document.frameId,
backendNodeId: node.backendNodeId,
tag: node.tag,
rect: node.rect,
...(node.localRect !== undefined ? { localRect: node.localRect } : {}),
})),
);
}
export function projectRecordSafeObservation(input: {
rootFrameId: string;
frameDocuments: CapturedFrameDocument<FrameAxNode>[];
rendered: VomResult;
surfaceProbes?: CapturedSurfaceProbe[];
hoverProbe?: HoverProbeReport;
}): CaptureVomObservationResult {
return {
text: input.rendered.text,
refs: input.rendered.refs,
truncated: input.rendered.truncated,
rootFrameId: input.rootFrameId,
frames: projectFrames(input.frameDocuments),
matchNodes: projectMatchNodes(input.frameDocuments),
surfaceProbes: input.surfaceProbes,
...(input.hoverProbe ? { hoverProbe: input.hoverProbe } : {}),
};
}
@@ -0,0 +1,154 @@
import type { Viewport } from "@browser-skill/vom";
import type { FrameDocument } from "../frame-document";
import {
frameAxKey,
frameBackendKey,
type SemanticAxNode,
type SemanticFrame,
type SemanticGraph,
type SemanticGraphNode,
type SemanticNodeId,
} from "./types";
export interface BuildSemanticGraphInput {
documents: FrameDocument<SemanticAxNode>[];
viewport: Viewport;
rootFrameId?: string;
excludedBackendNodeIds?: ReadonlySet<number>;
}
function domNodeId(frameId: string, backendNodeId: number): SemanticNodeId {
return JSON.stringify([frameId, "dom", backendNodeId]);
}
function axNodeId(frameId: string, nodeId: string): SemanticNodeId {
return JSON.stringify([frameId, "ax", nodeId]);
}
export function buildSemanticGraph(input: BuildSemanticGraphInput): SemanticGraph {
const rootFrameId =
input.rootFrameId ??
input.documents.find((document) => !document.parentFrameId)?.frameId ??
input.documents[0]?.frameId ??
"root";
const frames = new Map<string, SemanticFrame>();
const nodes = new Map<SemanticNodeId, SemanticGraphNode>();
const nodeIdsByFrame = new Map<string, SemanticNodeId[]>();
const nodeByFrameBackend = new Map<string, SemanticNodeId>();
const nodeByFrameAx = new Map<string, SemanticNodeId>();
for (const [frameOrder, document] of input.documents.entries()) {
frames.set(document.frameId, {
frameId: document.frameId,
...(document.parentFrameId ? { parentFrameId: document.parentFrameId } : {}),
...(document.ownerBackendNodeId !== undefined
? { ownerBackendNodeId: document.ownerBackendNodeId }
: {}),
contextScopeId: document.contextScopeId,
target: document.target,
...(document.url ? { url: document.url } : {}),
order: frameOrder,
});
const frameNodeIds: SemanticNodeId[] = [];
nodeIdsByFrame.set(document.frameId, frameNodeIds);
for (const [index, dom] of document.domNodes.entries()) {
const id = domNodeId(document.frameId, dom.backendNodeId);
const node: SemanticGraphNode = {
id,
frameId: document.frameId,
backendNodeId: dom.backendNodeId,
dom,
sourceOrder: index,
excluded:
document.frameId === rootFrameId &&
input.excludedBackendNodeIds?.has(dom.backendNodeId) === true,
};
nodes.set(id, node);
frameNodeIds.push(id);
nodeByFrameBackend.set(frameBackendKey(document.frameId, dom.backendNodeId), id);
}
for (const [index, ax] of document.axNodes.entries()) {
const joinedId =
typeof ax.backendDOMNodeId === "number"
? nodeByFrameBackend.get(frameBackendKey(document.frameId, ax.backendDOMNodeId))
: undefined;
const id = joinedId ?? axNodeId(document.frameId, ax.nodeId);
const existing = nodes.get(id);
if (existing) {
existing.ax = ax;
existing.axNodeId = ax.nodeId;
} else {
nodes.set(id, {
id,
frameId: document.frameId,
...(typeof ax.backendDOMNodeId === "number"
? { backendNodeId: ax.backendDOMNodeId }
: {}),
axNodeId: ax.nodeId,
ax,
sourceOrder: document.domNodes.length + index,
excluded:
document.frameId === rootFrameId &&
typeof ax.backendDOMNodeId === "number" &&
input.excludedBackendNodeIds?.has(ax.backendDOMNodeId) === true,
});
frameNodeIds.push(id);
}
nodeByFrameAx.set(frameAxKey(document.frameId, ax.nodeId), id);
}
}
for (const document of input.documents) {
for (const dom of document.domNodes) {
const id = nodeByFrameBackend.get(frameBackendKey(document.frameId, dom.backendNodeId));
const node = id ? nodes.get(id) : undefined;
if (!node || dom.parentBackendNodeId === null) continue;
node.domParentId = nodeByFrameBackend.get(
frameBackendKey(document.frameId, dom.parentBackendNodeId),
);
}
for (const ax of document.axNodes) {
const id = nodeByFrameAx.get(frameAxKey(document.frameId, ax.nodeId));
const node = id ? nodes.get(id) : undefined;
if (!node || !ax.parentId) continue;
node.axParentId = nodeByFrameAx.get(frameAxKey(document.frameId, ax.parentId));
}
}
const axChildren = new Map<SemanticNodeId, SemanticNodeId[]>();
const excludedQueue: SemanticNodeId[] = [];
for (const node of nodes.values()) {
if (node.axParentId) {
const children = axChildren.get(node.axParentId) ?? [];
children.push(node.id);
axChildren.set(node.axParentId, children);
}
if (node.excluded) excludedQueue.push(node.id);
}
for (let index = 0; index < excludedQueue.length; index += 1) {
for (const childId of axChildren.get(excludedQueue[index]) ?? []) {
const child = nodes.get(childId);
if (!child || child.excluded) continue;
child.excluded = true;
excludedQueue.push(childId);
}
}
for (const frame of frames.values()) {
if (!frame.parentFrameId || frame.ownerBackendNodeId === undefined) continue;
frame.ownerNodeId = nodeByFrameBackend.get(
frameBackendKey(frame.parentFrameId, frame.ownerBackendNodeId),
);
}
return {
viewport: input.viewport,
rootFrameId,
frames,
nodes,
nodeIdsByFrame,
nodeByFrameBackend,
};
}
@@ -0,0 +1,59 @@
import type { Viewport, VomScene } from "@browser-skill/vom";
import type { FrameDocument } from "../frame-document";
import { buildSemanticGraph as buildGraph } from "./build";
import { projectSemanticGraph as projectGraph } from "./project";
import { resolveSemanticGraph as resolveGraph } from "./resolve";
import { normalizeSemanticStructure as normalizeStructure } from "./structure";
import type {
ResolvedSemanticGraph,
SemanticAxNode,
SemanticGraph,
StructuredSemanticGraph,
} from "./types";
export type { SemanticAxNode } from "./types";
export interface SemanticVomInput {
documents: FrameDocument<SemanticAxNode>[];
viewport: Viewport;
rootFrameId?: string;
excludedBackendNodeIds?: ReadonlySet<number>;
}
export function buildSemanticGraph(input: SemanticVomInput): SemanticGraph {
return buildGraph(input);
}
export function resolveSemanticGraph(
graph: SemanticGraph,
options: {
supplementalNames?: ReadonlyMap<string, string>;
identifierFallback?: boolean;
} = {},
): ResolvedSemanticGraph {
return resolveGraph(graph, options);
}
export function normalizeSemanticStructure(graph: ResolvedSemanticGraph): StructuredSemanticGraph {
return normalizeStructure(graph);
}
export function projectSemanticGraph(graph: StructuredSemanticGraph): VomScene {
return projectGraph(graph);
}
export function buildSemanticVomScene(
input: SemanticVomInput & {
supplementalNames?: ReadonlyMap<string, string>;
identifierFallback?: boolean;
},
): VomScene {
return projectSemanticGraph(
normalizeSemanticStructure(
resolveSemanticGraph(buildSemanticGraph(input), {
supplementalNames: input.supplementalNames,
identifierFallback: input.identifierFallback,
}),
),
);
}
@@ -0,0 +1,34 @@
const IDENTIFIER_ATTRS = ["data-testid", "data-test-id", "id"] as const;
const GENERIC_TOKENS = new Set(["btn", "button", "control", "icon", "item", "test", "tool"]);
const GENERATED_PREFIXES = new Set(["ember", "headlessui", "mui", "radix", "react", "rc"]);
function identifierTokens(value: string): string[] {
return value
.replace(/([a-z0-9])([A-Z])/g, "$1 $2")
.split(/[^a-zA-Z0-9]+/)
.map((token) => token.toLowerCase())
.filter(Boolean);
}
function readableIdentifier(value: string | undefined): string | undefined {
const raw = value?.trim();
if (!raw || raw.length < 3 || raw.length > 80) return undefined;
if (/^[a-f0-9]{8,}$/i.test(raw)) return undefined;
const tokens = identifierTokens(raw);
if (GENERATED_PREFIXES.has(tokens[0])) return undefined;
while (tokens.length > 1 && GENERIC_TOKENS.has(tokens[0])) tokens.shift();
while (tokens.length > 1 && GENERIC_TOKENS.has(tokens[tokens.length - 1])) tokens.pop();
if (tokens.length === 0 || !tokens.some((token) => /[a-z]{3,}/.test(token))) return undefined;
return tokens
.map((token, index) => (index === 0 ? `${token[0].toUpperCase()}${token.slice(1)}` : token))
.join(" ");
}
export function stableIdentifierName(attrs: Readonly<Record<string, string>>): string | undefined {
for (const name of IDENTIFIER_ATTRS) {
const label = readableIdentifier(attrs[name]);
if (label) return label;
}
return undefined;
}
@@ -0,0 +1,85 @@
import type { VomNode, VomScene } from "@browser-skill/vom";
import type { SemanticNodeId, StructuredSemanticGraph, StructuredSemanticNode } from "./types";
function retained(node: StructuredSemanticNode): boolean {
return node.structure.kind === "keep" || node.structure.kind === "inferred-group";
}
function connectedFrameIds(graph: StructuredSemanticGraph): Set<string> {
const connected = new Set<string>([graph.rootFrameId]);
const childrenByParent = new Map<string, string[]>();
for (const frame of graph.frames.values()) {
if (!frame.parentFrameId) continue;
const children = childrenByParent.get(frame.parentFrameId) ?? [];
children.push(frame.frameId);
childrenByParent.set(frame.parentFrameId, children);
}
const queue = [graph.rootFrameId];
for (let index = 0; index < queue.length; index += 1) {
for (const frameId of childrenByParent.get(queue[index]) ?? []) {
if (connected.has(frameId)) continue;
const frame = graph.frames.get(frameId);
if (!frame?.ownerNodeId) continue;
const owner = graph.nodes.get(frame.ownerNodeId);
if (!owner || !retained(owner)) continue;
connected.add(frame.frameId);
queue.push(frame.frameId);
}
}
return connected;
}
function domAncestors(
graph: StructuredSemanticGraph,
node: StructuredSemanticNode,
numericId: ReadonlyMap<SemanticNodeId, number>,
): number[] {
const ancestors: number[] = [];
const seen = new Set<SemanticNodeId>();
let current = node.domParentId;
while (current && !seen.has(current)) {
seen.add(current);
const id = numericId.get(current);
if (id !== undefined) ancestors.push(id);
current = graph.nodes.get(current)?.domParentId;
}
return ancestors;
}
export function projectSemanticGraph(graph: StructuredSemanticGraph): VomScene {
const connectedFrames = connectedFrameIds(graph);
const orderedNodes = [...graph.frames.values()]
.filter((frame) => connectedFrames.has(frame.frameId))
.sort((a, b) => a.order - b.order)
.flatMap((frame) =>
(graph.nodeIdsByFrame.get(frame.frameId) ?? [])
.map((id) => graph.nodes.get(id))
.filter((node): node is StructuredSemanticNode => node !== undefined && retained(node))
.sort((a, b) => a.sourceOrder - b.sourceOrder),
);
const numericId = new Map<SemanticNodeId, number>();
orderedNodes.forEach((node, index) => numericId.set(node.id, index + 1));
const nodes: VomNode[] = orderedNodes.map((node) => ({
...node.vom,
id: numericId.get(node.id) as number,
parentId: node.semanticParentId ? (numericId.get(node.semanticParentId) ?? null) : null,
...(node.backendNodeId !== undefined ? { backendNodeId: node.backendNodeId } : {}),
frameId: node.frameId,
contextScopeId: graph.frames.get(node.frameId)?.contextScopeId ?? node.frameId,
referenceable: node.referenceable,
...(node.dom
? {
domParentId:
node.domParentId === undefined ? null : (numericId.get(node.domParentId) ?? null),
domAncestorIds: domAncestors(graph, node, numericId),
}
: {}),
}));
return {
viewport: graph.viewport,
nodes,
rootFrameId: graph.rootFrameId,
};
}

Some files were not shown because too many files have changed in this diff Show More