Compare commits

..
Author SHA1 Message Date
Ljy-0827 27b5d988be fix(vom): bug fix 2026-09-03 16:37:33 +08:00
Ljy-0827 28892ea01a feat(vom): two channels to deal with canvas 2026-09-02 17:27:29 +08:00
Ljy-0827 a6d4c14968 feat(vom): fixture test 2026-09-02 17:26:43 +08:00
Ljy-0827 d1c1778054 fix(vom): canvas recognition 2026-09-02 17:26:43 +08:00
Ljy-0827 d75495878a feat(vom): vom can distinguish important canvas area 2026-09-02 17:26:43 +08:00
Hannah Liu 59e2c3224e Merge pull request #123 from Tencent/feat/upload-download
feat(bsk): add secure, session-scoped file upload and download | 支持安全、会话级的文件上传与下载
2026-09-02 16:47:19 +08:00
drakezhangandCursor d70df753f6 merge: resolve SKILL.md conflicts with main
Keep the expanded command-reference tables from feat/upload-download
(including the new File Transfer section) over main's compact two-line
format.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 16:42:15 +08:00
Zhang GH 66ae555e40 Merge pull request #172 from Tencent/fix/pip-close-and-obs-mediatype
fix(dsh-plugin): PiP window close button & observation thumbnail media type detection
2026-09-02 16:25:33 +08:00
drakezhangandCursor fee7c0983c fix: sort import order to satisfy biome organizeImports
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 16:21:26 +08:00
drakezhangandCursor 51473c01ff fix(dsh-plugin): add PiP close button and sniff observation thumbnail media type
The Document PiP window had no close/return entry once popped out —
users could not dismiss it from inside the window. Add an onClosePip
callback and render a close button in the PiP header; clicking it
calls pipWindow.close() and the content returns to its carrier.

The observation thumbnail publisher hardcoded mediaType as "image/png"
regardless of the actual capture bytes. Reuse the existing
sniffImageMediaType helper (exported from image.ts) so the HTTP
thumbnail endpoint declares the correct Content-Type when Chromium
returns JPEG captures.

Closes #153 (Bug 2 fully; Bug 1 was fixed in #135, this covers the
remaining observation-thumbnail path).

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 16:14:24 +08:00
Zhang GH 49744233f7 Merge pull request #171 from Tencent/fix/borrow-confirmation-ux
fix(borrow): improve confirmation UX — proactive focus and longer tim…
2026-09-02 15:23:12 +08:00
drakezhangandCursor 95571f8b70 fix(borrow): improve confirmation UX — proactive focus and longer timeout
The borrow confirmation overlay was easily missed because the Agent
Window stole focus and the 5-second auto-deny was too aggressive for
users who had context-switched away.

- Focus the user window proactively when a borrow request is issued so
  the overlay is immediately visible even when the Agent Window was
  last focused.
- Raise CONFIRMATION_TIMEOUT_MS from 5 s to 60 s, giving users enough
  time to notice and respond — the OS notification with
  requireInteraction: true now has a matching window of usefulness.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 15:11:40 +08:00
Zhang GH 8344e311fa Merge pull request #169 from Tencent/perf/vom-hover-probe-opt-in
Perf/vom hover probe opt in
2026-09-02 14:27:29 +08:00
drakezhangandCursor 63338cd98e Merge branch 'main' into perf/vom-hover-probe-opt-in
main rewrote SKILL.md into a leaner core (#167) while this branch documented
the opt-in --probe-hover flag against the old structure. Keep main's rewritten
prose and re-apply the probe-hover guidance on top of it: the [has-submenu] /
[expanded] markers, when reaching for --probe-hover is warranted, and its slot
in the page-reading escalation ladder ahead of get-html/screenshot.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 12:18:06 +08:00
Zhang GH bd193ceb5d Merge pull request #167 from Tencent/refactor/lean-browser-skill
Refactor/lean browser skill
2026-09-02 12:09:42 +08:00
Zhang GH 5ed828736f Merge pull request #165 from Tencent/perf/vom-algorithmic-optimizations
perf(vom): optimize rendering algorithms
2026-09-02 12:09:22 +08:00
drakezhang ee620acf64 docs(skill): give probe-hover a decidable trigger condition
The previous wording asked the agent to notice when a page "looks like"
it hides content behind hover. That is not something an agent reading a
text tree can judge: hover-only content is invisible by definition, and
the marker that used to point at it came from the probing that is now off
by default.

Replaces it with a condition the agent can actually evaluate — an
expected control is missing and no [has-submenu] marker points at a
trigger — and places it on the escalation ladder ahead of get-html and
screenshot, which is where that dead end is currently reached.

Also documents [has-submenu]/[expanded] as the primary hover signal. It
comes from aria-haspopup in the accessibility tree, costs nothing, and is
unaffected by probing being off, so it covers menus that are marked up
correctly.
2026-09-02 10:59:18 +08:00
drakezhang 05a36075b8 perf(vom): make active hover probing opt-in on observe
Every tool.observe moved the real cursor across the page to look for
hover-only menus and tooltips. Measured against real Chrome, that cost a
median of 2.5s and up to 4.7s per observation — 73% to 99% of total
observe wall clock — and returned 3 hits across 12 pages. A positive
control confirms the probes work; the yield on real pages simply does not
justify putting them on the default path.

observe now probes only when asked, via `probe_hover` (`--probe-hover`).
Agents that know which element hides a menu should keep using
`bsk hover <ref>`, which is both cheaper and precise. A held hover latch
still suppresses probing, since probing would move the cursor off the
element the caller is deliberately holding.

Correctness fixes that apply whether or not probing is requested:

- Hover probing ran between the DOM snapshot and the accessibility tree
  fetch, so a hover that opened a menu left one half of an observation
  describing the page before the change and the other half after it. Both
  chains now run after all capture is complete.
- The overlay bypass toggled once per chain and swallowed restore
  failures, which could leave the agent overlay hidden for the rest of
  the tab's life. It is now reference counted across both chains, and a
  failed restore is reported instead of discarded.
- Observations report whether they hovered the page and whether that
  revealed content the static tree lacks, so callers can judge how far
  the live page may have drifted from the returned snapshot.

Probe budgets were checked at the top of the loop, so a candidate could
start with 1ms remaining and still run a full settle window, overshooting
by up to 634ms. Candidates are now admitted only if they fit. Raising
MAX_HOVER_PROBE_MS to 2600 is part of that fix rather than a relaxation:
2000 was never the real ceiling, and enforcing it literally would have
silently dropped a candidate the positive control depends on.

Tooltip misses are cached too, with a shorter TTL than hits, so repeat
observations of the same page no longer re-probe every unlabelled control
from scratch.
2026-09-02 10:59:18 +08:00
Ljy-0827 1bd13d5c3c fix(bsk): fix upload download racing 2026-09-02 10:56:02 +08:00
drakezhangandCursor 59d6533eff docs(skill): trim agent instructions to a leaner core
The CLI skill had grown to 20.8 KB of command tables and flag catalogs that
`bsk --help` already answers at runtime, so every agent paid for them on every
task. Rewrite both skills around the durable rules an agent cannot rediscover
-- session lifecycle, the observe/act/observe loop, the Agent Window boundary,
and the stop-on-success contract -- and delegate flag details to the CLI's own
help output. The CLI skill drops to 6.9 KB (-67%) and the DSH skill to 5.1 KB.

Keep the parts that help discovery has no answer for: the literal
`[hover first: ...]` observation marker, the `request-help` outcome values, the
complete command-name inventory, and the required flags agents guess wrong
(`fill --value`, `select --value`, `screenshot --out`, `emulate --device`).
Also bind `--no-focus` explicitly to `bsk session start`, since a bare mention
led agents to pass it to `bsk navigate`.

Validated with 36 agent runs of the `core` eval suite against a real browser,
comparing the old and new instructions with everything else held constant.
Pass rate did not regress (the lean version was better on `form-controls`) and
token cost per task came out even. The inventory and flag signatures cut
`--help` round-trips from 1.7 to 0.7 per task and removed the invented
`bsk back` / `bsk forward` calls the first draft produced.

Tighten the DSH size assertion to a range so the budget is bounded from both
sides and accidental truncation fails the test.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-01 18:39:19 +08:00
drakezhangandCursor 536d3a9844 chore: ignore local pnpm store and git worktrees
Both directories are local scratch space that kept showing up as untracked
noise in `git status`.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-01 18:39:06 +08:00
drakezhang 641d934337 perf(vom): optimize rendering algorithms 2026-09-01 14:55:36 +08:00
Ljy-0827 3b21f2f0a7 fix(bsk): download bug fix 2026-08-31 17:58:33 +08:00
Ljy-0827 405433ab65 fix(bsk): fix ci 2026-08-31 17:58:33 +08:00
Ljy-0827 58c7f38a05 feat(bsk): harden browser file transfer transactions 2026-08-31 17:58:33 +08:00
Ljy-0827 e0c88295c7 fix(bsk): fix ci 2026-08-31 17:58:33 +08:00
Ljy-0827 dc36e38cb9 fix(bsk): upload function safety 2026-08-31 17:58:33 +08:00
Ljy-0827 d5de9b0154 feat(bsk): fix upload function that cannot bypass layout 2026-08-31 17:58:33 +08:00
Ljy-0827 995061514c feat(bsk): fix different type of upload 2026-08-31 17:58:33 +08:00
Ljy-0827 406bb21b7e feat(bsk): support upload and download 2026-08-31 17:58:33 +08:00
Zhang GH f06a514bd2 Merge pull request #149 from Tencent/fix/protocol-upgrade-reminder
fix:将协议版本差异改为升级提醒
2026-08-31 12:25:29 +08:00
Zhang GH 1e53ee63ae Merge pull request #155 from Tencent/ci/edge-addons-publishing
ci(extension): publish updates to Microsoft Edge Add-ons
2026-08-31 11:24:22 +08:00
Zhang GH 120dc441bb Merge pull request #158 from Tencent/codex/fix-dsh-screenshot-test-name
test(dsh-plugin): use renamed screenshot tool
2026-08-29 16:08:01 +08:00
drakezhang 710b1219f4 test(dsh-plugin): use renamed screenshot tool 2026-08-29 16:05:06 +08:00
Zhang GH 177ecd3ebf Merge pull request #157 from Tencent/refactor/simplify-dsh-commands
Refactor/simplify dsh commands
2026-08-29 15:55:16 +08:00
Zhang GH 0e6b9c4542 Merge pull request #135 from cyhano/fix/screenshot-media-type
fix: use detected media type for screenshots instead of hard-coded image/png
2026-08-29 15:52:43 +08:00
drakezhang 506adbe708 Merge remote-tracking branch 'origin/main' into refactor/simplify-dsh-commands 2026-08-29 15:16:31 +08:00
Zhang GH 984d017b0f Merge pull request #156 from Tencent/fix/unify-dsh-plugin-package-id
fix(dsh-plugin): use the published package name as the Cordis id
2026-08-29 15:12:02 +08:00
drakezhang 5df93143c5 refactor(dsh-plugin): simplify browser commands 2026-08-29 14:38:48 +08:00
Zhang GH 7163f2f9b0 Merge pull request #151 from xiyanjun/docs/evaluate-json-ok
docs(skill): clarify bsk evaluate exit-code vs --json .ok semantics
close #133
2026-08-28 22:06:04 +08:00
drakezhang e9f0584af3 ci(extension): publish updates to Microsoft Edge Add-ons 2026-08-28 21:39:53 +08:00
drakezhangandCursor 2997f495db fix(dsh-plugin): use the published package name as the Cordis id
dsh 0.1.0-rc.8 imports and registers the client bundle by export const name, so a leftover dsh-plugin-browserskill id fails to load the 0.1.1 web UI.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-28 21:28:06 +08:00
xiyanjun f96bd72c3c docs(skill): clarify bsk evaluate exit-code vs --json .ok semantics
The exit code reflects RPC success only, not JS success. Document that programmatic callers must use --json and check the .ok field to detect JS errors. Addresses the discoverability gap raised in issue #133.
2026-08-28 18:02:38 +08:00
Zhang GH 329c68054b Merge pull request #150 from Tencent/codex/browser-eval-harness
test: 增加可扩展的浏览器能力回归测试框架
2026-08-28 16:41:00 +08:00
drakezhang 5cd43553ed docs(eval): clarify agent-neutral scope 2026-08-28 15:36:38 +08:00
drakezhang 9ad33801c7 test: add extensible browser evaluation harness 2026-08-28 12:16:39 +08:00
Zhang GH 7aaa5a2d01 Merge pull request #147 from Tencent/fix/dsh-plugin-session-lifecycle
fix(dsh-plugin): stabilize skill loading and browser sessions
2026-08-28 12:11:08 +08:00
drakezhang 210b21b637 style(dsh-plugin): format observation 2026-08-28 12:06:47 +08:00
Hannah Liu 48f64ec0df Merge pull request #122 from Tencent/feat/vom-functional-refactor
feat(vom): refactor semantic structure generation for grouped observations | 重构语义结构生成,支持分组式 VOM 观察
2026-08-28 11:24:34 +08:00
“haonan”andCursor c48a53269e style: rustfmt doctor protocol-version test asserts
cargo fmt --check failed on two long assert lines in PR CI.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-27 22:50:01 +08:00
“haonan”andCursor 3bbd6ab19f fix: treat protocol version differences as upgrade reminders
Agents were stopping on popup and status copy that looked like a hard incompatibility. Keep the connection usable, reframe exit 5 as an unknown method, and ask users to upgrade instead of aborting.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-27 22:22:30 +08:00
drakezhang 1a70d2200e fix(dsh-plugin): stabilize skill loading and browser sessions
Register bundled browser instructions at agent scope, prioritize foreground commands over thumbnail capture, and unify cancellation and session cleanup.
2026-08-27 19:59:20 +08:00
Zhang GH 554861d6b1 Merge pull request #131 from Tencent/feat/dsh-browser-tools-parity
feat(dsh-plugin): add Browser tool parity and dedicated Harness skill
2026-08-27 16:29:30 +08:00
Ljy-0827 0d7f976335 fix(vom): fix repeat names 2026-08-26 15:28:27 +08:00
Ljy-0827 d1d3eb6a94 fix(vom): bug fix 2026-08-26 15:28:27 +08:00
Ljy-0827 38fd7d3f4c fix(vom): safety policy and rebase bug fix 2026-08-26 15:28:27 +08:00
Ljy-0827 b5062ac19b feat(vom): fix name lost 2026-08-26 15:28:27 +08:00
Ljy-0827 41cb576256 feat(vom): function refactor 2026-08-26 15:28:27 +08:00
Hannah Liu f060a7c074 Merge pull request #142 from Tencent/chore/release-ext-0.1.7
chore(release): extension 0.1.7
2026-08-26 15:19:33 +08:00
Ljy-0827 60b62d5420 chore(release): extension 0.1.7 2026-08-26 15:12:44 +08:00
Hannah Liu 70b851a3e8 Merge pull request #141 from Tencent/chore/release-bsk-0.1.11
chore(release): bsk 0.1.11
2026-08-26 15:06:30 +08:00
Ljy-0827 da3534d066 chore(release): bsk 0.1.11 2026-08-26 14:59:09 +08:00
BB-fat a004291848 feat(dsh-plugin): better-sidebar 集成、会话关停与归档清理、悬浮窗修复 (#130)
* better-sidebar 集成:observation 视图迁入 better-sidebar 载体,按会话作用域展示 bsk 会话
* 会话关停与归档清理:支持会话 dispose 与归档自动清理
* 悬浮窗修复:悬浮 overlay 拖拽/缩放/PiP 弹出等问题修复
* 修复 Document PiP 空白窗口:载体卸载(关 tab、切会话、插件 HMR)时主动 close() PiP 窗口,避免残留空白窗口
2026-08-25 10:33:46 +08:00
cyhanoandDeepSeek Harness 60fbf30e24 fix: use detected media type for screenshots instead of hard-coded image/png
browser_screenshot declared every capture as image/png regardless of the
actual bytes. On Chromium/Edge builds where captureVisibleTab returns JPEG
bytes, the stored attachment reference recorded mediaType "image/png" against
JPEG bytes, so dsh-attachment-local's read-time metadata verification fails
with ATTACHMENT_CORRUPT ("Stored attachment metadata does not match its
reference"), surfacing as code UNKNOWN and triggering endless LLM retries.

- image.ts: sniff the capture's magic bytes (PNG/JPEG/WebP/GIF) and declare
  that real media type to attachments.saveImage, instead of always "image/png".
- tools.ts: assemble the tool result from ref.mediaType (the store's verified
  type) rather than re-declaring "image/png" as const, and relax the output
  schema's const to the accepted media-type enum.
- tests: add a regression case asserting JPEG capture bytes surface as
  image/jpeg.

Co-Authored-By: DeepSeek Harness <noreply@deepseek.com>
2026-08-24 15:31:34 +08:00
Hannah Liu 4b6cdde168 Merge pull request #115 from Tencent/fix/recorder-iframe
fix(recording): support user action recording in iframes and OOPIFs
2026-08-24 13:18:05 +08:00
Ljy-0827 4aade4b94f fix(recorder): bug fix, oopif load too late 2026-08-24 10:42:46 +08:00
Ljy-0827 616af1099f fix(record): saftey policy and rebase bug fix 2026-08-24 10:42:46 +08:00
Ljy-0827 3656f0dd3c fix(recorder): ci fix 2026-08-24 10:42:46 +08:00
Ljy-0827 9742d83213 fix(recorder): bug fix 2026-08-24 10:42:46 +08:00
Ljy-0827 615e866608 fix(recorder): add iframe support 2026-08-24 10:42:46 +08:00
Ljy-0827 c08fed5de3 fix(record): saftey policy and rebase bug fix 2026-08-24 10:42:45 +08:00
“haonan” 887d1ed0b9 fix(record): drop 'hover' in tracev2 for compatible 2026-08-24 10:42:45 +08:00
Ljy-0827 1a3d3b7de8 fix(record): bug fix 2026-08-24 10:42:45 +08:00
“haonan”andCursor 72834980d2 fix(record): record switch_tab and keep Trace v3 states per tab
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-24 10:42:45 +08:00
drakezhang a4b6a61d29 docs(dsh-plugin): clarify package description 2026-08-22 21:29:33 +08:00
drakezhang cad0b5cbc9 feat(dsh-plugin): add Browser tool parity 2026-08-22 21:15:53 +08:00
234 changed files with 22779 additions and 2689 deletions
+177
View File
@@ -325,3 +325,180 @@ jobs:
echo "Chrome Web Store submission created successfully:"
jq '{itemId, state, warningInfo}' "$PUBLISH_RESPONSE"
publish-edge-addons:
needs: [resolve, build-extension]
runs-on: ubuntu-latest
environment: edge-addons
permissions:
contents: read
steps:
- name: Download build artifact
uses: actions/download-artifact@v8
with:
name: extension-zip
path: dist
- name: Validate Microsoft Edge Add-ons configuration
shell: bash
env:
API_KEY: ${{ secrets.EDGE_ADDONS_API_KEY }}
CLIENT_ID: ${{ vars.EDGE_ADDONS_CLIENT_ID }}
PRODUCT_ID: ${{ vars.EDGE_ADDONS_PRODUCT_ID }}
run: |
set -euo pipefail
if [ -z "$API_KEY" ]; then
echo "::error::Environment secret EDGE_ADDONS_API_KEY is not configured"
exit 1
fi
if [ -z "$CLIENT_ID" ]; then
echo "::error::Environment variable EDGE_ADDONS_CLIENT_ID is not configured"
exit 1
fi
if [ -z "$PRODUCT_ID" ]; then
echo "::error::Environment variable EDGE_ADDONS_PRODUCT_ID is not configured"
exit 1
fi
if ! [[ "$PRODUCT_ID" =~ ^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$ ]]; then
echo "::error::EDGE_ADDONS_PRODUCT_ID must be the Partner Center product GUID, not the public Edge Add-ons extension ID"
exit 1
fi
ARCHIVE="$(find dist -maxdepth 1 -name '*.zip' -type f -print -quit)"
if [ -z "$ARCHIVE" ]; then
echo "::error::No extension zip found in the downloaded artifact"
exit 1
fi
echo "ARCHIVE=$ARCHIVE" >> "$GITHUB_ENV"
- name: Upload and publish Microsoft Edge Add-ons extension
shell: bash
env:
API_KEY: ${{ secrets.EDGE_ADDONS_API_KEY }}
CLIENT_ID: ${{ vars.EDGE_ADDONS_CLIENT_ID }}
PRODUCT_ID: ${{ vars.EDGE_ADDONS_PRODUCT_ID }}
RELEASE_VERSION: ${{ needs.resolve.outputs.version }}
run: |
set -euo pipefail
API_ROOT="https://api.addons.microsoftedge.microsoft.com/v1"
UPLOAD_HEADERS="$RUNNER_TEMP/edge-upload-headers.txt"
UPLOAD_RESPONSE="$RUNNER_TEMP/edge-upload-response.json"
STATUS_RESPONSE="$RUNNER_TEMP/edge-status-response.json"
PUBLISH_HEADERS="$RUNNER_TEMP/edge-publish-headers.txt"
PUBLISH_RESPONSE="$RUNNER_TEMP/edge-publish-response.json"
operation_id_from_headers() {
local headers="$1"
local location
location="$(awk '
tolower($1) == "location:" {
sub(/\r$/, "")
sub(/^[^:]*:[[:space:]]*/, "")
print
exit
}
' "$headers")"
location="${location%%\?*}"
printf '%s' "${location##*/}"
}
poll_operation() {
local operation_name="$1"
local status_url="$2"
local status="InProgress"
local http_status
for attempt in $(seq 1 30); do
sleep 5
http_status="$(curl --silent --show-error \
--output "$STATUS_RESPONSE" \
--write-out '%{http_code}' \
--header "Authorization: ApiKey $API_KEY" \
--header "X-ClientID: $CLIENT_ID" \
"$status_url")"
if [ "$http_status" -lt 200 ] || [ "$http_status" -ge 300 ]; then
echo "::error::Microsoft Edge Add-ons $operation_name status check failed with HTTP $http_status"
jq . "$STATUS_RESPONSE" || cat "$STATUS_RESPONSE"
exit 1
fi
status="$(jq -r '.status // empty' "$STATUS_RESPONSE")"
echo "$operation_name status check $attempt/30: $status"
case "$status" in
Succeeded)
return 0
;;
InProgress)
;;
Failed)
echo "::error::Microsoft Edge Add-ons $operation_name failed"
jq . "$STATUS_RESPONSE"
exit 1
;;
*)
echo "::error::Microsoft Edge Add-ons returned an unknown $operation_name status: $status"
jq . "$STATUS_RESPONSE"
exit 1
;;
esac
done
echo "::error::Microsoft Edge Add-ons $operation_name did not complete after 30 checks"
jq . "$STATUS_RESPONSE"
exit 1
}
HTTP_STATUS="$(curl --silent --show-error \
--output "$UPLOAD_RESPONSE" \
--dump-header "$UPLOAD_HEADERS" \
--write-out '%{http_code}' \
--request POST \
--header "Authorization: ApiKey $API_KEY" \
--header "X-ClientID: $CLIENT_ID" \
--header "Content-Type: application/zip" \
--data-binary "@$ARCHIVE" \
"$API_ROOT/products/$PRODUCT_ID/submissions/draft/package")"
if [ "$HTTP_STATUS" != "202" ]; then
echo "::error::Microsoft Edge Add-ons upload failed with HTTP $HTTP_STATUS"
jq . "$UPLOAD_RESPONSE" || cat "$UPLOAD_RESPONSE"
exit 1
fi
UPLOAD_OPERATION_ID="$(operation_id_from_headers "$UPLOAD_HEADERS")"
if [ -z "$UPLOAD_OPERATION_ID" ]; then
echo "::error::Microsoft Edge Add-ons upload response has no Location operation ID"
cat "$UPLOAD_HEADERS"
exit 1
fi
poll_operation \
"package upload" \
"$API_ROOT/products/$PRODUCT_ID/submissions/draft/package/operations/$UPLOAD_OPERATION_ID"
HTTP_STATUS="$(curl --silent --show-error \
--output "$PUBLISH_RESPONSE" \
--dump-header "$PUBLISH_HEADERS" \
--write-out '%{http_code}' \
--request POST \
--header "Authorization: ApiKey $API_KEY" \
--header "X-ClientID: $CLIENT_ID" \
--header "Content-Type: application/json" \
--data "{\"notes\":\"Automated BrowserSkill extension v${RELEASE_VERSION} release from GitHub Actions.\"}" \
"$API_ROOT/products/$PRODUCT_ID/submissions")"
if [ "$HTTP_STATUS" != "202" ]; then
echo "::error::Microsoft Edge Add-ons publish failed with HTTP $HTTP_STATUS"
jq . "$PUBLISH_RESPONSE" || cat "$PUBLISH_RESPONSE"
exit 1
fi
PUBLISH_OPERATION_ID="$(operation_id_from_headers "$PUBLISH_HEADERS")"
if [ -z "$PUBLISH_OPERATION_ID" ]; then
echo "::error::Microsoft Edge Add-ons publish response has no Location operation ID"
cat "$PUBLISH_HEADERS"
exit 1
fi
poll_operation \
"submission" \
"$API_ROOT/products/$PRODUCT_ID/submissions/operations/$PUBLISH_OPERATION_ID"
echo "Microsoft Edge Add-ons submission created successfully:"
jq '{status, message, errorCode, errors}' "$STATUS_RESPONSE"
+8
View File
@@ -7,10 +7,18 @@ dist/
.output/
.wxt/
.web-extension-id
.pnpm-store/
# Git worktrees
.worktrees/
# Logs
*.log
# Local browser evaluation artifacts
evals/browser/results/
evals/browser/agents.local.json
# macOS
.DS_Store
Generated
+3 -2
View File
@@ -134,9 +134,10 @@ dependencies = [
[[package]]
name = "bsk"
version = "0.1.10"
version = "0.1.11"
dependencies = [
"anyhow",
"base64",
"bsk-protocol",
"clap",
"console",
@@ -168,7 +169,7 @@ dependencies = [
[[package]]
name = "bsk-protocol"
version = "0.1.10"
version = "0.1.11"
dependencies = [
"schemars",
"semver",
+2 -1
View File
@@ -7,7 +7,7 @@ members = [
default-members = ["crates/bsk-cli"]
[workspace.package]
version = "0.1.10"
version = "0.1.11"
edition = "2024"
rust-version = "1.85"
license = "MIT"
@@ -36,6 +36,7 @@ flate2 = "1.1.9"
tar = "0.4.46"
zip = { version = "7.2.0", default-features = false, features = ["deflate"] }
sha2 = "0.11.0"
base64 = "0.22"
tempfile = "3"
nix = { version = "0.29", features = ["signal", "process"] }
rand = "0.8"
+1
View File
@@ -208,6 +208,7 @@ The repository is a Cargo + pnpm workspace:
- `apps/extension` — browser extension
- `packages/ui` and `packages/i18n` — shared extension UI support
- `packages/dsh-plugin-browserskill` — DeepSeek Harness plugin (`@wxg-prc-cpg/browser-skill-dsh-plugin`)
- [`evals/browser`](evals/browser/README.md) — deterministic local pages and agent-neutral browser capability evaluation
## License
+1
View File
@@ -178,6 +178,7 @@ Agent 不直接与浏览器通信。它通过 `bsk` CLI 下发浏览器任务;
- `apps/extension` — 浏览器扩展
- `packages/ui` 和 `packages/i18n` — 扩展 UI 共享支持
- `packages/dsh-plugin-browserskill` — DeepSeek Harness 插件(`@wxg-prc-cpg/browser-skill-dsh-plugin`)
- [`evals/browser`](evals/browser/README.zh-CN.md) — 确定性本地页面与 Agent 无关的浏览器能力测试台
## 许可证
+6 -3
View File
@@ -1,6 +1,6 @@
# BrowserSkill — Privacy Policy
**Last updated:** May 25, 2026
**Last updated:** August 20, 2026
This Privacy Policy describes how the **BrowserSkill** browser extension (the "Extension") handles information when you install and use it. BrowserSkill is published as part of the open-source [BrowserSkill](https://github.com/Tencent/BrowserSkill) project. The source code is publicly auditable.
@@ -14,7 +14,7 @@ BrowserSkill is a local automation bridge that lets AI coding agents (such as Cu
## 2. Single Purpose
The Extension's single purpose is to expose browser automation primitives (navigation, DOM observation, screenshots, clicks, form filling, tab management) to a locally running BrowserSkill daemon over a WebSocket connection on `127.0.0.1`, so that an AI agent invoked by the user can interact with web pages on the user's behalf.
The Extension's single purpose is to expose browser automation primitives (navigation, DOM observation, screenshots, clicks, form filling, task-scoped file transfer, and tab management) to a locally running BrowserSkill daemon over a WebSocket connection on `127.0.0.1`, so that an AI agent invoked by the user can interact with web pages on the user's behalf.
## 3. Data the Extension Accesses
@@ -26,6 +26,7 @@ Depending on the commands the user (via their AI agent) sends to the local daemo
| **User input simulated by the agent** | Mouse clicks, keystrokes, and form values that the AI agent dispatches through the Chrome DevTools Protocol (CDP). | Required to perform automation actions the user has asked the agent to do. |
| **Tab and window metadata** | Tab IDs, URLs, titles, window IDs of the Agent Window and any tabs the user explicitly authorizes. | Required to target automation commands at the correct tab/window. |
| **Local extension storage** | A randomly generated 8-character instance ID and an optional user-supplied label. | Used so the local daemon can recognize this browser instance across reconnects. No personal data is stored. |
| **File transfers requested by the agent** | Local files explicitly supplied to `bsk upload`, and the file created by a single `bsk download` action. | Required to attach a task file to a web page or return a browser-generated download to the invoking local agent. |
| **OS notifications** | Permission to display a system notification when the agent requests to "borrow" one of the user's existing tabs. | Required to obtain explicit, per-tab user consent before the agent touches any pre-existing tab. |
## 4. Data the Extension Does **Not** Collect
@@ -34,7 +35,7 @@ BrowserSkill does **not**:
- Send any data to remote servers, the Extension's authors, or any third party.
- Call any LLM, AI, or cloud API. The Extension contains no API keys, model identifiers, or remote endpoints.
- Read or transmit cookies, browsing history, bookmarks, downloads, saved passwords, or autofill data.
- Read or transmit cookies, browsing history, bookmarks, saved passwords, or autofill data. It observes only the download initiated by an active `bsk download` call, not download history generally.
- Use webcam, microphone, geolocation, or any device sensor.
- Include analytics, telemetry, crash reporting, advertising SDKs, or fingerprinting code.
- Track users across websites or across sessions.
@@ -50,6 +51,7 @@ The Extension requests the following Chrome permissions. Each is used solely for
- **`alarms`** — Periodically wake the service worker to keep the local WebSocket connection alive.
- **`idle`** — Detect when the device returns from idle/locked so the Extension can promptly re-establish the local WebSocket connection after the machine wakes. No idle data is stored or transmitted.
- **`notifications`** — Show a system notification to obtain user approval before the agent borrows a user-owned tab.
- **`downloads`** — Correlate and route the one browser download initiated by an active `bsk download` command. If that claimed transaction fails, BrowserSkill cancels an in-progress file or removes its completed temporary browser file. It is not used to enumerate download history or alter unclaimed downloads.
- **`storage`** — Persist a random instance ID and optional label in `chrome.storage.local`.
- **Host permission `<all_urls>`** — Inject a small status overlay (showing "Agent Active") on pages controlled by the agent, and enable automation across whatever sites the user directs the agent to. The Extension does **not** read or transmit page content from sites the agent is not actively driving.
@@ -61,6 +63,7 @@ All Extension activity stays on the user's local device. The only network traffi
- The instance ID and optional label persist in `chrome.storage.local` until the user uninstalls the Extension or clears extension storage.
- Page content, screenshots, DOM snapshots, and other observed data are returned to the local daemon in response to commands and are **not retained by the Extension**. They live only as long as the agent's tool call.
- Upload and download bytes are staged by the local daemon in a private, session-scoped directory. Download staging is removed after it is copied to the requested destination. Upload staging is retained until the session ends so a later form submission can still read the attached file. Remaining staging is removed when the session ends or disconnects, or when the daemon next starts after a crash.
## 8. User Control
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@browser-skill/extension",
"version": "0.1.6",
"version": "0.1.7",
"private": true,
"type": "module",
"scripts": {
@@ -23,7 +23,7 @@ describe("BorrowConfirmationOverlay", () => {
id: "req-1",
isActiveTab: true,
tabTitle: "Example",
timeoutMs: 5000,
timeoutMs: 60_000,
onAllow,
onDeny,
},
@@ -32,7 +32,7 @@ describe("BorrowConfirmationOverlay", () => {
);
expect(screen.getByText("允许借用标签页?")).toBeTruthy();
for (let i = 0; i < 5; i++) {
for (let i = 0; i < 60; i++) {
await act(async () => {
await vi.advanceTimersByTimeAsync(1000);
});
@@ -66,7 +66,7 @@ describe("BorrowConfirmationOverlay", () => {
id: "req-fallback",
isActiveTab: true,
tabTitle: "Example",
timeoutMs: 5000,
timeoutMs: 60_000,
onAllow,
onDeny,
},
@@ -74,7 +74,7 @@ describe("BorrowConfirmationOverlay", () => {
/>,
);
for (let i = 0; i < 5; i++) {
for (let i = 0; i < 60; i++) {
await act(async () => {
await vi.advanceTimersByTimeAsync(1000);
});
@@ -1,6 +1,6 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { RECORD_START, RECORD_STOP, type RecordStepPayload } from "@/lib/record-bridge";
import { handleRecordContentMessage, startRecordCapture } from "../record-capture";
import type { RecordStepPayload } from "@/lib/record-bridge";
import { startRecordCapture } from "../record-capture";
vi.stubGlobal("chrome", {
runtime: {
@@ -48,118 +48,6 @@ function click(el: Element): void {
el.dispatchEvent(new MouseEvent("click", { bubbles: true, button: 0, detail: 1 }));
}
describe("handleRecordContentMessage stop/cancel", () => {
it("ignores STOP when no recording is active", () => {
const dispose = vi.fn();
const onStop = vi.fn();
const setActiveRequestId = vi.fn();
const setCapture = vi.fn();
const sendResponse = vi.fn();
const needsAsync = handleRecordContentMessage(
{ type: RECORD_STOP, requestId: "rec-stale" },
{
activeRequestId: null,
capture: { dispose },
setActiveRequestId,
setCapture,
onStart: vi.fn(),
onStop,
},
sendResponse,
);
expect(needsAsync).toBe(false);
expect(dispose).not.toHaveBeenCalled();
expect(onStop).not.toHaveBeenCalled();
expect(setActiveRequestId).not.toHaveBeenCalled();
expect(setCapture).not.toHaveBeenCalled();
expect(sendResponse).not.toHaveBeenCalled();
});
it("ignores STOP for a mismatched requestId", () => {
const dispose = vi.fn();
const onStop = vi.fn();
const needsAsync = handleRecordContentMessage(
{ type: RECORD_STOP, requestId: "rec-other" },
{
activeRequestId: "rec-1",
capture: { dispose },
setActiveRequestId: vi.fn(),
setCapture: vi.fn(),
onStart: vi.fn(),
onStop,
},
vi.fn(),
);
expect(needsAsync).toBe(false);
expect(dispose).not.toHaveBeenCalled();
expect(onStop).not.toHaveBeenCalled();
});
it("keeps a failed STOP retryable and redelivers its recorded step", async () => {
document.body.innerHTML = `<label for="retry">Draft</label><input id="retry" />`;
const sendMessage = vi.mocked(chrome.runtime.sendMessage);
sendMessage.mockReset();
sendMessage.mockRejectedValueOnce(new Error("service worker unavailable"));
sendMessage.mockRejectedValueOnce(new Error("service worker still unavailable"));
sendMessage.mockResolvedValueOnce({ ok: true, sequence: 1 });
let activeRequestId: string | null = null;
let capture: ReturnType<typeof startRecordCapture> | null = null;
const onStop = vi.fn();
const dispatch = (
message:
| { type: typeof RECORD_START; requestId: string; startedAtMs?: number }
| { type: typeof RECORD_STOP; requestId: string },
sendResponse?: (response: unknown) => void,
) =>
handleRecordContentMessage(
message,
{
activeRequestId,
capture,
setActiveRequestId: (id) => {
activeRequestId = id;
},
setCapture: (next) => {
capture = next;
},
onStart: vi.fn(),
onStop,
},
sendResponse,
);
dispatch({ type: RECORD_START, requestId: "rec-retry" });
const input = document.querySelector("input")!;
input.dispatchEvent(new FocusEvent("focusin", { bubbles: true }));
input.value = "dirty final value";
input.dispatchEvent(new Event("input", { bubbles: true }));
const firstResponse = vi.fn();
expect(dispatch({ type: RECORD_STOP, requestId: "rec-retry" }, firstResponse)).toBe(true);
await vi.waitFor(() =>
expect(firstResponse).toHaveBeenCalledWith({
ok: false,
error: "failed to deliver one or more recorded steps",
}),
);
expect(activeRequestId).toBe("rec-retry");
expect(onStop).not.toHaveBeenCalled();
const retryResponse = vi.fn();
expect(dispatch({ type: RECORD_STOP, requestId: "rec-retry" }, retryResponse)).toBe(true);
await vi.waitFor(() => expect(retryResponse).toHaveBeenCalledWith({ ok: true }));
expect(sendMessage).toHaveBeenCalledTimes(3);
expect(activeRequestId).toBeNull();
expect(onStop).toHaveBeenCalledTimes(1);
});
});
describe("record-capture semantic", () => {
let steps: RecordStepPayload[];
@@ -236,6 +124,19 @@ describe("record-capture semantic", () => {
]);
});
it("does not emit page navigation from a child Document capture", () => {
const originalUrl = location.href;
const capture = startRecordCapture("rec-child", (step) => steps.push(step), {
captureNavigation: false,
});
history.pushState({}, "", "#inside-frame");
expect(steps).toEqual([]);
capture.dispose();
history.replaceState({}, "", originalUrl);
});
it("does not record clicks on anonymous layout divs", () => {
document.body.innerHTML = `
<div id="chrome">page chrome</div>
@@ -0,0 +1,102 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import type { RecordFramePortMessage } from "@/lib/recording/frame-bridge";
import { RECORD_FRAME_START } from "@/lib/recording/frame-bridge";
import { RECORD_DOCUMENT_ATTRIBUTE } from "@/shared/recording-document-identity";
import { RecordFrameAgent } from "../recording/frame-agent";
class PortListeners<T extends (...args: never[]) => unknown> {
readonly values = new Set<T>();
addListener = (listener: T) => this.values.add(listener);
removeListener = (listener: T) => this.values.delete(listener);
}
function portHarness() {
const onMessage = new PortListeners<(message: unknown) => void>();
const onDisconnect = new PortListeners<() => void>();
const outbound: RecordFramePortMessage[] = [];
const port = {
name: "bsk-record-frame",
onMessage,
onDisconnect,
postMessage(message: RecordFramePortMessage) {
outbound.push(message);
if (message.type === "ready") {
queueMicrotask(() => {
for (const listener of onMessage.values) {
listener({
type: "ready_ack",
requestId: message.requestId,
producerId: message.producerId,
});
}
});
}
},
disconnect: vi.fn(),
} as unknown as chrome.runtime.Port;
return {
port,
outbound,
receive(message: RecordFramePortMessage) {
for (const listener of onMessage.values) listener(message);
},
};
}
describe("RecordFrameAgent", () => {
afterEach(() => {
document.documentElement.removeAttribute(RECORD_DOCUMENT_ATTRIBUTE);
document.body.replaceChildren();
});
it("keeps a failed stop retryable and flushes the final dirty fill", async () => {
const harness = portHarness();
const sendMessage = vi
.fn<(message: { sequence: number }) => Promise<unknown>>()
.mockRejectedValueOnce(new Error("offline"))
.mockRejectedValueOnce(new Error("still offline"))
.mockImplementation(async (message) => ({ ok: true, sequence: message.sequence }));
vi.stubGlobal("chrome", {
runtime: {
connect: vi.fn(() => harness.port),
sendMessage,
},
});
document.body.innerHTML = `<label for="draft">Draft</label><input id="draft" />`;
const agent = new RecordFrameAgent();
await expect(
agent.start({ type: RECORD_FRAME_START, requestId: "rec-1", startedAtMs: 10 }),
).resolves.toEqual({ ok: true });
expect(document.documentElement.hasAttribute(RECORD_DOCUMENT_ATTRIBUTE)).toBe(true);
const input = document.querySelector("input")!;
input.dispatchEvent(new FocusEvent("focusin", { bubbles: true }));
input.value = "final value";
input.dispatchEvent(new Event("input", { bubbles: true }));
harness.receive({ type: "stop", requestId: "rec-1", commandId: "stop-1" });
await vi.waitFor(() =>
expect(harness.outbound).toContainEqual({
type: "stopped",
requestId: "rec-1",
commandId: "stop-1",
ok: false,
error: "failed to deliver one or more recorded steps",
}),
);
expect(document.documentElement.hasAttribute(RECORD_DOCUMENT_ATTRIBUTE)).toBe(true);
harness.receive({ type: "stop", requestId: "rec-1", commandId: "stop-2" });
await vi.waitFor(() =>
expect(harness.outbound).toContainEqual({
type: "stopped",
requestId: "rec-1",
commandId: "stop-2",
ok: true,
}),
);
expect(sendMessage).toHaveBeenCalledTimes(3);
expect(document.documentElement.hasAttribute(RECORD_DOCUMENT_ATTRIBUTE)).toBe(false);
});
});
+30 -131
View File
@@ -12,21 +12,7 @@ import {
hasDirectHoverInteractiveSignal,
hasStrongHoverExpansionSignal,
} from "@/lib/hover-trigger-policy";
import {
isRecordCancelMessage,
isRecordStartMessage,
isRecordStopMessage,
RECORD_CANCEL,
RECORD_START,
RECORD_STEP,
RECORD_STOP,
type RecordCancelMessage,
type RecordStartAck,
type RecordStartMessage,
type RecordStepPayload,
type RecordStopAck,
type RecordStopMessage,
} from "@/lib/record-bridge";
import type { RecordStepPayload } from "@/lib/record-bridge";
import { shouldRecordPress } from "@/lib/recording/draft-policy";
import {
closestHoverSurfaceCandidate,
@@ -36,18 +22,6 @@ import {
isHoverSurfaceCandidateElement,
isLikelyHoverSurfaceOwner,
} from "./record-hover-surface";
import { RecordStepDelivery } from "./record-step-delivery";
const stepDeliveries = new Map<string, RecordStepDelivery>();
const pendingStopFlushes = new Map<string, Promise<RecordStopAck>>();
function deliveryFor(requestId: string): RecordStepDelivery {
const existing = stepDeliveries.get(requestId);
if (existing) return existing;
const created = new RecordStepDelivery(requestId);
stepDeliveries.set(requestId, created);
return created;
}
export interface RecordCaptureController {
dispose(): void;
@@ -385,6 +359,7 @@ function scheduleInputCompletionCommit(
export function startRecordCapture(
_requestId: string,
sendStep: (step: RecordStepPayload) => void,
options: { captureNavigation?: boolean } = {},
): RecordCaptureController {
const emitStep = (step: RecordStepPayload) => {
sendStep({ page_url: location.href, ...step });
@@ -921,26 +896,33 @@ export function startRecordCapture(
document.addEventListener("change", onChange, true);
document.addEventListener("keydown", onKeyDown, true);
const urlObserver = new MutationObserver(() => emitNavigateIfChanged());
urlObserver.observe(document, { subtree: true, childList: true });
const captureNavigation = options.captureNavigation ?? true;
const urlObserver = captureNavigation
? new MutationObserver(() => emitNavigateIfChanged())
: null;
urlObserver?.observe(document, { subtree: true, childList: true });
const onUrlEvent = () => emitNavigateIfChanged();
// Wrap: passing commitFillSession directly would forward the DOM event as
// the `commit` argument and stamp it onto the recorded fill step.
const onPageHide = () => commitFillSession();
window.addEventListener("hashchange", onUrlEvent);
window.addEventListener("popstate", onUrlEvent);
if (captureNavigation) {
window.addEventListener("hashchange", onUrlEvent);
window.addEventListener("popstate", onUrlEvent);
}
window.addEventListener("pagehide", onPageHide);
const originalPushState = history.pushState;
const originalReplaceState = history.replaceState;
history.pushState = function (...args: Parameters<History["pushState"]>) {
originalPushState.apply(this, args);
emitNavigateIfChanged(navigationActionPending ? true : undefined);
};
history.replaceState = function (...args: Parameters<History["replaceState"]>) {
originalReplaceState.apply(this, args);
emitNavigateIfChanged(navigationActionPending ? true : undefined);
};
if (captureNavigation) {
history.pushState = function (...args: Parameters<History["pushState"]>) {
originalPushState.apply(this, args);
emitNavigateIfChanged(navigationActionPending ? true : undefined);
};
history.replaceState = function (...args: Parameters<History["replaceState"]>) {
originalReplaceState.apply(this, args);
emitNavigateIfChanged(navigationActionPending ? true : undefined);
};
}
return {
dispose() {
@@ -954,99 +936,16 @@ export function startRecordCapture(
document.removeEventListener("compositionend", onCompositionEnd, true);
document.removeEventListener("change", onChange, true);
document.removeEventListener("keydown", onKeyDown, true);
urlObserver.disconnect();
window.removeEventListener("hashchange", onUrlEvent);
window.removeEventListener("popstate", onUrlEvent);
urlObserver?.disconnect();
if (captureNavigation) {
window.removeEventListener("hashchange", onUrlEvent);
window.removeEventListener("popstate", onUrlEvent);
}
window.removeEventListener("pagehide", onPageHide);
history.pushState = originalPushState;
history.replaceState = originalReplaceState;
if (captureNavigation) {
history.pushState = originalPushState;
history.replaceState = originalReplaceState;
}
},
};
}
export type RecordContentMessage = RecordStartMessage | RecordStopMessage | RecordCancelMessage;
export function isRecordContentMessage(msg: unknown): msg is RecordContentMessage {
return isRecordStartMessage(msg) || isRecordStopMessage(msg) || isRecordCancelMessage(msg);
}
export function handleRecordContentMessage(
message: RecordContentMessage,
state: {
activeRequestId: string | null;
capture: RecordCaptureController | null;
setActiveRequestId(id: string | null): void;
setCapture(capture: RecordCaptureController | null): void;
onStart(requestId: string, startedAtMs?: number): void;
onStop(): void;
},
sendResponse?: (response: RecordStartAck | RecordStopAck) => void,
): boolean {
if (isRecordStartMessage(message)) {
const delivery = deliveryFor(message.requestId);
state.capture?.dispose();
state.setCapture(
startRecordCapture(message.requestId, (step) => {
delivery.enqueue(step);
}),
);
state.setActiveRequestId(message.requestId);
state.onStart(message.requestId, message.startedAtMs);
sendResponse?.({ ok: true });
return sendResponse !== undefined;
}
if (isRecordStopMessage(message) || isRecordCancelMessage(message)) {
// Require an active recording that matches this requestId — otherwise a
// stray STOP/CANCEL (e.g. after teardown) would still run finishStop and
// clear overlay state even though nothing was capturing.
if (!state.activeRequestId || state.activeRequestId !== message.requestId) {
return false;
}
state.capture?.dispose();
state.setCapture(null);
const finishStop = () => {
state.onStop();
state.setActiveRequestId(null);
};
if (isRecordStopMessage(message) && sendResponse) {
const existingFlush = pendingStopFlushes.get(message.requestId);
if (existingFlush) {
void existingFlush.then(sendResponse);
return true;
}
const flush = deliveryFor(message.requestId)
.flush()
.then((succeeded): RecordStopAck => {
if (succeeded) {
stepDeliveries.delete(message.requestId);
finishStop();
}
return succeeded
? { ok: true }
: {
ok: false,
error: "failed to deliver one or more recorded steps",
};
});
pendingStopFlushes.set(message.requestId, flush);
void flush.then(sendResponse).finally(() => {
if (pendingStopFlushes.get(message.requestId) === flush) {
pendingStopFlushes.delete(message.requestId);
}
});
return true;
}
if (isRecordCancelMessage(message)) {
stepDeliveries.delete(message.requestId);
pendingStopFlushes.delete(message.requestId);
}
finishStop();
return false;
}
return false;
}
export { RECORD_CANCEL, RECORD_START, RECORD_STEP, RECORD_STOP };
@@ -0,0 +1,209 @@
import {
markRecordingDocument,
type RecordingDocumentMarker,
waitForDocumentElement,
} from "@/lib/recording/document-marker";
import {
isRecordFrameStartMessage,
RECORD_FRAME_PORT,
RECORD_FRAME_QUERY,
RECORD_FRAME_START,
type RecordFramePortMessage,
type RecordFrameQueryResponse,
type RecordFrameStartMessage,
} from "@/lib/recording/frame-bridge";
import { type RecordCaptureController, startRecordCapture } from "../record-capture";
import { RecordStepDelivery } from "../record-step-delivery";
interface ActiveFrameRecording {
requestId: string;
producerId: string;
port: chrome.runtime.Port;
delivery: RecordStepDelivery;
capture: RecordCaptureController;
marker: RecordingDocumentMarker;
}
const RECORD_FRAME_REGISTER_TIMEOUT_MS = 5_000;
export class RecordFrameAgent {
#active: ActiveFrameRecording | null = null;
#startPromise: Promise<{ ok: true } | { ok: false; error: string }> | null = null;
#disposed = false;
async start(
message: RecordFrameStartMessage,
): Promise<{ ok: true } | { ok: false; error: string }> {
if (this.#active?.requestId === message.requestId) return { ok: true };
if (this.#disposed) return { ok: false, error: "recording document is disposed" };
if (this.#startPromise) {
await this.#startPromise;
return this.start(message);
}
if (this.#active) this.cancel(this.#active.requestId);
const starting = this.#start(message);
this.#startPromise = starting;
try {
return await starting;
} finally {
if (this.#startPromise === starting) this.#startPromise = null;
}
}
cancel(requestId: string): void {
const active = this.#active;
if (!active || active.requestId !== requestId) return;
active.capture.dispose();
active.marker.restore();
active.port.disconnect();
this.#active = null;
}
dispose(): void {
this.#disposed = true;
if (this.#active) this.cancel(this.#active.requestId);
}
async #start(
message: RecordFrameStartMessage,
): Promise<{ ok: true } | { ok: false; error: string }> {
const producerId = crypto.randomUUID();
const root = await waitForDocumentElement();
if (this.#disposed) return { ok: false, error: "recording document is disposed" };
const marker = markRecordingDocument(producerId, root);
const port = chrome.runtime.connect({ name: RECORD_FRAME_PORT });
try {
await registerPort(port, message.requestId, producerId);
} catch {
marker.restore();
port.disconnect();
return { ok: false, error: "failed to register recording document" };
}
if (this.#disposed) {
marker.restore();
port.disconnect();
return { ok: false, error: "recording document is disposed" };
}
const delivery = new RecordStepDelivery(message.requestId, undefined, producerId);
const capture = startRecordCapture(
message.requestId,
(step) => {
marker.ensure();
delivery.enqueue(step);
},
{ captureNavigation: window.top === window },
);
const active: ActiveFrameRecording = {
requestId: message.requestId,
producerId,
port,
delivery,
capture,
marker,
};
this.#active = active;
port.onMessage.addListener((raw: unknown) => {
const command = raw as Partial<RecordFramePortMessage>;
if (command.type === "stop" && command.requestId === active.requestId) {
void this.#stop(active, command.commandId);
} else if (command.type === "cancel" && command.requestId === active.requestId) {
this.cancel(active.requestId);
}
});
port.onDisconnect.addListener(() => {
if (this.#active !== active) return;
active.capture.dispose();
active.marker.restore();
this.#active = null;
});
return { ok: true };
}
async #stop(active: ActiveFrameRecording, commandId: string | undefined): Promise<void> {
if (!commandId || this.#active !== active) return;
active.capture.dispose();
const succeeded = await active.delivery.flush();
try {
active.port.postMessage({
type: "stopped",
requestId: active.requestId,
commandId,
ok: succeeded,
...(succeeded ? {} : { error: "failed to deliver one or more recorded steps" }),
} satisfies RecordFramePortMessage);
} catch {
return;
}
if (!succeeded || this.#active !== active) return;
active.marker.restore();
this.#active = null;
active.port.disconnect();
}
}
function registerPort(
port: chrome.runtime.Port,
requestId: string,
producerId: string,
): Promise<void> {
return new Promise((resolve, reject) => {
const timer = setTimeout(() => {
cleanup();
reject(new Error("recording document registration timed out"));
}, RECORD_FRAME_REGISTER_TIMEOUT_MS);
const cleanup = () => {
clearTimeout(timer);
port.onMessage.removeListener(onMessage);
port.onDisconnect.removeListener(onDisconnect);
};
const onMessage = (raw: unknown) => {
const message = raw as Partial<RecordFramePortMessage>;
if (
message.type !== "ready_ack" ||
message.requestId !== requestId ||
message.producerId !== producerId
) {
return;
}
cleanup();
resolve();
};
const onDisconnect = () => {
cleanup();
reject(new Error("recording document port disconnected"));
};
port.onMessage.addListener(onMessage);
port.onDisconnect.addListener(onDisconnect);
port.postMessage({ type: "ready", requestId, producerId } satisfies RecordFramePortMessage);
});
}
export function attachRecordFrameAgent(): () => void {
const agent = new RecordFrameAgent();
const onMessage = (
message: unknown,
_sender: chrome.runtime.MessageSender,
sendResponse: (response: { ok: true } | { ok: false; error: string }) => void,
) => {
if (!isRecordFrameStartMessage(message)) return false;
void agent.start(message).then(sendResponse);
return true;
};
chrome.runtime.onMessage.addListener(onMessage);
void chrome.runtime
.sendMessage({ type: RECORD_FRAME_QUERY })
.then((response: RecordFrameQueryResponse | undefined) => {
if (!response?.active || !response.requestId || response.startedAtMs === undefined) return;
return agent.start({
type: RECORD_FRAME_START,
requestId: response.requestId,
startedAtMs: response.startedAtMs,
});
})
.catch(() => {});
return () => {
chrome.runtime.onMessage.removeListener(onMessage);
agent.dispose();
};
}
+25 -22
View File
@@ -21,6 +21,7 @@ import {
type OverlayMode,
} from "@/lib/overlay-bridge";
import { POPUP_PORT_NAME, type PopupInbound, type PopupOutbound } from "@/lib/popup-bridge";
import { recordFrameCoordinator } from "@/lib/recording/frame-coordinator";
import { attachSessionsLiveFlag } from "@/lib/sessions-live-flag";
import { createDisconnectCleanup } from "@/session-manager/disconnect-cleanup";
import { attachSessionEventHandler } from "@/session-manager/event-handler";
@@ -38,6 +39,7 @@ import {
attachRecordFinishListener,
attachRecordQueryListener,
attachRecordStepListener,
type RecordRuntimeDeps,
} from "@/tools/record";
import { detectBrowserMeta } from "@/transport/handshake";
import type { Transport } from "@/transport/transport";
@@ -145,10 +147,33 @@ export default defineBackground(() => {
void sessionsLive.syncFromManager();
},
});
const recordDeps = {
tabsApi: chrome.tabs,
cdp,
frameCoordinator: recordFrameCoordinator,
sendToTab: (tabId: number, msg: Parameters<typeof chrome.tabs.sendMessage>[1]) =>
chrome.tabs.sendMessage(tabId, msg),
bypassOverlay: async (tabId: number, enabled: boolean) => {
try {
await chrome.tabs.sendMessage(tabId, {
type: OVERLAY_AUTOMATION_BYPASS,
enabled,
});
} catch {
// Content script may be unavailable on restricted pages.
}
},
} satisfies RecordRuntimeDeps;
recordFrameCoordinator.attach();
attachRecordStepListener(recordDeps);
attachRecordFinishListener(recordDeps);
attachRecordQueryListener(recordDeps);
const dispatcher = new ToolDispatcher({
transport,
sessions,
cdp,
recording: recordDeps,
onSessionsChanged: onOverlaySessionStateChanged,
onBrowserControlResumed,
approveBorrow: (ctx) =>
@@ -170,28 +195,6 @@ export default defineBackground(() => {
}),
});
dispatcher.start();
const recordDeps = {
tabsApi: chrome.tabs,
cdp,
sendToTab: (tabId: number, msg: Parameters<typeof chrome.tabs.sendMessage>[1]) =>
chrome.tabs.sendMessage(tabId, msg),
bypassOverlay: async (tabId: number, enabled: boolean) => {
try {
await chrome.tabs.sendMessage(tabId, {
type: OVERLAY_AUTOMATION_BYPASS,
enabled,
});
} catch {
// Content script may be unavailable on restricted pages.
}
},
};
// Message listeners stay up (cheap; fire only for record message types).
// Tab / webNavigation observation attaches lazily while a recording is
// active — see ensureBrowserObservationListeners in tools/record.ts.
attachRecordStepListener(recordDeps);
attachRecordFinishListener(recordDeps);
attachRecordQueryListener(recordDeps);
if (typeof chrome.notifications?.onClicked?.addListener === "function") {
attachBorrowNotificationClickHandler({
onClicked: chrome.notifications.onClicked,
+34 -45
View File
@@ -11,11 +11,6 @@ import { createHelpRequestData } from "@/content/help-request";
import overlayCss from "@/content/overlay.css?inline";
import { OverlayController, shouldShowAgentControlOverlay } from "@/content/overlay-controller";
import { RecordOverlay } from "@/content/RecordOverlay";
import {
handleRecordContentMessage,
isRecordContentMessage,
type RecordCaptureController,
} from "@/content/record-capture";
import {
type CaptureSuppressAck,
type CaptureSuppressMessage,
@@ -45,11 +40,17 @@ import {
} from "@/lib/overlay-bridge";
import { sendInterrupt } from "@/lib/overlay-interrupt-client";
import {
isRecordCancelMessage,
isRecordStartMessage,
isRecordStopMessage,
RECORD_FINISH,
RECORD_QUERY,
type RecordCancelMessage,
type RecordQueryResponse,
type RecordStartAck,
type RecordStartMessage,
type RecordStopAck,
type RecordStopMessage,
} from "@/lib/record-bridge";
import type {
BorrowCancelMessage,
@@ -69,7 +70,6 @@ export default defineContentScript({
if (window.top !== window) return;
const overlays = new OverlayController();
let recordCapture: RecordCaptureController | null = null;
let activeRecordRequestId: string | null = null;
let reactRoot: ReactDOM.Root | null = null;
let overlayHost: HTMLElement | null = null;
@@ -228,8 +228,6 @@ export default defineContentScript({
if (previousHelp) {
void sendHelpFinish(previousHelp.id, "cancelled");
}
recordCapture?.dispose();
recordCapture = null;
activeRecordRequestId = null;
renderAll();
}
@@ -259,6 +257,9 @@ export default defineContentScript({
| HelpRequestMessage
| HelpCancelMessage
| CaptureSuppressMessage
| RecordStartMessage
| RecordStopMessage
| RecordCancelMessage
| OverlayAgentOverlayResetMessage
| OverlayAgentStateMessage
| OverlayAutomationBypassMessage,
@@ -269,41 +270,32 @@ export default defineContentScript({
return captureSuppress.handleMessage(message, sendResponse);
}
if (isRecordContentMessage(message)) {
const needsAsync = handleRecordContentMessage(
message,
{
activeRequestId: activeRecordRequestId,
capture: recordCapture,
setActiveRequestId: (id) => {
activeRecordRequestId = id;
},
setCapture: (capture) => {
recordCapture = capture;
},
onStart: (requestId, startedAtMs) => {
overlays.setAgentRecordRequest({
id: requestId,
...(typeof startedAtMs === "number" ? { startedAtMs } : {}),
onFinish: () => {
void chrome.runtime.sendMessage({
type: RECORD_FINISH,
requestId,
});
},
});
renderAll();
},
onStop: () => {
overlays.clearAgentRecordRequest(activeRecordRequestId ?? undefined);
renderAll();
},
if (isRecordStartMessage(message)) {
activeRecordRequestId = message.requestId;
overlays.setAgentRecordRequest({
id: message.requestId,
...(typeof message.startedAtMs === "number" ? { startedAtMs: message.startedAtMs } : {}),
onFinish: () => {
void chrome.runtime.sendMessage({
type: RECORD_FINISH,
requestId: message.requestId,
});
},
sendResponse as unknown as
| ((response: RecordStartAck | RecordStopAck) => void)
| undefined,
);
return needsAsync;
});
renderAll();
(sendResponse as unknown as (response: RecordStartAck) => void)({ ok: true });
return false;
}
if (isRecordStopMessage(message) || isRecordCancelMessage(message)) {
if (activeRecordRequestId !== message.requestId) return false;
overlays.clearAgentRecordRequest(message.requestId);
activeRecordRequestId = null;
renderAll();
if (isRecordStopMessage(message)) {
(sendResponse as unknown as (response: RecordStopAck) => void)({ ok: true });
}
return false;
}
if (
@@ -526,9 +518,6 @@ export default defineContentScript({
chrome.runtime.onMessage.removeListener(onMessage);
chrome.storage.onChanged.removeListener(onStorageChange);
window.removeEventListener("pageshow", onPageShow);
// Restore history hooks / remove capture listeners before the CS unloads.
recordCapture?.dispose();
recordCapture = null;
activeRecordRequestId = null;
});
},
@@ -247,6 +247,41 @@ describe("App", () => {
expect(copyButton.getAttribute("disabled")).not.toBeNull();
expect(screen.getByText("连接后可用")).toBeTruthy();
});
it("treats protocol drift as still connected and prompts an upgrade", async () => {
mockUseConnectionState.mockReturnValue({
snapshot: {
...baseSnapshot,
state: "version_skew",
instanceId: "03c3e47f",
handshake: {
server: "bh",
version: mockDaemonVersion,
protocol_version: "1.0",
},
},
statusState: "version_skew",
setLabel,
setConnectionEnabled,
});
render(<App />);
expect(screen.getByText("已连接")).toBeTruthy();
expect(screen.getByText("可升级")).toBeTruthy();
expect(screen.queryByText("协议不一致")).toBeNull();
expect(screen.queryByText("Action needed")).toBeNull();
expect(screen.queryByText("兼容")).toBeNull();
const warning = screen.getByText(/协议版本不同,请及时升级/);
expect(warning.textContent).toContain("CLI 协议");
expect(warning.textContent).toContain("扩展协议");
openRecordView();
const copyButton = screen.getByRole("button", { name: "复制录制指令" });
expect(copyButton.getAttribute("disabled")).toBeNull();
fireEvent.click(copyButton);
expect(navigator.clipboard.writeText).toHaveBeenCalledTimes(1);
});
});
describe("control hints toggle", () => {
+4 -3
View File
@@ -79,6 +79,7 @@ export function App() {
}, [copiedTick]);
const isSkewed = statusState === "version_skew";
const connectionLive = statusState === "connected" || isSkewed;
const daemonVersion = snapshot.handshake?.version ?? "—";
const daemonProtocol = snapshot.handshake?.protocol_version ?? "—";
const extensionVersion = snapshot.extensionVersion || "—";
@@ -90,7 +91,7 @@ export function App() {
setCopiedInstanceId(true);
};
const recordReady = statusState === "connected" && Boolean(snapshot.instanceId);
const recordReady = connectionLive && Boolean(snapshot.instanceId);
const recordPurpose = purposeDraft.trim();
const recordStartUrl = startUrlDraft.trim();
const recordCommand = snapshot.instanceId
@@ -206,12 +207,12 @@ export function App() {
</div>
{isSkewed && (
<p
className="mt-2 text-xs leading-snug text-amber-600 dark:text-amber-400"
className="mt-2 text-xs leading-snug text-muted-foreground"
data-slot="popup-version-skew-warning"
>
{t("popup.versionSkewWarning", {
extensionProtocol: PROTOCOL_VERSION,
daemonProtocol,
cliProtocol: daemonProtocol,
})}
</p>
)}
@@ -0,0 +1,14 @@
import { attachRecordFrameAgent } from "@/content/recording/frame-agent";
export default defineContentScript({
matches: ["<all_urls>"],
runAt: "document_start",
allFrames: true,
matchAboutBlank: true,
matchOriginAsFallback: true,
main(ctx) {
const dispose = attachRecordFrameAgent();
ctx.onInvalidated(dispose);
},
});
@@ -0,0 +1,213 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { RECORD_FRAME_PORT, RECORD_FRAME_QUERY } from "../recording/frame-bridge";
import { RecordFrameCoordinator } from "../recording/frame-coordinator";
class ListenerSet<T extends (...args: never[]) => unknown> {
readonly listeners = new Set<T>();
addListener = (listener: T) => this.listeners.add(listener);
removeListener = (listener: T) => this.listeners.delete(listener);
}
function fakePort(sender: chrome.runtime.MessageSender) {
const messages: unknown[] = [];
const onMessage = new ListenerSet<(message: unknown) => void>();
const onDisconnect = new ListenerSet<() => void>();
return {
port: {
name: RECORD_FRAME_PORT,
sender,
onMessage,
onDisconnect,
postMessage: vi.fn((message: unknown) => messages.push(message)),
disconnect: vi.fn(),
} as unknown as chrome.runtime.Port,
messages,
receive(message: unknown) {
for (const listener of onMessage.listeners) listener(message);
},
};
}
describe("RecordFrameCoordinator", () => {
const onConnect = new ListenerSet<(port: chrome.runtime.Port) => void>();
const onMessage = new ListenerSet<
(
message: unknown,
sender: chrome.runtime.MessageSender,
sendResponse: (response: unknown) => void,
) => boolean
>();
beforeEach(() => {
onConnect.listeners.clear();
onMessage.listeners.clear();
vi.stubGlobal("chrome", {
runtime: { onConnect, onMessage },
});
});
it("answers child-document queries for the initial tab before it is armed", () => {
const coordinator = new RecordFrameCoordinator({
getAllFrames: async () => [],
sendToDocument: async () => ({ ok: true }),
});
coordinator.attach();
coordinator.begin("rec-1", 10, 3);
const query = (tabId: number) => {
let response: unknown;
for (const listener of onMessage.listeners) {
listener(
{ type: RECORD_FRAME_QUERY },
{
tab: { id: tabId },
frameId: 7,
documentId: "child-document",
} as chrome.runtime.MessageSender,
(value) => {
response = value;
},
);
}
return response;
};
expect(query(3)).toEqual({ active: true, requestId: "rec-1", startedAtMs: 10 });
expect(query(4)).toEqual({ active: false });
});
it("starts a child Document that appears after the initial frame snapshot", async () => {
let notifyFrameNavigation:
| ((frame: {
tabId: number;
frameId: number;
documentId?: string;
lifecycle: "committed" | "completed";
}) => void)
| undefined;
const sendToDocument = vi.fn(async () => ({ ok: true }));
const coordinator = new RecordFrameCoordinator({
getAllFrames: async () => [{ frameId: 0, documentId: "top-document" }],
sendToDocument,
subscribeFrameNavigation(listener) {
notifyFrameNavigation = listener;
return () => {};
},
});
coordinator.attach();
const onDocumentReady = vi.fn();
coordinator.begin("rec-1", 10, 3, onDocumentReady);
await coordinator.armTab("rec-1", 3);
sendToDocument.mockClear();
notifyFrameNavigation?.({
tabId: 3,
frameId: 7,
documentId: "late-child",
lifecycle: "completed",
});
await vi.waitFor(() => {
expect(sendToDocument).toHaveBeenCalledWith(
3,
{ type: "bsk-record-frame-start", requestId: "rec-1", startedAtMs: 10 },
{ documentId: "late-child" },
);
});
expect(onDocumentReady).not.toHaveBeenCalled();
const child = fakePort({
tab: { id: 3 },
frameId: 7,
documentId: "late-child",
} as chrome.runtime.MessageSender);
for (const listener of onConnect.listeners) listener(child.port);
child.receive({ type: "ready", requestId: "rec-1", producerId: "producer-1" });
notifyFrameNavigation?.({
tabId: 3,
frameId: 7,
documentId: "late-child",
lifecycle: "completed",
});
await vi.waitFor(() => {
expect(onDocumentReady).toHaveBeenCalledWith({
tabId: 3,
documentId: "late-child",
browserFrameId: 7,
producerId: "producer-1",
});
});
});
it("binds a producer to its sender Document and keeps final steps valid while stopping", async () => {
const sendToDocument = vi.fn(async () => ({ ok: true }));
const coordinator = new RecordFrameCoordinator({
getAllFrames: async () => [
{ frameId: 0, documentId: "top-document" },
{ frameId: 7, documentId: "child-document" },
],
sendToDocument,
});
coordinator.attach();
coordinator.begin("rec-1", 10, 3);
await expect(coordinator.armTab("rec-1", 3)).resolves.toBe(true);
const sender = {
tab: { id: 3 },
frameId: 7,
documentId: "child-document",
} as chrome.runtime.MessageSender;
const frame = fakePort(sender);
for (const listener of onConnect.listeners) listener(frame.port);
frame.receive({ type: "ready", requestId: "rec-1", producerId: "producer-1" });
expect(coordinator.sourceFor("rec-1", "producer-1", sender)).toEqual({
tabId: 3,
documentId: "child-document",
browserFrameId: 7,
producerId: "producer-1",
});
const stopping = coordinator.stop("rec-1");
expect(coordinator.sourceFor("rec-1", "producer-1", sender)).not.toBeNull();
const stop = frame.messages.find(
(message): message is { type: "stop"; commandId: string } =>
typeof message === "object" &&
message !== null &&
(message as { type?: string }).type === "stop",
);
expect(stop).toBeDefined();
frame.receive({
type: "stopped",
requestId: "rec-1",
commandId: stop!.commandId,
ok: true,
});
await expect(stopping).resolves.toBe(true);
expect(coordinator.sourceFor("rec-1", "producer-1", sender)).toBeNull();
});
it("rejects a producer used from another Document", async () => {
const coordinator = new RecordFrameCoordinator({
getAllFrames: async () => [{ frameId: 0, documentId: "top-document" }],
sendToDocument: async () => ({ ok: true }),
});
coordinator.attach();
coordinator.begin("rec-1", 10, 3);
await coordinator.armTab("rec-1", 3);
const sender = {
tab: { id: 3 },
frameId: 0,
documentId: "top-document",
} as chrome.runtime.MessageSender;
const frame = fakePort(sender);
for (const listener of onConnect.listeners) listener(frame.port);
frame.receive({ type: "ready", requestId: "rec-1", producerId: "producer-1" });
expect(
coordinator.sourceFor("rec-1", "producer-1", {
...sender,
documentId: "other-document",
}),
).toBeNull();
});
});
@@ -0,0 +1,70 @@
import { describe, expect, it } from "vitest";
import {
RECORD_DOCUMENT_ATTRIBUTE,
recordingDocumentMarkerValue,
} from "@/shared/recording-document-identity";
import { markRecordingDocument } from "../recording/document-marker";
import { ObservationNodeIndex } from "../recording/observation-capture";
describe("recording document marker", () => {
it("rejects a non-random Document identity", () => {
expect(() => markRecordingDocument("user@example.com")).toThrow(
"recording Document identity must be a random UUID",
);
});
it("restores the Document attribute exactly", () => {
document.documentElement.setAttribute(RECORD_DOCUMENT_ATTRIBUTE, "page-value");
const producerId = "123e4567-e89b-42d3-a456-426614174000";
const marker = markRecordingDocument(producerId);
expect(document.documentElement.getAttribute(RECORD_DOCUMENT_ATTRIBUTE)).toBe(
recordingDocumentMarkerValue(producerId),
);
marker.restore();
expect(document.documentElement.getAttribute(RECORD_DOCUMENT_ATTRIBUTE)).toBe("page-value");
document.documentElement.removeAttribute(RECORD_DOCUMENT_ATTRIBUTE);
});
it("indexes the CDP scope from the safe frame identity", () => {
const index = new ObservationNodeIndex({
rootFrameId: "root",
frames: [
{
frameId: "child",
target: { tabId: 3, sessionId: "oopif-session" },
recordingDocumentId: "producer-1",
},
],
matchNodes: [],
refs: [],
});
expect(index.documentScope("producer-1")).toEqual({
frameId: "child",
target: { tabId: 3, sessionId: "oopif-session" },
});
});
it("fails closed when a recording identity maps to multiple Documents", () => {
const index = new ObservationNodeIndex({
rootFrameId: "root",
frames: [
{
frameId: "left",
target: { tabId: 3, sessionId: "left-session" },
recordingDocumentId: "duplicate",
},
{
frameId: "right",
target: { tabId: 3, sessionId: "right-session" },
recordingDocumentId: "duplicate",
},
],
matchNodes: [],
refs: [],
});
expect(index.documentScope("duplicate")).toBeUndefined();
});
});
@@ -2,20 +2,27 @@ import { afterEach, describe, expect, it, vi } from "vitest";
import type { CdpRunner, ChromeTabsApi } from "@/tools/shared";
const captureRecordingObservation = vi.hoisted(() => vi.fn());
const waitForDocumentSettled = vi.hoisted(() => vi.fn(async () => "quiet" as const));
vi.mock("../recording/observation-capture", async (importOriginal) => {
const actual = await importOriginal<typeof import("../recording/observation-capture")>();
return { ...actual, captureRecordingObservation };
});
vi.mock("../recording/document-settle", async (importOriginal) => {
const actual = await importOriginal<typeof import("../recording/document-settle")>();
return { ...actual, waitForDocumentSettled };
});
import { ObservationNodeIndex } from "../recording/observation-capture";
import { RecordingObservationRuntime } from "../recording/recording-runtime";
import type { RecordingDraftStep } from "../recording/types";
function observation() {
function observation(url = "https://example.com/") {
return {
rootFrameId: "root",
index: new ObservationNodeIndex({ rootFrameId: "root", matchNodes: [], refs: [] }),
url: "https://example.com/",
url,
title: "Example",
vomText: '@vom 1\nRootWebArea "Example"',
truncated: false,
@@ -30,7 +37,10 @@ function runtime(): RecordingObservationRuntime {
}
describe("RecordingObservationRuntime", () => {
afterEach(() => captureRecordingObservation.mockReset());
afterEach(() => {
captureRecordingObservation.mockReset();
waitForDocumentSettled.mockClear();
});
it("shares one initial capture and includes it in flush", async () => {
let release!: (value: ReturnType<typeof observation>) => void;
@@ -130,4 +140,150 @@ describe("RecordingObservationRuntime", () => {
expect(dumped).not.toContain("backendNodeId");
expect(dumped).not.toContain("ObservationNodeIndex");
});
it("captures a tab transition without sharing observation cursors", async () => {
captureRecordingObservation
.mockResolvedValueOnce(observation("https://example.com/first"))
.mockResolvedValueOnce(observation("https://example.com/second"));
const recording = runtime();
await recording.captureInitial(4);
const transition = await recording.captureTabTransition(4, 5);
await recording.captureInitial(4);
await recording.captureInitial(5);
expect(transition).toEqual({
preStateId: "s1",
postStateId: "s2",
targetUrl: "https://example.com/second",
});
expect(captureRecordingObservation.mock.calls.map(([input]) => input.tabId)).toEqual([4, 5]);
});
it("binds an iframe draft to its marked CDP Document scope", async () => {
captureRecordingObservation.mockResolvedValueOnce({
...observation(),
index: new ObservationNodeIndex({
rootFrameId: "root",
frames: [
{
frameId: "child",
target: { tabId: 7, sessionId: "oopif-session" },
recordingDocumentId: "producer-1",
},
],
matchNodes: [
{
backendNodeId: 42,
frameId: "child",
tag: "button",
rect: { x: 410, y: 20, w: 100, h: 30 },
localRect: { x: 10, y: 20, w: 100, h: 30 },
},
],
refs: [{ ref: "e1", backendNodeId: 42, frameId: "child", line: 1 }],
}),
});
const recording = runtime();
const drafts: RecordingDraftStep[] = [
{
op: "click" as const,
captureTarget: { tag: "button", role: "button", name: "Save" },
targetHint: {
geometry: { rect: { x: 10, y: 20, w: 100, h: 30 }, tag: "button" },
},
},
];
await recording.captureInitial(7);
await recording.processDraft(7, drafts, 0, "producer-1");
recording.cancel();
const draft = drafts[0];
expect(draft?.op).toBe("click");
if (!draft || draft.op !== "click") throw new Error("expected click draft");
expect(draft.targetHint).toMatchObject({
frameId: "child",
geometrySpace: "local",
});
expect(draft.matchedTarget?.ref).toBe("e1");
});
it("refreshes a late iframe before matching its first action", async () => {
const iframeObservation = {
...observation(),
index: new ObservationNodeIndex({
rootFrameId: "root",
frames: [
{
frameId: "child",
target: { tabId: 7, sessionId: "oopif-session" },
recordingDocumentId: "producer-1",
},
],
matchNodes: [
{
backendNodeId: 42,
frameId: "child",
tag: "button",
rect: { x: 410, y: 20, w: 100, h: 30 },
localRect: { x: 10, y: 20, w: 100, h: 30 },
},
],
refs: [
{
ref: "e1",
backendNodeId: 42,
frameId: "child",
role: "button",
name: "表格视图",
line: 1,
},
],
}),
};
captureRecordingObservation
.mockResolvedValueOnce(observation())
.mockResolvedValueOnce(iframeObservation)
.mockResolvedValueOnce(iframeObservation);
const recording = runtime();
await recording.captureInitial(7);
await recording.refreshDocument(7, "producer-1");
const drafts: RecordingDraftStep[] = [
{
op: "click",
captureTarget: { tag: "button", role: "button", name: "表格视图" },
targetHint: {
geometry: { rect: { x: 10, y: 20, w: 100, h: 30 }, tag: "button" },
},
},
];
await recording.processDraft(7, drafts, 0, "producer-1");
recording.cancel();
expect(captureRecordingObservation).toHaveBeenCalledTimes(3);
expect(waitForDocumentSettled).toHaveBeenCalledWith(
expect.anything(),
{ frameId: "child", target: { tabId: 7, sessionId: "oopif-session" } },
{ signal: expect.any(AbortSignal) },
);
const draft = drafts[0];
expect(draft?.op).toBe("click");
if (!draft || draft.op !== "click") throw new Error("expected click draft");
expect(draft.matchedTarget).toMatchObject({ ref: "e1", name: "表格视图" });
});
it("does not poison final flush when a frame readiness refresh fails", async () => {
captureRecordingObservation
.mockResolvedValueOnce(observation())
.mockRejectedValueOnce(new Error("child Document replaced"));
const recording = runtime();
await recording.captureInitial(7);
await expect(recording.refreshDocument(7, "producer-1")).rejects.toThrow(
"child Document replaced",
);
await expect(recording.flush()).resolves.toBeUndefined();
});
});
@@ -3,7 +3,7 @@ import { appendRecordedPayload, observeRecordedNavigation } from "../recording/s
describe("recording-step-buffer", () => {
it("stores semantic click without summary", () => {
const buffer = { steps: [], pendingNavigation: false };
const buffer = { steps: [], navigation: { pendingNavigation: false } };
appendRecordedPayload(buffer, {
op: "click",
target: { tag: "button", role: "button", name: "发布" },
@@ -15,11 +15,11 @@ describe("recording-step-buffer", () => {
captureTarget: { tag: "button", role: "button", name: "发布" },
},
]);
expect(buffer.pendingNavigation).toBe(true);
expect(buffer.navigation.pendingNavigation).toBe(true);
});
it("keeps the hovered element description as a capture fallback", () => {
const buffer = { steps: [], pendingNavigation: false };
const buffer = { steps: [], navigation: { pendingNavigation: false } };
appendRecordedPayload(
buffer,
{
@@ -59,9 +59,11 @@ describe("recording-step-buffer", () => {
captureTarget: { tag: "button", role: "button", name: "发布" },
},
],
currentUrl: "https://example.com/a",
pendingNavigation: true,
pendingNavigationDeadline: Date.now() + 5_000,
navigation: {
currentUrl: "https://example.com/a",
pendingNavigation: true,
pendingNavigationDeadline: Date.now() + 5_000,
},
};
observeRecordedNavigation(buffer, "https://example.com/b", true);
expect(buffer.steps).toEqual([
@@ -83,9 +85,11 @@ describe("recording-step-buffer", () => {
values: ["tech"],
},
],
currentUrl: "https://example.com/list",
pendingNavigation: true,
pendingNavigationDeadline: Date.now() + 5_000,
navigation: {
currentUrl: "https://example.com/list",
pendingNavigation: true,
pendingNavigationDeadline: Date.now() + 5_000,
},
};
observeRecordedNavigation(buffer, "https://example.com/list?cat=tech", true);
expect(buffer.steps).toEqual([
@@ -101,8 +105,7 @@ describe("recording-step-buffer", () => {
it("emits navigate for uncaused URL changes", () => {
const buffer = {
steps: [],
currentUrl: "https://example.com/a",
pendingNavigation: false,
navigation: { currentUrl: "https://example.com/a", pendingNavigation: false },
};
const result = observeRecordedNavigation(buffer, "https://example.com/b", false);
expect(result).toEqual({ kind: "appended", index: 0 });
@@ -115,8 +118,10 @@ describe("recording-step-buffer", () => {
it("asks the recorder to coalesce redirect hops instead of emitting each one", () => {
const buffer = {
steps: [],
currentUrl: "https://passport.example/login",
pendingNavigation: false,
navigation: {
currentUrl: "https://passport.example/login",
pendingNavigation: false,
},
};
const hop1 = observeRecordedNavigation(
buffer,
@@ -139,14 +144,16 @@ describe("recording-step-buffer", () => {
url: "https://app.example/dashboard",
});
expect(buffer.steps).toEqual([]);
expect(buffer.currentUrl).toBe("https://app.example/dashboard");
expect(buffer.navigation.currentUrl).toBe("https://app.example/dashboard");
});
it("does not reuse redirect metadata for a later content-observed URL change", () => {
const buffer = {
steps: [],
currentUrl: "https://example.com/redirect",
pendingNavigation: false,
navigation: {
currentUrl: "https://example.com/redirect",
pendingNavigation: false,
},
};
const result = observeRecordedNavigation(buffer, "https://example.com/spa");
@@ -0,0 +1,39 @@
import { describe, expect, it } from "vitest";
import { RecordingTabCoordinator } from "../recording/tab-coordinator";
describe("RecordingTabCoordinator", () => {
it("keeps navigation state isolated per tab", () => {
const tabs = new RecordingTabCoordinator(4, "https://example.com/first");
const first = tabs.navigation(4);
first.pendingNavigation = true;
const second = tabs.navigation(5, "https://example.com/second");
expect(second).toEqual({
currentUrl: "https://example.com/second",
pendingNavigation: false,
});
expect(tabs.navigation(4)).toBe(first);
expect(tabs.navigation(4).pendingNavigation).toBe(true);
});
it("invalidates an earlier activation when a newer tab becomes active", () => {
const tabs = new RecordingTabCoordinator(4);
const second = tabs.noteActivation(5);
const third = tabs.noteActivation(6);
expect(tabs.isLatest(second)).toBe(false);
expect(tabs.isLatest(third)).toBe(true);
tabs.commit(6);
expect(tabs.currentTabId).toBe(6);
});
it("commits the freshly observed URL when returning to an existing tab", () => {
const tabs = new RecordingTabCoordinator(4);
tabs.navigation(5, "https://example.com/old");
tabs.commit(5, "https://example.com/new");
expect(tabs.currentTabId).toBe(5);
expect(tabs.navigation(5).currentUrl).toBe("https://example.com/new");
});
});
@@ -69,6 +69,23 @@ describe("matchObservationTarget", () => {
expect(target.ref).toBe("e2");
});
it("matches iframe capture geometry in the child viewport coordinate space", () => {
const child = node(42, "child", 410);
child.localRect = { x: 10, y: 20, w: 100, h: 30 };
const target = matchObservationTarget({
observation: observation(
[child],
[{ ref: "e1", backendNodeId: 42, frameId: "child", line: 1 }],
),
hint: {
frameId: "child",
geometrySpace: "local",
geometry: { rect: { x: 10, y: 20, w: 100, h: 30 }, tag: "button" },
},
});
expect(target.ref).toBe("e1");
});
it("restricts a missing frame hint to the root frame", () => {
const target = matchObservationTarget({
observation: observation(
@@ -81,6 +98,21 @@ describe("matchObservationTarget", () => {
expect(target).toEqual({ name: "发布", unmatched: true });
});
it("fails closed when the source Document has no VOM frame mapping", () => {
const target = matchObservationTarget({
observation: observation(
[node(42, "root")],
[{ ref: "e1", backendNodeId: 42, role: "button", name: "发布", line: 1 }],
),
hint: {
frameId: null,
geometry: { rect: { x: 10, y: 20, w: 100, h: 30 }, tag: "button" },
},
fallback: { tag: "button", role: "button", name: "发布" },
});
expect(target).toEqual({ role: "button", name: "发布", unmatched: true });
});
it("returns unmatched for ambiguous geometry", () => {
const target = matchObservationTarget({
observation: observation(
@@ -27,6 +27,17 @@ describe("shouldRecordPress", () => {
});
describe("reduceTraceSteps", () => {
it("does not expose internal tab transitions in trace v2", () => {
const trace = reduceTraceSteps([
{
op: "switch_tab",
preStateId: "s1",
postStateId: "s2",
},
]);
expect(trace.steps).toEqual([]);
});
it("builds steps with pages dictionary and page id references", () => {
const drafts: RecordingDraftStep[] = [
{
@@ -160,7 +171,7 @@ describe("reduceTraceSteps", () => {
});
});
it("keeps hover steps before menu clicks", () => {
it("drops hover steps, which only exist in trace v3", () => {
const { steps } = reduceTraceSteps(
[
{
@@ -176,12 +187,7 @@ describe("reduceTraceSteps", () => {
],
"https://example.com/app",
);
expect(steps.map((s) => s.op)).toEqual(["hover", "click"]);
expect(steps[0]).toMatchObject({
op: "hover",
page: "p1",
target: { name: "Account" },
});
expect(steps.map((s) => s.op)).toEqual(["click"]);
});
it("resolveTraceStartUrl prefers explicit start URL", () => {
@@ -33,6 +33,49 @@ describe("trace reducer v3", () => {
expect(JSON.stringify(reduced.steps)).not.toContain("private-account-id");
});
it("emits tab transitions only for callers that advertised support", () => {
const drafts: RecordingDraftStep[] = [
{
op: "switch_tab",
preStateId: "s1",
postStateId: "s2",
},
];
expect(reduceTraceStepsV3(drafts).steps).toEqual([]);
expect(reduceTraceStepsV3(drafts, { includeTabSwitches: true }).steps).toEqual([
{ op: "switch_tab", id: 1, state: "s1", result: { state: "s2" } },
]);
});
it("keeps hover steps before menu clicks", () => {
const drafts: RecordingDraftStep[] = [
{
op: "hover",
captureTarget: { tag: "span", role: "button", name: "Account" },
preStateId: "s1",
postStateId: "s1",
},
{
op: "click",
captureTarget: { tag: "a", role: "link", name: "Profile" },
preStateId: "s1",
postStateId: "s1",
},
];
expect(reduceTraceStepsV3(drafts).steps.map((s) => s.op)).toEqual(["hover", "click"]);
});
it("does not export a tab transition without both observation endpoints", () => {
const sourceOnly: RecordingDraftStep = { op: "switch_tab", preStateId: "s1" };
const targetOnly: RecordingDraftStep = { op: "switch_tab", postStateId: "s2" };
expect(
reduceTraceStepsV3([sourceOnly, targetOnly], { includeTabSwitches: true }).steps,
).toEqual([]);
});
it("collapses redirect hops while retaining draft-to-step identity", () => {
const drafts: RecordingDraftStep[] = [
{ op: "navigate", url: "https://example.com/start", preStateId: "s1", postStateId: "s2" },
@@ -0,0 +1,39 @@
import {
RECORD_DOCUMENT_ATTRIBUTE,
recordingDocumentMarkerValue,
} from "@/shared/recording-document-identity";
export interface RecordingDocumentMarker {
restore(): void;
ensure(): void;
}
export function waitForDocumentElement(): Promise<HTMLElement> {
if (document.documentElement) return Promise.resolve(document.documentElement);
return new Promise((resolve) => {
const observer = new MutationObserver(() => {
if (!document.documentElement) return;
observer.disconnect();
resolve(document.documentElement);
});
observer.observe(document, { childList: true });
});
}
export function markRecordingDocument(
producerId: string,
root: HTMLElement = document.documentElement,
): RecordingDocumentMarker {
const previous = root.getAttribute(RECORD_DOCUMENT_ATTRIBUTE);
const markerValue = recordingDocumentMarkerValue(producerId);
const ensure = () => root.setAttribute(RECORD_DOCUMENT_ATTRIBUTE, markerValue);
ensure();
return {
ensure,
restore() {
if (root.getAttribute(RECORD_DOCUMENT_ATTRIBUTE) !== markerValue) return;
if (previous === null) root.removeAttribute(RECORD_DOCUMENT_ATTRIBUTE);
else root.setAttribute(RECORD_DOCUMENT_ATTRIBUTE, previous);
},
};
}
@@ -0,0 +1,65 @@
export const RECORD_FRAME_PORT = "bsk-record-frame";
export const RECORD_FRAME_QUERY = "bsk-record-frame-query";
export const RECORD_FRAME_START = "bsk-record-frame-start";
export interface RecordFrameQueryMessage {
type: typeof RECORD_FRAME_QUERY;
}
export interface RecordFrameQueryResponse {
active: boolean;
requestId?: string;
startedAtMs?: number;
}
export interface RecordFrameStartMessage {
type: typeof RECORD_FRAME_START;
requestId: string;
startedAtMs: number;
}
export type RecordFramePortMessage =
| {
type: "ready";
requestId: string;
producerId: string;
}
| {
type: "ready_ack";
requestId: string;
producerId: string;
}
| {
type: "stop";
requestId: string;
commandId: string;
}
| {
type: "cancel";
requestId: string;
}
| {
type: "stopped";
requestId: string;
commandId: string;
ok: boolean;
error?: string;
};
export function isRecordFrameQueryMessage(value: unknown): value is RecordFrameQueryMessage {
return (
typeof value === "object" &&
value !== null &&
(value as { type?: unknown }).type === RECORD_FRAME_QUERY
);
}
export function isRecordFrameStartMessage(value: unknown): value is RecordFrameStartMessage {
if (typeof value !== "object" || value === null) return false;
const message = value as Partial<RecordFrameStartMessage>;
return (
message.type === RECORD_FRAME_START &&
typeof message.requestId === "string" &&
typeof message.startedAtMs === "number"
);
}
@@ -0,0 +1,399 @@
import {
isRecordFrameQueryMessage,
RECORD_FRAME_PORT,
RECORD_FRAME_START,
type RecordFramePortMessage,
type RecordFrameQueryResponse,
type RecordFrameStartMessage,
} from "./frame-bridge";
export interface RecordingCaptureScope {
tabId: number;
documentId: string;
browserFrameId: number;
producerId: string;
}
interface ArmedRecording {
requestId: string;
startedAtMs: number;
tabIds: Set<number>;
agents: Map<string, FrameAgent>;
onDocumentReady?: (scope: RecordingCaptureScope) => void | Promise<void>;
finishing: boolean;
}
interface FrameAgent extends RecordingCaptureScope {
requestId: string;
port: chrome.runtime.Port;
stopWaiters: Map<string, (ok: boolean) => void>;
}
interface BrowserFrame {
frameId: number;
documentId?: string;
}
interface BrowserFrameNavigation extends BrowserFrame {
tabId: number;
lifecycle: "committed" | "completed";
}
const RECORD_FRAME_STOP_TIMEOUT_MS = 5_000;
export interface RecordFrameCoordinatorDeps {
getAllFrames(tabId: number): Promise<BrowserFrame[]>;
sendToDocument(
tabId: number,
message: RecordFrameStartMessage,
target: { documentId?: string; frameId?: number },
): Promise<unknown>;
subscribeFrameNavigation?(listener: (frame: BrowserFrameNavigation) => void): () => void;
}
function documentKey(tabId: number, documentId: string): string {
return `${tabId}:${documentId}`;
}
function senderAddress(
sender: chrome.runtime.MessageSender,
): { tabId: number; documentId: string; browserFrameId: number } | null {
const tabId = sender.tab?.id;
const documentId = sender.documentId;
const browserFrameId = sender.frameId;
if (
typeof tabId !== "number" ||
typeof documentId !== "string" ||
typeof browserFrameId !== "number"
) {
return null;
}
return { tabId, documentId, browserFrameId };
}
function defaultDeps(): RecordFrameCoordinatorDeps {
return {
async getAllFrames(tabId) {
return (await chrome.webNavigation.getAllFrames({ tabId })) ?? [];
},
sendToDocument(tabId, message, target) {
return chrome.tabs.sendMessage(tabId, message, target);
},
subscribeFrameNavigation(listener) {
const onCommitted = (details: chrome.webNavigation.WebNavigationTransitionCallbackDetails) =>
listener({ ...details, lifecycle: "committed" });
const onCompleted = (details: chrome.webNavigation.WebNavigationFramedCallbackDetails) =>
listener({ ...details, lifecycle: "completed" });
chrome.webNavigation.onCommitted.addListener(onCommitted);
chrome.webNavigation.onCompleted.addListener(onCompleted);
return () => {
chrome.webNavigation.onCommitted.removeListener(onCommitted);
chrome.webNavigation.onCompleted.removeListener(onCompleted);
};
},
};
}
export class RecordFrameCoordinator {
readonly #deps: RecordFrameCoordinatorDeps;
readonly #recordings = new Map<string, ArmedRecording>();
#attached = false;
#detachFrameNavigation: (() => void) | null = null;
constructor(deps: RecordFrameCoordinatorDeps = defaultDeps()) {
this.#deps = deps;
}
attach(): () => void {
if (this.#attached) return () => {};
this.#attached = true;
chrome.runtime.onConnect.addListener(this.#onConnect);
chrome.runtime.onMessage.addListener(this.#onMessage);
this.#detachFrameNavigation =
this.#deps.subscribeFrameNavigation?.(this.#onFrameNavigation) ?? null;
return () => {
if (!this.#attached) return;
this.#attached = false;
chrome.runtime.onConnect.removeListener(this.#onConnect);
chrome.runtime.onMessage.removeListener(this.#onMessage);
this.#detachFrameNavigation?.();
this.#detachFrameNavigation = null;
for (const recording of this.#recordings.values()) this.#cancelAgents(recording);
this.#recordings.clear();
};
}
begin(
requestId: string,
startedAtMs: number,
initialTabId: number,
onDocumentReady?: (scope: RecordingCaptureScope) => void | Promise<void>,
): void {
const previous = this.#recordings.get(requestId);
if (previous) this.#cancelAgents(previous);
this.#recordings.set(requestId, {
requestId,
startedAtMs,
// Register the initial tab before navigation begins. A newly-created
// child document can query at document_start before armTab's frame
// snapshot sees it, especially when Chromium promotes it to an OOPIF.
tabIds: new Set([initialTabId]),
agents: new Map(),
...(onDocumentReady ? { onDocumentReady } : {}),
finishing: false,
});
}
async armTab(requestId: string, tabId: number): Promise<boolean> {
const recording = this.#recordings.get(requestId);
if (!recording || recording.finishing) return false;
recording.tabIds.add(tabId);
let frames: BrowserFrame[];
try {
frames = await this.#deps.getAllFrames(tabId);
} catch {
frames = [{ frameId: 0 }];
}
if (!frames.some((frame) => frame.frameId === 0)) frames.unshift({ frameId: 0 });
const results = await Promise.all(
frames.map(async (frame) => {
const started = await this.#startFrame(recording, tabId, frame);
return { frameId: frame.frameId, started };
}),
);
return results.some((result) => result.frameId === 0 && result.started);
}
sourceFor(
requestId: string,
producerId: string,
sender: chrome.runtime.MessageSender,
): RecordingCaptureScope | null {
const address = senderAddress(sender);
const recording = this.#recordings.get(requestId);
if (!address || !recording) return null;
const agent = recording.agents.get(documentKey(address.tabId, address.documentId));
if (!agent || agent.producerId !== producerId) return null;
return {
tabId: agent.tabId,
documentId: agent.documentId,
browserFrameId: agent.browserFrameId,
producerId: agent.producerId,
};
}
async stop(requestId: string): Promise<boolean> {
const recording = this.#recordings.get(requestId);
if (!recording) return true;
recording.finishing = true;
const commandId = crypto.randomUUID();
const results = await Promise.all(
[...recording.agents.values()].map((agent) => this.#stopAgent(agent, requestId, commandId)),
);
const succeeded = results.every(Boolean);
if (succeeded) this.#recordings.delete(requestId);
else recording.finishing = false;
return succeeded;
}
cancel(requestId: string): void {
const recording = this.#recordings.get(requestId);
if (!recording) return;
this.#cancelAgents(recording);
this.#recordings.delete(requestId);
}
readonly #onMessage = (
message: unknown,
sender: chrome.runtime.MessageSender,
sendResponse: (response: RecordFrameQueryResponse) => void,
): boolean => {
if (!isRecordFrameQueryMessage(message)) return false;
const tabId = sender.tab?.id;
const recording =
typeof tabId === "number"
? [...this.#recordings.values()].find(
(candidate) => !candidate.finishing && candidate.tabIds.has(tabId),
)
: undefined;
sendResponse(
recording
? {
active: true,
requestId: recording.requestId,
startedAtMs: recording.startedAtMs,
}
: { active: false },
);
return false;
};
readonly #onFrameNavigation = (frame: BrowserFrameNavigation): void => {
if (frame.frameId === 0) return;
for (const recording of this.#recordings.values()) {
if (recording.finishing || !recording.tabIds.has(frame.tabId)) continue;
// onCommitted can race document_start, while onCompleted runs after the
// content script is available. Starting twice is safe and closes both
// the early-query and late-SPA-frame gaps.
void (async () => {
const started = await this.#startFrame(recording, frame.tabId, frame);
if (!started || frame.lifecycle !== "completed" || recording.finishing) return;
const agent = frame.documentId
? recording.agents.get(documentKey(frame.tabId, frame.documentId))
: [...recording.agents.values()].find(
(candidate) =>
candidate.tabId === frame.tabId && candidate.browserFrameId === frame.frameId,
);
if (!agent) return;
await recording.onDocumentReady?.({
tabId: agent.tabId,
documentId: agent.documentId,
browserFrameId: agent.browserFrameId,
producerId: agent.producerId,
});
})();
}
};
readonly #onConnect = (port: chrome.runtime.Port): void => {
if (port.name !== RECORD_FRAME_PORT) return;
const address = senderAddress(port.sender ?? {});
if (!address) {
port.disconnect();
return;
}
let registered: FrameAgent | null = null;
port.onMessage.addListener((raw: unknown) => {
const message = raw as Partial<RecordFramePortMessage>;
if (message.type === "ready") {
if (typeof message.requestId !== "string" || typeof message.producerId !== "string") {
port.disconnect();
return;
}
const requestId = message.requestId;
const producerId = message.producerId;
if (registered) {
if (registered.requestId === requestId && registered.producerId === producerId) {
port.postMessage({
type: "ready_ack",
requestId,
producerId,
} satisfies RecordFramePortMessage);
} else {
port.disconnect();
}
return;
}
const recording = this.#recordings.get(requestId);
if (
!recording ||
recording.finishing ||
!recording.tabIds.has(address.tabId) ||
producerId.length === 0
) {
port.disconnect();
return;
}
const agent: FrameAgent = {
...address,
requestId,
producerId,
port,
stopWaiters: new Map(),
};
registered = agent;
const key = documentKey(address.tabId, address.documentId);
const previous = recording.agents.get(key);
if (previous && previous !== agent) previous.port.disconnect();
recording.agents.set(key, agent);
port.postMessage({
type: "ready_ack",
requestId,
producerId,
} satisfies RecordFramePortMessage);
return;
}
if (
message.type === "stopped" &&
registered &&
message.requestId === registered.requestId &&
typeof message.commandId === "string"
) {
registered.stopWaiters.get(message.commandId)?.(message.ok === true);
registered.stopWaiters.delete(message.commandId);
}
});
port.onDisconnect.addListener(() => {
if (!registered) return;
for (const resolve of registered.stopWaiters.values()) resolve(true);
const recording = this.#recordings.get(registered.requestId);
const key = documentKey(registered.tabId, registered.documentId);
if (recording?.agents.get(key) === registered) recording.agents.delete(key);
});
};
#stopAgent(agent: FrameAgent, requestId: string, commandId: string): Promise<boolean> {
return new Promise((resolve) => {
const timer = setTimeout(() => {
agent.stopWaiters.delete(commandId);
resolve(false);
}, RECORD_FRAME_STOP_TIMEOUT_MS);
agent.stopWaiters.set(commandId, (ok) => {
clearTimeout(timer);
resolve(ok);
});
try {
agent.port.postMessage({
type: "stop",
requestId,
commandId,
} satisfies RecordFramePortMessage);
} catch {
clearTimeout(timer);
agent.stopWaiters.delete(commandId);
resolve(true);
}
});
}
async #startFrame(
recording: ArmedRecording,
tabId: number,
frame: BrowserFrame,
): Promise<boolean> {
const message: RecordFrameStartMessage = {
type: RECORD_FRAME_START,
requestId: recording.requestId,
startedAtMs: recording.startedAtMs,
};
try {
const response = await this.#deps.sendToDocument(tabId, message, {
...(frame.documentId ? { documentId: frame.documentId } : { frameId: frame.frameId }),
});
return isStarted(response);
} catch {
return false;
}
}
#cancelAgents(recording: ArmedRecording): void {
for (const agent of recording.agents.values()) {
try {
agent.port.postMessage({
type: "cancel",
requestId: recording.requestId,
} satisfies RecordFramePortMessage);
} catch {
// A destroyed Document has no remaining capture state to cancel.
}
}
}
}
function isStarted(value: unknown): boolean {
return typeof value === "object" && value !== null && (value as { ok?: unknown }).ok === true;
}
export const recordFrameCoordinator = new RecordFrameCoordinator();
@@ -1,4 +1,5 @@
import type { RenderedRef } from "@browser-skill/vom";
import type { CdpTarget } from "@/browser-driver/frame-graph";
import {
type CaptureVomMatchNode,
type CaptureVomObservationResult,
@@ -28,6 +29,11 @@ export interface RegisteredObservation {
url: string;
}
export interface RecordingDocumentScope {
frameId: string;
target: CdpTarget;
}
function nodeKey(frameId: string, backendNodeId: number): string {
return `${frameId}:${backendNodeId}`;
}
@@ -40,8 +46,12 @@ export class ObservationNodeIndex {
readonly #nodesByFrameTag = new Map<string, IndexedObservationNode[]>();
readonly #refById = new Map<string, RenderedRef>();
readonly #refsByFrame = new Map<string, RenderedRef[]>();
readonly #scopeByProducer = new Map<string, RecordingDocumentScope | null>();
constructor(input: Pick<CaptureVomObservationResult, "rootFrameId" | "matchNodes" | "refs">) {
constructor(
input: Pick<CaptureVomObservationResult, "rootFrameId" | "matchNodes" | "refs"> &
Partial<Pick<CaptureVomObservationResult, "frames">>,
) {
const refByNode = new Map<string, RenderedRef>();
for (const ref of input.refs) {
const frameId = ref.frameId ?? input.rootFrameId;
@@ -51,6 +61,16 @@ export class ObservationNodeIndex {
frameRefs.push(ref);
this.#refsByFrame.set(frameId, frameRefs);
}
for (const frame of input.frames ?? []) {
const documentId = frame.recordingDocumentId;
if (!documentId) continue;
this.#scopeByProducer.set(
documentId,
this.#scopeByProducer.has(documentId)
? null
: { frameId: frame.frameId, target: frame.target },
);
}
for (const geometry of input.matchNodes) {
const { frameId } = geometry;
const entry = {
@@ -76,6 +96,10 @@ export class ObservationNodeIndex {
refs(frameId: string): readonly RenderedRef[] {
return this.#refsByFrame.get(frameId) ?? [];
}
documentScope(producerId: string): RecordingDocumentScope | undefined {
return this.#scopeByProducer.get(producerId) ?? undefined;
}
}
async function readTabMeta(
@@ -28,7 +28,13 @@ function abortableDelay(ms: number, signal?: AbortSignal): Promise<void> {
}
function isTargeted(draft: RecordingDraftStep): draft is TargetedRecordingDraft {
return draft.op !== "navigate" && draft.op !== "scroll";
return (
draft.op === "click" ||
draft.op === "hover" ||
draft.op === "fill" ||
draft.op === "press" ||
draft.op === "select"
);
}
export class RecordingObservationSession {
@@ -1,6 +1,7 @@
import type { CdpRunner, ChromeTabsApi } from "@/tools/shared";
import type { StopReason, TraceV3 } from "@/transport/types";
import type { DocumentSettleScope } from "./document-settle";
import { type DocumentSettleScope, waitForDocumentSettled } from "./document-settle";
import type { RegisteredObservation } from "./observation-capture";
import { RecordingObservationSession } from "./observation-session";
import { inferMissingPostStates, SettleController } from "./settle-controller";
import { RecordingStateRegistry } from "./state-registry";
@@ -10,11 +11,13 @@ import type { RecordingDraftStep, StepAnnotation } from "./types";
interface TabRecordingContext {
session: RecordingObservationSession;
settle: SettleController;
initialCapture: PendingInitialCapture | null;
pendingCapture: PendingCapture | null;
frameRefresh: Promise<void>;
frameRefreshAbort: AbortController | null;
}
interface PendingInitialCapture {
promise: Promise<void>;
interface PendingCapture {
promise: Promise<RegisteredObservation>;
abort: AbortController;
}
@@ -56,37 +59,113 @@ export class RecordingObservationRuntime {
tabsApi: this.#tabsApi,
tabId,
}),
initialCapture: null,
pendingCapture: null,
frameRefresh: Promise.resolve(),
frameRefreshAbort: null,
};
this.#contexts.set(tabId, context);
return context;
}
async captureInitial(tabId: number): Promise<void> {
async #capture(tabId: number): Promise<RegisteredObservation> {
const context = this.#context(tabId);
if (context.session.cursor.lastSettled) return;
if (context.initialCapture) return context.initialCapture.promise;
if (context.pendingCapture) return context.pendingCapture.promise;
const abort = new AbortController();
let pending!: PendingInitialCapture;
let pending!: PendingCapture;
const promise = context.session.capture(this.#cdp, this.#tabsApi, tabId, abort.signal);
pending = {
abort,
promise: promise
.then(() => {})
.finally(() => {
if (context.initialCapture === pending) context.initialCapture = null;
}),
promise: promise.finally(() => {
if (context.pendingCapture === pending) context.pendingCapture = null;
}),
};
context.initialCapture = pending;
context.pendingCapture = pending;
return pending.promise;
}
async captureInitial(tabId: number): Promise<void> {
const context = this.#context(tabId);
if (context.session.cursor.lastSettled) return;
await this.#capture(tabId);
}
/** Refresh the safe observation after a late child Document becomes usable. */
async refreshDocument(tabId: number, producerId: string): Promise<void> {
const context = this.#context(tabId);
context.frameRefreshAbort?.abort();
const abort = new AbortController();
context.frameRefreshAbort = abort;
const previous = context.frameRefresh;
const refresh = (async () => {
await previous;
if (abort.signal.aborted) return;
if (context.pendingCapture) await Promise.allSettled([context.pendingCapture.promise]);
if (abort.signal.aborted) return;
// The first capture discovers the safe producer -> CDP Document scope.
await this.#capture(tabId);
if (abort.signal.aborted) return;
const scope = context.session.cursor.lastSettled?.index.documentScope(producerId);
if (scope) {
await waitForDocumentSettled(this.#cdp, scope, { signal: abort.signal });
}
if (abort.signal.aborted) return;
// Capture again after the child SPA is quiet so the next action can be
// matched against refs that actually exist in its pre-action state.
await this.#capture(tabId);
})();
const tracked = refresh.catch(() => {});
context.frameRefresh = tracked;
try {
await refresh;
} finally {
if (context.frameRefresh === tracked) context.frameRefreshAbort = null;
}
}
async captureTabTransition(
fromTabId: number,
toTabId: number,
): Promise<{ preStateId?: string; postStateId?: string; targetUrl?: string }> {
const from = this.#context(fromTabId);
await this.#flushContext(from);
if (!from.session.cursor.lastSettled) {
try {
await this.captureInitial(fromTabId);
} catch {
// A target-tab observation can still preserve the forward transition.
}
}
const to = this.#context(toTabId);
let target = to.session.cursor.lastSettled;
try {
target = await this.#capture(toTabId);
} catch {
// Reuse the last settled target state when a refresh races navigation.
}
const source = from.session.cursor.lastSettled;
return {
...(source ? { preStateId: source.stateId } : {}),
...(target ? { postStateId: target.stateId, targetUrl: target.url } : {}),
};
}
bindTabTransition(
draft: Extract<RecordingDraftStep, { op: "switch_tab" }>,
draftId: number,
): void {
if (draft.preStateId) this.#registry.markStep(draft.preStateId, draftId);
}
async processDraft(
tabId: number,
drafts: RecordingDraftStep[],
draftIndex: number,
scope?: DocumentSettleScope,
producerId?: string,
): Promise<void> {
const draft = drafts[draftIndex];
if (!draft) return;
@@ -98,6 +177,23 @@ export class RecordingObservationRuntime {
// Post-action settle can still provide a usable state.
}
}
const scope = producerId
? context.session.cursor.lastSettled?.index.documentScope(producerId)
: undefined;
if (
producerId &&
(draft.op === "click" ||
draft.op === "hover" ||
draft.op === "fill" ||
draft.op === "press" ||
draft.op === "select")
) {
draft.targetHint = {
...(draft.targetHint ?? {}),
frameId: scope?.frameId ?? null,
geometrySpace: "local",
};
}
context.session.bindDraft(draft, draftIndex + 1, context.settle.hasPending);
context.settle.schedule(drafts, draftIndex, scope);
}
@@ -119,23 +215,24 @@ export class RecordingObservationRuntime {
this.#context(tabId).settle.scheduleRedirect(drafts, url);
}
async flushRedirects(): Promise<void> {
for (const context of this.#contexts.values()) await context.settle.flushRedirects();
async #flushContext(context: TabRecordingContext): Promise<void> {
await context.frameRefresh;
if (context.pendingCapture) await Promise.allSettled([context.pendingCapture.promise]);
await context.settle.flushRedirects();
await context.settle.flush();
}
async flushRedirects(tabId: number): Promise<void> {
await this.#contexts.get(tabId)?.settle.flushRedirects();
}
async flush(): Promise<void> {
await Promise.allSettled(
[...this.#contexts.values()].flatMap((context) =>
context.initialCapture ? [context.initialCapture.promise] : [],
),
);
await this.flushRedirects();
for (const context of this.#contexts.values()) await context.settle.flush();
for (const context of this.#contexts.values()) await this.#flushContext(context);
}
async settleTrailing(tabId: number, drafts: RecordingDraftStep[]): Promise<void> {
const context = this.#context(tabId);
if (context.initialCapture) await Promise.allSettled([context.initialCapture.promise]);
if (context.pendingCapture) await Promise.allSettled([context.pendingCapture.promise]);
if (!context.session.cursor.lastSettled) {
try {
await this.captureInitial(tabId);
@@ -149,7 +246,8 @@ export class RecordingObservationRuntime {
cancel(): void {
for (const context of this.#contexts.values()) {
context.initialCapture?.abort.abort();
context.frameRefreshAbort?.abort();
context.pendingCapture?.abort.abort();
context.settle.cancel();
}
}
@@ -161,6 +259,7 @@ export class RecordingObservationRuntime {
startUrl?: string;
stoppedBy: StopReason;
bskVersion: string;
includeTabSwitches?: boolean;
}): TraceV3 {
return buildTraceV3({
registry: this.#registry,
@@ -172,6 +271,7 @@ export class RecordingObservationRuntime {
stoppedBy: input.stoppedBy,
bskVersion: input.bskVersion,
redactValues: this.#redactValues,
includeTabSwitches: input.includeTabSwitches,
});
}
}
+17 -12
View File
@@ -4,6 +4,10 @@ import type { RecordingDraftStep, TargetMatchHint } from "./types";
export interface RecordingStepBuffer {
steps: RecordingDraftStep[];
navigation: RecordingNavigationCursor;
}
export interface RecordingNavigationCursor {
currentUrl?: string;
pendingNavigation: boolean;
pendingNavigationDeadline?: number;
@@ -87,22 +91,23 @@ export function observeRecordedNavigation(
transitionType?: string,
transitionQualifiers?: string[],
): NavigationObserveResult {
if (!url || url === buffer.currentUrl) return { kind: "noop" };
buffer.currentUrl = url;
const navigation = buffer.navigation;
if (!url || url === navigation.currentUrl) return { kind: "noop" };
navigation.currentUrl = url;
const pendingIsCurrent =
buffer.pendingNavigation &&
(buffer.pendingNavigationDeadline === undefined ||
buffer.pendingNavigationDeadline >= Date.now());
navigation.pendingNavigation &&
(navigation.pendingNavigationDeadline === undefined ||
navigation.pendingNavigationDeadline >= Date.now());
if (causedByAction === true || (causedByAction === undefined && pendingIsCurrent)) {
buffer.pendingNavigation = false;
buffer.pendingNavigationDeadline = undefined;
navigation.pendingNavigation = false;
navigation.pendingNavigationDeadline = undefined;
const annotatedIndex = annotateLastStepNavigation(buffer, url);
if (annotatedIndex >= 0) return { kind: "annotated", index: annotatedIndex };
} else {
buffer.pendingNavigation = false;
buffer.pendingNavigationDeadline = undefined;
navigation.pendingNavigation = false;
navigation.pendingNavigationDeadline = undefined;
}
if (hasRedirectQualifier(transitionQualifiers)) {
@@ -135,14 +140,14 @@ export function appendRecordedPayload(
return result.kind === "appended" ? result.index : null;
}
const step = toDraftStep(
{ ...payload, page_url: payload.page_url ?? buffer.currentUrl },
{ ...payload, page_url: payload.page_url ?? buffer.navigation.currentUrl },
targetHint,
);
if (!step) return null;
buffer.steps.push(step);
if (step.op === "click" || step.op === "press" || step.op === "select" || step.op === "fill") {
buffer.pendingNavigation = payload.expects_navigation === true;
buffer.pendingNavigationDeadline = buffer.pendingNavigation
buffer.navigation.pendingNavigation = payload.expects_navigation === true;
buffer.navigation.pendingNavigationDeadline = buffer.navigation.pendingNavigation
? Date.now() + NAVIGATION_TRIGGER_WINDOW_MS
: undefined;
}
@@ -0,0 +1,61 @@
import type { RecordingNavigationCursor } from "./step-buffer";
export interface TabActivation {
tabId: number;
revision: number;
}
export class RecordingTabCoordinator {
readonly #navigationByTab = new Map<number, RecordingNavigationCursor>();
#activeTabId: number;
#currentTabId: number;
#activationRevision = 0;
constructor(initialTabId: number, initialUrl?: string) {
this.#activeTabId = initialTabId;
this.#currentTabId = initialTabId;
this.#navigationByTab.set(initialTabId, {
currentUrl: initialUrl,
pendingNavigation: false,
});
}
get activeTabId(): number {
return this.#activeTabId;
}
get currentTabId(): number {
return this.#currentTabId;
}
noteActivation(tabId: number): TabActivation {
this.#activeTabId = tabId;
this.#activationRevision += 1;
return { tabId, revision: this.#activationRevision };
}
isLatest(activation: TabActivation): boolean {
return (
activation.tabId === this.#activeTabId && activation.revision === this.#activationRevision
);
}
commit(tabId: number, currentUrl?: string): void {
if (currentUrl !== undefined) this.navigation(tabId).currentUrl = currentUrl;
this.#currentTabId = tabId;
}
navigation(tabId: number, fallbackUrl?: string): RecordingNavigationCursor {
const existing = this.#navigationByTab.get(tabId);
if (existing) {
if (!existing.currentUrl && fallbackUrl) existing.currentUrl = fallbackUrl;
return existing;
}
const created: RecordingNavigationCursor = {
currentUrl: fallbackUrl,
pendingNavigation: false,
};
this.#navigationByTab.set(tabId, created);
return created;
}
}
@@ -14,8 +14,13 @@ function rectMatches(a: TargetGeometry["rect"], b: TargetGeometry["rect"]): bool
return close(a.x, b.x) && close(a.y, b.y) && close(a.w, b.w) && close(a.h, b.h);
}
function candidateMatches(candidate: IndexedObservationNode, geometry: TargetGeometry): boolean {
return candidate.geometry.rect !== null && rectMatches(geometry.rect, candidate.geometry.rect);
function candidateMatches(
candidate: IndexedObservationNode,
geometry: TargetGeometry,
geometrySpace: "top" | "local",
): boolean {
const rect = geometrySpace === "local" ? candidate.geometry.localRect : candidate.geometry.rect;
return rect != null && rectMatches(geometry.rect, rect);
}
export function unmatchedTarget(fallback?: CaptureTargetDescriptor): TargetDescriptorV3 {
@@ -53,12 +58,17 @@ export function matchObservationTarget(input: {
hint?: TargetMatchHint;
fallback?: CaptureTargetDescriptor;
}): TargetDescriptorV3 {
if (input.hint?.frameId === null) return unmatchedTarget(input.fallback);
const frameId = input.hint?.frameId ?? input.observation.rootFrameId;
const geometry = input.hint?.geometry;
if (geometry) {
const matches = input.observation.index
.candidates(frameId, geometry.tag)
.filter((candidate) => candidate.ref && candidateMatches(candidate, geometry));
.filter(
(candidate) =>
candidate.ref &&
candidateMatches(candidate, geometry, input.hint?.geometrySpace ?? "top"),
);
if (matches.length === 1) return descriptor(matches[0]!.ref!);
const semanticMatches = matches.filter(
(candidate) => candidate.ref && matchesSemantics(candidate.ref, input.fallback),
@@ -40,8 +40,12 @@ export function buildTraceV3(input: {
stoppedBy: StopReason;
bskVersion: string;
redactValues?: boolean;
includeTabSwitches?: boolean;
}): TraceV3 {
const reduced = reduceTraceStepsV3(input.drafts, { redactValues: input.redactValues });
const reduced = reduceTraceStepsV3(input.drafts, {
includeTabSwitches: input.includeTabSwitches,
redactValues: input.redactValues,
});
const entries = publishedEntries(input.registry, reduced.steps);
const publishedId = new Map(entries.map((entry, index) => [entry.id, `s${index + 1}`]));
const annotationsByState = new Map<string, StepAnnotation[]>();
@@ -119,14 +119,6 @@ function toV2Step(
},
effectForNavigation(step.navigatedTo, urlToId),
);
case "hover":
if (!step.captureTarget) return null;
return {
op: "hover",
id,
page,
target: step.captureTarget,
};
case "fill":
if (!step.captureTarget) return null;
return {
@@ -161,7 +153,11 @@ function toV2Step(
},
effectForNavigation(step.navigatedTo, urlToId),
);
// `hover` only exists in Trace v3. Peers that negotiate v2 predate the
// step variant and fail to decode the whole result if we emit it.
case "hover":
case "scroll":
case "switch_tab":
return null;
}
}
@@ -58,8 +58,26 @@ function selection(values: string[], labels?: string[]): Array<{ value: string;
}));
}
function reduceDraft(draft: RecordingDraftStep, id: number, redactValues: boolean): StepV3 | null {
interface ReduceDraftOptions {
includeTabSwitches: boolean;
redactValues: boolean;
}
function reduceDraft(
draft: RecordingDraftStep,
id: number,
options: ReduceDraftOptions,
): StepV3 | null {
if (!shouldIncludeDraft(draft)) return null;
if (draft.op === "switch_tab") {
if (!options.includeTabSwitches || !draft.preStateId || !draft.postStateId) return null;
return {
op: "switch_tab",
id,
state: draft.preStateId,
result: { state: draft.postStateId },
};
}
const state = draft.preStateId ?? draft.postStateId;
const resultState = draft.postStateId ?? draft.preStateId;
if (!state || !resultState) return null;
@@ -81,7 +99,7 @@ function reduceDraft(draft: RecordingDraftStep, id: number, redactValues: boolea
target: draft.matchedTarget ?? unmatchedTarget(draft.captureTarget),
};
case "fill":
const fillIsRedacted = redactValues || draft.redacted === true;
const fillIsRedacted = options.redactValues || draft.redacted === true;
return {
op: "fill",
...common,
@@ -105,7 +123,7 @@ function reduceDraft(draft: RecordingDraftStep, id: number, redactValues: boolea
op: "select",
...common,
target: draft.matchedTarget ?? unmatchedTarget(draft.captureTarget),
...(!redactValues ? { selection: selection(draft.values, draft.labels) } : {}),
...(!options.redactValues ? { selection: selection(draft.values, draft.labels) } : {}),
};
case "scroll":
return { op: "scroll", ...common };
@@ -119,12 +137,15 @@ export interface ReducedTraceV3 {
export function reduceTraceStepsV3(
steps: RecordingDraftStep[],
options: { redactValues?: boolean } = {},
options: { includeTabSwitches?: boolean; redactValues?: boolean } = {},
): ReducedTraceV3 {
const output: StepV3[] = [];
const stepIdByDraftId = new Map<number, number>();
for (const { draft, draftIds } of collapseRedirects(steps)) {
const step = reduceDraft(draft, output.length + 1, options.redactValues ?? false);
const step = reduceDraft(draft, output.length + 1, {
includeTabSwitches: options.includeTabSwitches === true,
redactValues: options.redactValues === true,
});
if (!step) continue;
output.push(step);
for (const draftId of draftIds) stepIdByDraftId.set(draftId, step.id);
+4 -2
View File
@@ -15,8 +15,9 @@ export interface TargetGeometry {
export interface TargetMatchHint {
geometry?: TargetGeometry;
/** Missing means the current top frame, never an unrestricted frame search. */
frameId?: string;
/** Missing means top frame; null means the source Document could not be resolved. */
frameId?: string | null;
geometrySpace?: "top" | "local";
}
export interface StepAnnotation {
@@ -69,6 +70,7 @@ export type RecordingDraftStep =
DraftTarget &
DraftNavigationEffect)
| ({ op: "scroll" } & DraftStateLink)
| ({ op: "switch_tab" } & DraftStateLink)
| ({
op: "navigate";
url: string;
@@ -13,6 +13,8 @@ describe("RefStore", () => {
backendNodeId: 42,
tabId: 7,
generation: 0,
kind: "dom",
capabilities: ["interact", "screenshot"],
});
expect(s.size()).toBe(1);
expect(s.isEmpty()).toBe(false);
@@ -48,4 +50,31 @@ describe("RefStore", () => {
cdpSessionId: "child-session",
});
});
it("preserves explicit capabilities for visual surface refs", () => {
const s = new RefStore();
s.set("e1", 42, {
tabId: 7,
kind: "surface",
capabilities: ["screenshot"],
});
expect(s.resolveEntry("e1")).toMatchObject({
kind: "surface",
capabilities: ["screenshot"],
});
});
it("never grants ordinary interaction to a surface through defaults or caller input", () => {
const s = new RefStore();
s.set("e1", 42, { tabId: 7, kind: "surface" });
s.set("e2", 43, {
tabId: 7,
kind: "surface",
capabilities: ["interact", "screenshot"],
});
expect(s.resolveEntry("e1")?.capabilities).toEqual(["screenshot"]);
expect(s.resolveEntry("e2")?.capabilities).toEqual(["screenshot"]);
});
});
@@ -1,3 +1,5 @@
import type { Rect } from "@browser-skill/vom";
/**
* Per-session map from `@e<N>` snapshot refs to a CDP node address.
*
@@ -12,12 +14,17 @@
* inside the `SessionContext`.
*/
export type BackendNodeId = number;
export type RefCapability = "interact" | "screenshot";
export type RefTargetKind = "dom" | "surface";
export interface RefEntry {
backendNodeId: BackendNodeId;
tabId: number | null;
frameId?: string;
cdpSessionId?: string;
visibleRect?: Rect;
kind: RefTargetKind;
capabilities: RefCapability[];
generation: number;
}
@@ -28,8 +35,19 @@ export type RefInput =
tabId: number;
frameId?: string;
cdpSessionId?: string;
visibleRect?: Rect;
kind?: RefTargetKind;
capabilities?: RefCapability[];
};
function capabilitiesFor(
kind: RefTargetKind,
capabilities: RefCapability[] | undefined,
): RefCapability[] {
if (kind === "surface") return ["screenshot"];
return capabilities ?? ["interact", "screenshot"];
}
export class RefStore {
private readonly map = new Map<string, RefEntry>();
private generation = 0;
@@ -70,13 +88,20 @@ export class RefStore {
tabId?: number;
frameId?: string;
cdpSessionId?: string;
visibleRect?: Rect;
kind?: RefTargetKind;
capabilities?: RefCapability[];
} = {},
): void {
const kind = opts.kind ?? "dom";
this.map.set(normaliseRef(ref), {
backendNodeId: id,
tabId: opts.tabId ?? null,
...(opts.frameId ? { frameId: opts.frameId } : {}),
...(opts.cdpSessionId ? { cdpSessionId: opts.cdpSessionId } : {}),
...(opts.visibleRect ? { visibleRect: opts.visibleRect } : {}),
kind,
capabilities: capabilitiesFor(kind, opts.capabilities),
generation: this.generation,
});
}
@@ -94,14 +119,20 @@ export class RefStore {
return {
backendNodeId: input,
tabId: null,
kind: "dom",
capabilities: ["interact", "screenshot"],
generation: this.generation,
};
}
const kind = input.kind ?? "dom";
return {
backendNodeId: input.backendNodeId,
tabId: input.tabId,
...(input.frameId ? { frameId: input.frameId } : {}),
...(input.cdpSessionId ? { cdpSessionId: input.cdpSessionId } : {}),
...(input.visibleRect ? { visibleRect: input.visibleRect } : {}),
kind,
capabilities: capabilitiesFor(kind, input.capabilities),
generation: this.generation,
};
}
@@ -0,0 +1,22 @@
export const RECORD_DOCUMENT_ATTRIBUTE = "data-bsk-record-document";
const RECORDING_DOCUMENT_ID_PATTERN =
/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i;
const RECORDING_DOCUMENT_MARKER_PREFIX = "bsk:";
export function recordingDocumentMarkerValue(identity: string): string {
if (!RECORDING_DOCUMENT_ID_PATTERN.test(identity)) {
throw new TypeError("recording Document identity must be a random UUID");
}
return `${RECORDING_DOCUMENT_MARKER_PREFIX}${identity}`;
}
/** Read BrowserSkill's opaque per-Document recording identity from captured attributes. */
export function readRecordingDocumentIdentity(
attrs: Readonly<Record<string, string>>,
): string | undefined {
const marker = attrs[RECORD_DOCUMENT_ATTRIBUTE];
if (!marker?.startsWith(RECORDING_DOCUMENT_MARKER_PREFIX)) return undefined;
const identity = marker.slice(RECORDING_DOCUMENT_MARKER_PREFIX.length);
return identity && RECORDING_DOCUMENT_ID_PATTERN.test(identity) ? identity : undefined;
}
@@ -127,6 +127,38 @@ describe("requestBorrowConfirmation", () => {
expect(message.requestId.length).toBeGreaterThan(0);
});
it("proactively focuses the user window before sending the overlay message", async () => {
tabs.get.mockResolvedValueOnce({ id: 42, title: "Tab" });
windows.getLastFocused.mockResolvedValueOnce(
userWindowWithActiveTab({ windowId: 77, tabId: 7, url: "https://app.example/" }),
);
tabs.sendMessage.mockResolvedValueOnce({ type: "borrow-response", allowed: true });
const pending = requestBorrowConfirmation(42, {
deps: { tabs, windows, notifications },
});
await vi.runAllTimersAsync();
await pending;
expect(windows.update).toHaveBeenCalledWith(77, { focused: true });
});
it("still proceeds when proactive window focus fails", async () => {
tabs.get.mockResolvedValueOnce({ id: 42, title: "Tab" });
windows.getLastFocused.mockResolvedValueOnce(
userWindowWithActiveTab({ windowId: 77, tabId: 7, url: "https://app.example/" }),
);
windows.update.mockRejectedValueOnce(new Error("window gone"));
tabs.sendMessage.mockResolvedValueOnce({ type: "borrow-response", allowed: true });
const pending = requestBorrowConfirmation(42, {
deps: { tabs, windows, notifications },
});
await vi.runAllTimersAsync();
expect(await pending).toBe(true);
});
it("sends confirmation to the borrow target when it is the active tab", async () => {
tabs.get.mockResolvedValueOnce({ id: 42, title: "Active Tab" });
windows.getLastFocused.mockResolvedValueOnce(
@@ -8,6 +8,7 @@ import type {
RequestFrame,
} from "@/transport/types";
import { ToolDispatcher } from "../dispatcher";
import { resetBrowserObservationForTests, setBrowserObservationAttachForTests } from "../record";
type TestDispatcherCdp = NonNullable<ConstructorParameters<typeof ToolDispatcher>[0]["cdp"]>;
@@ -44,9 +45,72 @@ function makeRequest(method: string, params: unknown): RequestFrame {
describe("ToolDispatcher", () => {
afterEach(() => {
resetBrowserObservationForTests();
vi.unstubAllGlobals();
});
it("uses the configured recording runtime for start and stop", async () => {
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
agentWindow: {
create: vi.fn(async () => 4242),
remove: vi.fn(),
ensureActiveTab: vi.fn(async () => {}),
},
});
await sessions.start("aa11");
const tab = {
id: 7,
windowId: 4242,
active: true,
status: "complete",
url: "https://example.com/start",
} as chrome.tabs.Tab;
const frameCoordinator = {
begin: vi.fn(),
armTab: vi.fn(async () => true),
sourceFor: vi.fn(() => null),
stop: vi.fn(async () => true),
cancel: vi.fn(),
};
const sendToTab = vi.fn(async () => ({ ok: true }));
setBrowserObservationAttachForTests(
() => () => {},
() => () => {},
);
const dispatcher = new ToolDispatcher({
transport,
sessions,
recording: {
tabsApi: {
get: vi.fn(async () => tab),
query: vi.fn(async () => [tab]),
},
frameCoordinator,
sendToTab,
},
});
dispatcher.start();
deliver(
makeRequest("tool.record_start", {
session_id: "aa11",
url: "https://example.com/start",
}),
);
await vi.waitFor(() => expect(sent).toHaveLength(1));
expect(sent[0]).toMatchObject({ result: { tab_id: 7, recording: true } });
expect(frameCoordinator.begin).toHaveBeenCalledOnce();
expect(frameCoordinator.armTab).toHaveBeenCalledWith(expect.any(String), 7);
deliver({ id: "r-2", method: "tool.record_stop", params: { session_id: "aa11" } });
await vi.waitFor(() => expect(sent).toHaveLength(2));
expect(frameCoordinator.stop).toHaveBeenCalledOnce();
expect(sent[1]).toMatchObject({ id: "r-2", result: { trace: { steps: [] } } });
});
it("routes tool.session_start to the SessionManager and replies with the window id", async () => {
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
@@ -168,6 +232,107 @@ describe("ToolDispatcher", () => {
});
});
it("bypasses and restores the control overlay for an upload trigger click", async () => {
const sendMessage = vi.fn(async () => undefined);
vi.stubGlobal("chrome", {
tabs: {
get: vi.fn(async () => ({ id: 7, windowId: 4242, active: true })),
query: vi.fn(async () => [{ id: 7, windowId: 4242, active: true }]),
sendMessage,
},
});
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
agentWindow: {
create: vi.fn(async () => 4242),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
},
});
const ctx = await sessions.start("aa11");
ctx.refStore.set("e1", 123, { tabId: 7 });
const send = vi.fn(async <T>(_tabId: number, method: string, params?: object) => {
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } } as T;
}
if (method === "DOM.getContentQuads") {
return { quads: [[0, 0, 20, 0, 20, 20, 0, 20]] } as T;
}
if (method === "DOM.resolveNode") {
return { object: { objectId: "trigger-object" } } as T;
}
if (method === "DOM.describeNode") return { node: { backendNodeId: 456 } } as T;
if (method === "Runtime.callFunctionOn") {
const declaration = (params as { functionDeclaration?: string }).functionDeclaration ?? "";
if (declaration.includes("count: state.inputs.length")) {
return { result: { value: { count: 1, multiple: false } } } as T;
}
if (declaration.includes("inputs[0]")) {
return { result: { objectId: "input-object" } } as T;
}
return { result: { value: true } } as T;
}
if (method === "Runtime.evaluate") {
const expression = (params as { expression?: string }).expression ?? "";
if (expression.includes("overlayDetails")) {
return { result: { value: { hitIndex: 0 } } } as T;
}
if (expression.includes("overlayHostPresent")) {
return {
result: {
value: { overlayHostPresent: true, overlayHostConnected: true },
},
} as T;
}
if (expression.includes("count:")) {
return { result: { value: { count: 1, multiple: false } } } as T;
}
if (expression.includes("?.inputs[0]")) {
return { result: { objectId: "input-object" } } as T;
}
return { result: { value: true } } as T;
}
return {} as T;
});
const cdp = {
send,
detachSession: vi.fn(async () => {}),
ensureNetworkCapture: vi.fn(async () => {}),
networkEntriesSince: vi.fn(() => ({
tab_id: 7,
entries: [],
next_since: 0,
truncated: false,
})),
setDeviceMetricsOverride: vi.fn(async () => {}),
clearDeviceMetricsOverride: vi.fn(async () => {}),
setUserAgentOverride: vi.fn(async () => {}),
setTouchEmulationEnabled: vi.fn(async () => {}),
};
const dispatcher = new ToolDispatcher({ transport, sessions, cdp: cdp as TestDispatcherCdp });
dispatcher.start();
deliver(
makeRequest("tool.upload", {
session_id: "aa11",
ref: "@e1",
files: [{ transfer_id: "tr_1", name: "test.png", staged_path: "/stage/test.png" }],
}),
);
await flushMicrotasks();
await vi.waitFor(() => expect(sent).toHaveLength(1));
expect(sent[0]).toMatchObject({ result: { tab_id: 7, file_names: ["test.png"] } });
expect(sendMessage).toHaveBeenNthCalledWith(1, 7, {
type: "bh-automation-bypass",
enabled: true,
});
expect(sendMessage).toHaveBeenNthCalledWith(2, 7, {
type: "bh-automation-bypass",
enabled: false,
});
});
it("detaches CDP state before stopping a session", async () => {
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
@@ -0,0 +1,809 @@
import { describe, expect, it, vi } from "vitest";
import { SessionManager } from "@/session-manager/manager";
import { type DownloadsApi, handleDownload } from "../download";
import { captureBrowserDownload } from "../download-capture";
import { uploadThroughActivatedFileInput } from "../file-input-transaction";
import type { ResolvedActionTarget } from "../interaction";
import type { CdpRunner } from "../shared";
import { handleUpload } from "../upload";
function sessions() {
return new SessionManager({
agentWindow: {
create: vi.fn(async () => 100),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
},
});
}
function tabsApi() {
return {
get: vi.fn(
async (tabId: number) => ({ id: tabId, windowId: 100, active: true }) as chrome.tabs.Tab,
),
query: vi.fn(async () => [{ id: 4, windowId: 100, active: true } as chrome.tabs.Tab]),
};
}
function fakeEvent<T extends (...args: never[]) => unknown>() {
const listeners = new Set<T>();
return {
addListener: (listener: T) => listeners.add(listener),
removeListener: (listener: T) => listeners.delete(listener),
emit: (...args: Parameters<T>) => {
for (const listener of listeners) listener(...args);
},
};
}
function actionTarget(frameId?: string): ResolvedActionTarget {
return {
tab: { tabId: 4, windowId: 100, active: true },
backendNodeId: 123,
cdpTarget: { tabId: 4 },
...(frameId ? { frameId } : {}),
usedRef: "e3",
};
}
function uploadCdp(
options: {
inputCount?: number;
multiple?: boolean;
chooser?: { frameId?: string; backendNodeId?: number; mode?: string };
pendingResolve?: boolean;
} = {},
) {
const calls: Array<{ method: string; params?: object }> = [];
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const send = vi.fn(async (_tabId: number, method: string, params?: object) => {
calls.push({ method, params });
if (method === "Page.setInterceptFileChooserDialog") return {};
if (method === "Page.getLayoutMetrics")
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
if (method === "DOM.getContentQuads") return { quads: [[0, 0, 20, 0, 20, 20, 0, 20]] };
if (method === "DOM.resolveNode") {
if (options.pendingResolve) return new Promise<never>(() => {});
return { object: { objectId: "trigger-object" } };
}
if (method === "Runtime.callFunctionOn") {
const declaration = (params as { functionDeclaration?: string }).functionDeclaration ?? "";
if (declaration.includes("Object.defineProperty")) return { result: { value: true } };
if (declaration.includes("count: state.inputs.length")) {
return {
result: {
value: {
count: options.inputCount ?? 1,
multiple: options.multiple ?? true,
},
},
};
}
if (declaration.includes("inputs[0]")) {
return { result: { objectId: "input-object" } };
}
return { result: { value: true } };
}
if (method === "DOM.describeNode") return { node: { backendNodeId: 456 } };
if (
method === "Input.dispatchMouseEvent" &&
(params as { type?: string }).type === "mousePressed" &&
options.chooser
) {
cdpEvent?.({ tabId: 4 }, "Page.fileChooserOpened", options.chooser);
}
return {};
});
const cdp: CdpRunner = {
send: send as unknown as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
return {
calls,
emitChooser: (event: { frameId?: string; backendNodeId?: number; mode?: string }) =>
cdpEvent?.({ tabId: 4 }, "Page.fileChooserOpened", event),
cdp,
};
}
describe("file transfer tools", () => {
it("captures the file input activated by the requested click and injects only staged paths", async () => {
const manager = sessions();
const ctx = await manager.start("s1");
ctx.refStore.set("e3", 123, { tabId: 4 });
const { cdp, calls } = uploadCdp();
const result = await handleUpload(
manager,
{
session_id: "s1",
ref: "@e3",
files: [
{ transfer_id: "tr_1", name: "one.png", staged_path: "/private/stage/one" },
{ transfer_id: "tr_2", name: "two.png", staged_path: "/private/stage/two" },
],
},
{ cdp, tabsApi: tabsApi() },
);
expect(result).toMatchObject({ tab_id: 4, file_names: ["one.png", "two.png"] });
expect(calls[0]).toEqual({
method: "Page.setInterceptFileChooserDialog",
params: { enabled: true },
});
expect(calls).toContainEqual({
method: "Page.setInterceptFileChooserDialog",
params: { enabled: false, cancel: true },
});
expect(calls).toContainEqual({
method: "DOM.setFileInputFiles",
params: { files: ["/private/stage/one", "/private/stage/two"], backendNodeId: 456 },
});
});
it("fails immediately when the trigger does not activate a file input", async () => {
const { cdp } = uploadCdp({ inputCount: 0 });
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget(),
files: ["/private/stage/one"],
timeoutMs: 100,
trigger: async () => ({ tab_id: 4, x: 10, y: 10 }),
});
expect(result).toMatchObject({
code: "unsupported",
data: { reason: "file_input_not_activated", phase: "resolve_input" },
});
});
it("fails before clicking when chooser interception is unavailable", async () => {
const trigger = vi.fn(async () => ({ tab_id: 4, x: 10, y: 10 }));
const cdp: CdpRunner = {
send: vi.fn(async (_tabId: number, method: string) => {
if (method === "Page.setInterceptFileChooserDialog") {
throw new Error("method unavailable");
}
return {};
}) as CdpRunner["send"],
};
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget(),
files: ["/private/stage/one"],
timeoutMs: 100,
trigger,
});
expect(result).toMatchObject({
code: "cdp_failed",
data: { effect_state: "none", phase: "arm_interception" },
});
expect(trigger).not.toHaveBeenCalled();
});
it("uses an exact chooser event as an independent input-location signal", async () => {
const { cdp, calls, emitChooser } = uploadCdp({ inputCount: 0 });
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget("f1"),
files: ["/private/stage/one"],
timeoutMs: 100,
trigger: async () => {
emitChooser({ frameId: "f1", backendNodeId: 789, mode: "selectSingle" });
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(result).toMatchObject({ multiple: false });
expect(calls).toContainEqual({
method: "DOM.setFileInputFiles",
params: { files: ["/private/stage/one"], backendNodeId: 789 },
});
});
it("reports a File System Access picker without waiting for a timeout", async () => {
const { cdp, emitChooser } = uploadCdp({
inputCount: 0,
});
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget("f1"),
files: ["/private/stage/one"],
timeoutMs: 100,
trigger: async () => {
emitChooser({ frameId: "f1", mode: "selectSingle" });
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(result).toMatchObject({
code: "unsupported",
message: "upload trigger invoked a non-input file picker",
data: { reason: "file_input_not_activated", phase: "resolve_input" },
});
});
it("bounds a stuck file-input probe", async () => {
const { cdp } = uploadCdp({ pendingResolve: true });
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget(),
files: ["/private/stage/one"],
timeoutMs: 5,
trigger: vi.fn(),
});
expect(result).toMatchObject({
code: "timeout",
data: { reason: "file_input_probe_failed", phase: "arm_input_probe" },
});
});
it("does not dispatch browser work when upload is already cancelled", async () => {
const controller = new AbortController();
controller.abort();
const { cdp, calls } = uploadCdp();
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget(),
files: ["/private/stage/one"],
timeoutMs: 100,
signal: controller.signal,
trigger: vi.fn(),
});
expect(result).toMatchObject({
code: "cancelled",
data: { effect_state: "none", phase: "arm_interception" },
});
expect(calls).not.toContainEqual({
method: "Page.setInterceptFileChooserDialog",
params: { enabled: true },
});
expect(calls.some((call) => call.method === "DOM.setFileInputFiles")).toBe(false);
});
it("stops before file assignment when cancellation interrupts input resolution", async () => {
const controller = new AbortController();
const fixture = uploadCdp();
const originalSend = fixture.cdp.send;
let markProbeStarted!: () => void;
const probeStarted = new Promise<void>((resolve) => {
markProbeStarted = resolve;
});
fixture.cdp.send = vi.fn((tabId: number, method: string, params?: object) => {
const declaration = (params as { functionDeclaration?: string } | undefined)
?.functionDeclaration;
if (
method === "Runtime.callFunctionOn" &&
declaration?.includes("count: state.inputs.length")
) {
markProbeStarted();
return new Promise<never>(() => {});
}
return originalSend(tabId, method, params);
}) as CdpRunner["send"];
const transaction = uploadThroughActivatedFileInput({
cdp: fixture.cdp,
actionTarget: actionTarget(),
files: ["/private/stage/one"],
timeoutMs: 1_000,
signal: controller.signal,
trigger: async () => ({ tab_id: 4, x: 10, y: 10 }),
});
await probeStarted;
controller.abort();
const result = await transaction;
expect(result).toMatchObject({
code: "cancelled",
data: { effect_state: "none", phase: "resolve_input" },
});
expect(fixture.calls.some((call) => call.method === "DOM.setFileInputFiles")).toBe(false);
});
it("disarms chooser interception when its enable response times out", async () => {
const calls: Array<{ method: string; params?: object }> = [];
const cdp: CdpRunner = {
send: vi.fn(async (_tabId: number, method: string, params?: object) => {
calls.push({ method, params });
if (
method === "Page.setInterceptFileChooserDialog" &&
(params as { enabled?: boolean }).enabled
) {
return new Promise<never>(() => {});
}
return {};
}) as CdpRunner["send"],
};
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget(),
files: ["/private/stage/one"],
timeoutMs: 5,
trigger: vi.fn(),
});
expect(result).toMatchObject({
code: "timeout",
data: { effect_state: "none", phase: "arm_interception" },
});
expect(calls).toContainEqual({
method: "Page.setInterceptFileChooserDialog",
params: { enabled: false, cancel: true },
});
});
it("marks a timed-out file assignment unknown and detaches browser state", async () => {
const fixture = uploadCdp();
const originalSend = fixture.cdp.send;
const detach = vi.fn(async () => {});
fixture.cdp.detach = detach;
fixture.cdp.send = vi.fn((tabId: number, method: string, params?: object) => {
if (method === "DOM.setFileInputFiles") return new Promise<never>(() => {});
return originalSend(tabId, method, params);
}) as CdpRunner["send"];
const result = await uploadThroughActivatedFileInput({
cdp: fixture.cdp,
actionTarget: actionTarget(),
files: ["/private/stage/one"],
timeoutMs: 10,
trigger: async () => ({ tab_id: 4, x: 10, y: 10 }),
});
expect(result).toMatchObject({
code: "timeout",
data: { effect_state: "unknown", phase: "set_files" },
});
expect(detach).toHaveBeenCalledWith(4);
});
it("routes one exact-target download through a browser-relative capability", async () => {
const manager = sessions();
const ctx = await manager.start("s1");
ctx.refStore.set("e3", 123, { tabId: 4 });
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const initial = {
id: 9,
url: "https://example.test/result.zip",
finalUrl: "https://example.test/result.zip",
filename: "result.zip",
state: "in_progress",
fileSize: -1,
totalBytes: 12,
mime: "application/zip",
danger: "safe",
} as chrome.downloads.DownloadItem;
const completed = {
...initial,
filename: "/profile/Downloads/BrowserSkill/tr_1/result.zip",
state: "complete",
fileSize: 12,
} as chrome.downloads.DownloadItem;
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn(async () => [completed]),
cancel: vi.fn(async () => {}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
let suggested: chrome.downloads.DownloadFilenameSuggestion | undefined;
const send = vi.fn(async (_tabId: number, method: string, params?: object) => {
if (method === "Page.getLayoutMetrics")
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
if (method === "DOM.getContentQuads") return { quads: [[0, 0, 20, 0, 20, 20, 0, 20]] };
if (
method === "Input.dispatchMouseEvent" &&
(params as { type?: string }).type === "mousePressed"
) {
cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", {
url: initial.url,
suggestedFilename: "result.zip",
});
await new Promise<void>((resolve) => {
onDeterminingFilename.emit(initial, (value) => {
suggested = value;
resolve();
});
});
onCreated.emit(completed);
}
return {};
});
const cdp: CdpRunner = {
send: send as unknown as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
const result = await handleDownload(
manager,
{ session_id: "s1", ref: "@e3", browser_relative_dir: "BrowserSkill/tr_1" },
{ cdp, tabsApi: tabsApi(), downloads },
);
expect(suggested).toEqual({
filename: "BrowserSkill/tr_1/result.zip",
conflictAction: "overwrite",
});
expect(result).toMatchObject({
tab_id: 4,
suggested_filename: "result.zip",
byte_size: 12,
browser_path: completed.filename,
});
});
it("does not claim a download without an intent from the exact target", async () => {
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn(async () => []),
cancel: vi.fn(async () => {}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as unknown as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
const unrelated = {
id: 17,
url: "https://example.test/unrelated.zip",
finalUrl: "https://example.test/unrelated.zip",
filename: "unrelated.zip",
state: "in_progress",
} as chrome.downloads.DownloadItem;
let defaultSuggestionCalled = false;
const result = await captureBrowserDownload({
cdp,
target: { tabId: 4, sessionId: "expected-child" },
downloads,
browserRelativeDir: "BrowserSkill/tr_1",
timeoutMs: 5,
trigger: async () => {
cdpEvent?.({ tabId: 4, sessionId: "other-child" }, "Page.downloadWillBegin", {
url: unrelated.url,
suggestedFilename: unrelated.filename,
});
onDeterminingFilename.emit(unrelated, (suggestion) => {
defaultSuggestionCalled = suggestion === undefined;
});
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(defaultSuggestionCalled).toBe(true);
expect(downloads.cancel).not.toHaveBeenCalled();
expect(result).toMatchObject({
code: "cdp_failed",
data: { reason: "download_capture_failed" },
});
});
it("correlates a filename candidate that arrives before the CDP intent", async () => {
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const initial = {
id: 21,
url: "https://example.test/candidate-first.bin",
finalUrl: "https://example.test/candidate-first.bin",
filename: "candidate-first.bin",
state: "in_progress",
fileSize: -1,
totalBytes: 4,
bytesReceived: 0,
} as chrome.downloads.DownloadItem;
const complete = {
...initial,
state: "complete",
fileSize: 4,
} as chrome.downloads.DownloadItem;
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn(async () => [complete]),
cancel: vi.fn(async () => {}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
let suggestion: chrome.downloads.DownloadFilenameSuggestion | undefined;
const result = await captureBrowserDownload({
cdp,
target: { tabId: 4 },
downloads,
browserRelativeDir: "BrowserSkill/tr_21",
timeoutMs: 1_000,
trigger: async () => {
const suggested = new Promise<void>((resolve) => {
onDeterminingFilename.emit(initial, (value) => {
suggestion = value;
resolve();
});
});
cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", {
url: initial.url,
suggestedFilename: initial.filename,
});
await suggested;
onCreated.emit(complete);
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(suggestion).toEqual({
filename: "BrowserSkill/tr_21/candidate-first.bin",
conflictAction: "overwrite",
});
expect(result).toMatchObject({ item: { id: 21, state: "complete" } });
});
it("removes a completed download when the final size exceeds the limit", async () => {
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const initial = {
id: 22,
url: "https://example.test/oversized.bin",
finalUrl: "https://example.test/oversized.bin",
filename: "oversized.bin",
state: "in_progress",
fileSize: -1,
totalBytes: -1,
bytesReceived: 0,
} as chrome.downloads.DownloadItem;
const complete = {
...initial,
filename: "/profile/Downloads/BrowserSkill/tr_22/oversized.bin",
state: "complete",
fileSize: 8,
totalBytes: 8,
bytesReceived: 8,
} as chrome.downloads.DownloadItem;
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn(async () => [complete]),
cancel: vi.fn(async () => {}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
const result = await captureBrowserDownload({
cdp,
target: { tabId: 4 },
downloads,
browserRelativeDir: "BrowserSkill/tr_22",
maxByteSize: 4,
timeoutMs: 1_000,
trigger: async () => {
cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", {
url: initial.url,
suggestedFilename: initial.filename,
});
await new Promise<void>((resolve) => {
onDeterminingFilename.emit(initial, () => resolve());
});
onCreated.emit(initial);
onChanged.emit({ id: initial.id, state: { current: "complete" } });
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(result).toMatchObject({
code: "cdp_failed",
data: { reason: "download_capture_failed", effect_state: "committed" },
});
expect(downloads.removeFile).toHaveBeenCalledWith(initial.id);
expect(downloads.cancel).not.toHaveBeenCalled();
});
it.each([
["cancel resolves", false],
["cancel reports that completion won", true],
])("reconciles a download that completes while %s", async (_label, cancelThrows) => {
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const initial = {
id: 23,
url: "https://example.test/racing.bin",
finalUrl: "https://example.test/racing.bin",
filename: "racing.bin",
state: "in_progress",
fileSize: -1,
totalBytes: -1,
bytesReceived: 0,
} as chrome.downloads.DownloadItem;
const complete = {
...initial,
filename: "/profile/Downloads/BrowserSkill/tr_23/racing.bin",
state: "complete",
fileSize: 4,
totalBytes: 4,
bytesReceived: 4,
} as chrome.downloads.DownloadItem;
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn().mockResolvedValueOnce([initial]).mockResolvedValueOnce([complete]),
cancel: vi.fn(async () => {
if (cancelThrows) throw new Error("download already complete");
}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
const result = await captureBrowserDownload({
cdp,
target: { tabId: 4 },
downloads,
browserRelativeDir: "BrowserSkill/tr_23",
timeoutMs: 80,
trigger: async () => {
cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", {
url: initial.url,
suggestedFilename: initial.filename,
});
await new Promise<void>((resolve) => {
onDeterminingFilename.emit(initial, () => resolve());
});
onCreated.emit(initial);
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(result).toMatchObject({
code: "cdp_failed",
data: { reason: "download_capture_failed", effect_state: "committed" },
});
expect(downloads.cancel).toHaveBeenCalledWith(initial.id);
expect(downloads.removeFile).toHaveBeenCalledWith(initial.id);
expect(result).not.toMatchObject({ data: { cleanup_state: "failed" } });
});
it("rejects ambiguous attribution without cancelling either unclaimed download", async () => {
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn(async () => []),
cancel: vi.fn(async () => {}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
const defaults: number[] = [];
const candidate = (id: number) =>
({
id,
url: "https://example.test/same.bin",
finalUrl: "https://example.test/same.bin",
filename: "same.bin",
state: "in_progress",
}) as chrome.downloads.DownloadItem;
const result = await captureBrowserDownload({
cdp,
target: { tabId: 4 },
downloads,
browserRelativeDir: "BrowserSkill/tr_ambiguous",
timeoutMs: 100,
trigger: async () => {
cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", {
url: "https://example.test/same.bin",
suggestedFilename: "same.bin",
});
for (const id of [31, 32]) {
onDeterminingFilename.emit(candidate(id), (value) => {
if (value === undefined) defaults.push(id);
});
}
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(defaults.sort()).toEqual([31, 32]);
expect(downloads.cancel).not.toHaveBeenCalled();
expect(result).toMatchObject({
code: "cdp_failed",
data: { effect_state: "unknown", phase: "attribution" },
});
});
});
@@ -123,6 +123,29 @@ describe("handleClick", () => {
expect(fake.cdp.send).not.toHaveBeenCalled();
});
it("rejects screenshot-only visual surface refs before issuing CDP calls", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const ctx = await sm.start("aa11");
ctx.refStore.set("e3", 1234, {
tabId: 4,
kind: "surface",
capabilities: ["screenshot"],
});
const fake = makeFakeCdp({});
const res = await handleClick(
sm,
{ session_id: "aa11", ref: "@e3" },
{ cdp: fake.cdp, tabsApi: fake.tabsApi },
);
expect(res).toMatchObject({
code: "permission_denied",
data: { reason: "ref_capability_denied", required_capability: "interact" },
});
expect(fake.cdp.send).not.toHaveBeenCalled();
});
it("clicks by ref, computes the quad centre, dispatches three mouse events", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const ctx = await sm.start("aa11");
File diff suppressed because it is too large Load Diff
@@ -15,7 +15,12 @@ import {
const AGENT_WINDOW_ID = 100;
const TAB_ID = 4;
const START_URL = "https://example.com/";
const RECORD_START_V3 = { session_id: "abcd", url: START_URL, trace_version: 3 as const };
const RECORD_START_V3 = {
session_id: "abcd",
url: START_URL,
trace_version: 3 as const,
supports_tab_switch_steps: true,
};
const stepSequenceByProducer = new Map<string, number>();
function asTraceV3(trace: RecordedTrace): TraceV3 {
@@ -48,6 +53,7 @@ function chromeEvent<T extends (...args: never[]) => unknown>() {
function installChrome() {
const runtimeOnMessage = chromeEvent<RuntimeListener>();
const tabsOnActivated = chromeEvent<(activeInfo: chrome.tabs.TabActiveInfo) => unknown>();
const webNavigationOnCompleted =
chromeEvent<(details: chrome.webNavigation.WebNavigationFramedCallbackDetails) => unknown>();
const webNavigationOnCommitted =
@@ -57,7 +63,7 @@ function installChrome() {
vi.stubGlobal("chrome", {
runtime: { onMessage: runtimeOnMessage },
tabs: {
onActivated: chromeEvent(),
onActivated: tabsOnActivated,
onCreated: chromeEvent(),
onUpdated: chromeEvent(),
},
@@ -66,7 +72,7 @@ function installChrome() {
onCommitted: webNavigationOnCommitted,
},
});
return { runtimeOnMessage, webNavigationOnCompleted, webNavigationOnCommitted };
return { runtimeOnMessage, tabsOnActivated, webNavigationOnCompleted, webNavigationOnCommitted };
}
function fakeManager() {
@@ -118,7 +124,10 @@ type FakeCdp = CdpRunner & {
const LONG_IDLE_MS = 10_000;
/** AX-only page: DOMSnapshot calls throw so capture falls back to the AX tree. */
function makeFakeCdp(trees?: unknown[], options?: { failCaptures?: boolean }): FakeCdp {
function makeFakeCdp(
trees?: unknown[],
options?: { failCaptures?: boolean; treesByTab?: Record<number, unknown> },
): FakeCdp {
type EventListener = (source: chrome.debugger.Debuggee, method: string, params: unknown) => void;
const events: EventListener[] = [];
const script = [...(trees ?? [])];
@@ -153,6 +162,8 @@ function makeFakeCdp(trees?: unknown[], options?: { failCaptures?: boolean }): F
"Accessibility.enable": () => ({}),
"Accessibility.getFullAXTree": (_params, _tabId) => {
if (failing) throw new Error("Execution context was destroyed");
const tabTree = options?.treesByTab?.[_tabId];
if (tabTree) return tabTree;
if (script.length === 0) return axTree("Example Domain", ["Submit"]);
return script.length === 1 ? script[0] : script.shift();
},
@@ -202,6 +213,31 @@ function makeTabsApi() {
};
}
function makeMultiTabsApi(tabs: chrome.tabs.Tab[]) {
const byId = new Map(tabs.flatMap((tab) => (typeof tab.id === "number" ? [[tab.id, tab]] : [])));
return {
get: async (tabId: number) => {
const tab = byId.get(tabId);
if (!tab) throw new Error(`tab ${tabId} closed`);
return tab;
},
query: async (query: chrome.tabs.QueryInfo) =>
[...byId.values()].filter(
(tab) =>
(query.windowId === undefined || tab.windowId === query.windowId) &&
(query.active === undefined || tab.active === query.active),
),
activate(tabId: number) {
for (const tab of byId.values()) tab.active = tab.id === tabId;
},
goTo(tabId: number, url: string, title: string) {
const tab = byId.get(tabId);
if (!tab) throw new Error(`tab ${tabId} closed`);
Object.assign(tab, { url, title });
},
};
}
/** Outlast a settle on a quiet page plus the per-tab observation cooldown. */
function settleWait(): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, 800));
@@ -212,16 +248,19 @@ function runtimeOnMessageEmit(
requestId: string,
step: unknown,
tabId = TAB_ID,
): void {
active = true,
): ReturnType<typeof vi.fn> {
const producerId = `test-producer-${tabId}`;
const producerKey = `${requestId}:${producerId}`;
const sequence = (stepSequenceByProducer.get(producerKey) ?? 0) + 1;
stepSequenceByProducer.set(producerKey, sequence);
const sendResponse = vi.fn();
chromeApi.runtimeOnMessage.emit(
{ type: RECORD_STEP, requestId, producerId, sequence, step },
{ tab: { id: tabId } } as chrome.runtime.MessageSender,
() => {},
{ tab: { id: tabId, active } } as chrome.runtime.MessageSender,
sendResponse,
);
return sendResponse;
}
describe("recorded user steps reach the exported trace", () => {
@@ -1161,7 +1200,8 @@ describe("recorded user steps reach the exported trace", () => {
const trace = (stopped as RecordStopResult).trace;
expect("version" in trace).toBe(false);
expect("pages" in trace && trace.pages.length).toBeGreaterThan(0);
expect("steps" in trace && trace.steps.map((step) => step.op)).toEqual(["hover", "click"]);
// `hover` has no v2 counterpart on peers that negotiate v2.
expect("steps" in trace && trace.steps.map((step) => step.op)).toEqual(["click"]);
});
it("rejects unsupported trace_version values", async () => {
@@ -1173,4 +1213,391 @@ describe("recorded user steps reach the exported trace", () => {
);
expect(result).toMatchObject({ code: "invalid_params" });
});
it("records tab transitions and binds actions to each tab's observation", async () => {
const chromeApi = installChrome();
const manager = fakeManager();
const secondUrl = "https://example.com/second";
const tabsApi = makeMultiTabsApi([
{
id: TAB_ID,
windowId: AGENT_WINDOW_ID,
active: true,
status: "complete",
url: START_URL,
title: "First tab",
} as chrome.tabs.Tab,
{
id: 5,
windowId: AGENT_WINDOW_ID,
active: false,
status: "complete",
url: secondUrl,
title: "Second tab",
} as chrome.tabs.Tab,
]);
const cdp = makeFakeCdp(undefined, {
treesByTab: {
[TAB_ID]: axTree("First tab", ["First action"]),
5: axTree("Second tab", ["Second action"]),
},
});
let requestId = "";
const sendToTab = vi.fn(async (_tabId: number, message: unknown) => {
const typed = message as { type?: string; requestId?: string };
if (typed.type === RECORD_START && typed.requestId) requestId = typed.requestId;
return { ok: true };
});
const deps = { tabsApi, sendToTab, cdp };
await handleRecordStart(manager, RECORD_START_V3, deps);
attachRecordStepListener(deps);
tabsApi.activate(5);
chromeApi.tabsOnActivated.emit({ tabId: 5, windowId: AGENT_WINDOW_ID });
await settleWait();
runtimeOnMessageEmit(
chromeApi,
requestId,
{
op: "click",
page_url: secondUrl,
target: { role: "button", name: "Second action", tag: "button" },
},
5,
);
await settleWait();
tabsApi.activate(TAB_ID);
chromeApi.tabsOnActivated.emit({ tabId: TAB_ID, windowId: AGENT_WINDOW_ID });
await settleWait();
runtimeOnMessageEmit(chromeApi, requestId, {
op: "click",
page_url: START_URL,
target: { role: "button", name: "First action", tag: "button" },
});
await settleWait();
const stopped = await handleRecordStop(manager, { session_id: "abcd" }, deps);
const trace = asTraceV3((stopped as RecordStopResult).trace);
const stateUrl = (stateId: string) => trace.states.find((state) => state.id === stateId)?.url;
const switches = trace.steps.filter((step) => step.op === "switch_tab");
const clicks = trace.steps.filter((step) => step.op === "click");
expect(switches).toHaveLength(2);
expect(clicks).toHaveLength(2);
expect(stateUrl(switches[0]!.state)).toBe(START_URL);
expect(stateUrl(switches[0]!.result.state)).toBe(secondUrl);
expect(stateUrl(clicks[0]!.state)).toBe(secondUrl);
expect(stateUrl(switches[1]!.result.state)).toBe(START_URL);
expect(stateUrl(clicks[1]!.state)).toBe(START_URL);
});
it("keeps switch_tab internal when the v3 caller did not advertise support", async () => {
const chromeApi = installChrome();
const manager = fakeManager();
const tabsApi = makeMultiTabsApi([
{
id: TAB_ID,
windowId: AGENT_WINDOW_ID,
active: true,
status: "complete",
url: START_URL,
} as chrome.tabs.Tab,
{
id: 5,
windowId: AGENT_WINDOW_ID,
active: false,
status: "complete",
url: "https://example.com/second",
} as chrome.tabs.Tab,
]);
const deps = { tabsApi, sendToTab: vi.fn(async () => ({ ok: true })), cdp: makeFakeCdp() };
await handleRecordStart(
manager,
{ session_id: "abcd", url: START_URL, trace_version: 3 },
deps,
);
tabsApi.activate(5);
chromeApi.tabsOnActivated.emit({ tabId: 5, windowId: AGENT_WINDOW_ID });
await settleWait();
const stopped = await handleRecordStop(manager, { session_id: "abcd" }, deps);
const trace = asTraceV3((stopped as RecordStopResult).trace);
expect(trace.steps.some((step) => step.op === "switch_tab")).toBe(false);
});
it("keeps the latest rapid tab activation when an earlier rearm acknowledges late", async () => {
const chromeApi = installChrome();
const manager = fakeManager();
const tabsApi = makeMultiTabsApi(
[TAB_ID, 5, 6].map(
(id) =>
({
id,
windowId: AGENT_WINDOW_ID,
active: id === TAB_ID,
status: "complete",
url: id === TAB_ID ? START_URL : `https://example.com/tab-${id}`,
}) as chrome.tabs.Tab,
),
);
let releaseSecond!: () => void;
const secondGate = new Promise<void>((resolve) => {
releaseSecond = resolve;
});
let announceSecond!: () => void;
const secondStarted = new Promise<void>((resolve) => {
announceSecond = resolve;
});
const sendToTab = vi.fn(async (tabId: number, message: unknown) => {
if ((message as { type?: string }).type === RECORD_START && tabId === 5) {
announceSecond();
await secondGate;
}
return { ok: true };
});
const deps = { tabsApi, sendToTab, cdp: makeFakeCdp() };
await handleRecordStart(manager, RECORD_START_V3, deps);
tabsApi.activate(5);
chromeApi.tabsOnActivated.emit({ tabId: 5, windowId: AGENT_WINDOW_ID });
await secondStarted;
tabsApi.activate(6);
chromeApi.tabsOnActivated.emit({ tabId: 6, windowId: AGENT_WINDOW_ID });
await settleWait();
releaseSecond();
await new Promise((resolve) => setTimeout(resolve, 50));
const stopped = await handleRecordStop(manager, { session_id: "abcd" }, deps);
const trace = asTraceV3((stopped as RecordStopResult).trace);
const switches = trace.steps.filter((step) => step.op === "switch_tab");
const stateUrl = (stateId: string) => trace.states.find((state) => state.id === stateId)?.url;
expect(switches).toHaveLength(1);
expect(stateUrl(switches[0]!.result.state)).toBe("https://example.com/tab-6");
});
it("does not retry an in-flight tab rearm after recording stops", async () => {
const chromeApi = installChrome();
const manager = fakeManager();
const tabsApi = makeMultiTabsApi([
{
id: TAB_ID,
windowId: AGENT_WINDOW_ID,
active: true,
status: "complete",
url: START_URL,
} as chrome.tabs.Tab,
{
id: 5,
windowId: AGENT_WINDOW_ID,
active: false,
status: "complete",
url: "https://example.com/second",
} as chrome.tabs.Tab,
]);
let failRearm!: () => void;
const rearmGate = new Promise<never>((_resolve, reject) => {
failRearm = () => reject(new Error("rearm failed after stop"));
});
let announceRearm!: () => void;
const rearmStarted = new Promise<void>((resolve) => {
announceRearm = resolve;
});
const sendToTab = vi.fn(async (tabId: number, message: unknown) => {
if ((message as { type?: string }).type === RECORD_START && tabId === 5) {
announceRearm();
await rearmGate;
}
return { ok: true };
});
const deps = { tabsApi, sendToTab, cdp: makeFakeCdp() };
await handleRecordStart(manager, RECORD_START_V3, deps);
tabsApi.activate(5);
chromeApi.tabsOnActivated.emit({ tabId: 5, windowId: AGENT_WINDOW_ID });
await rearmStarted;
const stoppedPromise = handleRecordStop(manager, { session_id: "abcd" }, deps);
await vi.waitFor(() => {
expect(sendToTab).toHaveBeenCalledWith(5, expect.objectContaining({ type: RECORD_STOP }));
});
failRearm();
const stopped = await stoppedPromise;
await new Promise((resolve) => setTimeout(resolve, 600));
const startsForSecondTab = sendToTab.mock.calls.filter(
([tabId, message]) => tabId === 5 && (message as { type?: string }).type === RECORD_START,
);
expect(startsForSecondTab).toHaveLength(1);
expect(asTraceV3((stopped as RecordStopResult).trace).steps).toHaveLength(0);
});
it("keeps pending action settles scoped to their tab during an immediate switch", async () => {
const chromeApi = installChrome();
const manager = fakeManager();
const secondUrl = "https://example.com/second";
const tabsApi = makeMultiTabsApi([
{
id: TAB_ID,
windowId: AGENT_WINDOW_ID,
active: true,
status: "complete",
url: START_URL,
} as chrome.tabs.Tab,
{
id: 5,
windowId: AGENT_WINDOW_ID,
active: false,
status: "complete",
url: secondUrl,
} as chrome.tabs.Tab,
]);
let requestId = "";
const sendToTab = vi.fn(async (_tabId: number, message: unknown) => {
const typed = message as { type?: string; requestId?: string };
if (typed.type === RECORD_START && typed.requestId) requestId = typed.requestId;
return { ok: true };
});
const deps = { tabsApi, sendToTab, cdp: makeFakeCdp() };
await handleRecordStart(manager, RECORD_START_V3, deps);
attachRecordStepListener(deps);
runtimeOnMessageEmit(chromeApi, requestId, {
op: "click",
page_url: START_URL,
target: { role: "button", name: "First action", tag: "button" },
});
tabsApi.activate(5);
chromeApi.tabsOnActivated.emit({ tabId: 5, windowId: AGENT_WINDOW_ID });
runtimeOnMessageEmit(
chromeApi,
requestId,
{
op: "click",
page_url: secondUrl,
target: { role: "button", name: "Second action", tag: "button" },
},
5,
);
await settleWait();
const stopped = await handleRecordStop(manager, { session_id: "abcd" }, deps);
const trace = asTraceV3((stopped as RecordStopResult).trace);
const stateUrl = (stateId: string) => trace.states.find((state) => state.id === stateId)?.url;
expect(trace.steps.map((step) => step.op)).toEqual(["click", "switch_tab", "click"]);
expect(stateUrl(trace.steps[0]!.result.state)).toBe(START_URL);
expect(stateUrl(trace.steps[1]!.result.state)).toBe(secondUrl);
expect(stateUrl(trace.steps[2]!.state)).toBe(secondUrl);
});
it("tracks the same navigation URL independently for each tab", async () => {
const chromeApi = installChrome();
const manager = fakeManager();
const tabsApi = makeMultiTabsApi([
{
id: TAB_ID,
windowId: AGENT_WINDOW_ID,
active: true,
status: "complete",
url: START_URL,
} as chrome.tabs.Tab,
{
id: 5,
windowId: AGENT_WINDOW_ID,
active: false,
status: "complete",
url: "https://example.com/second",
} as chrome.tabs.Tab,
]);
const deps = { tabsApi, sendToTab: vi.fn(async () => ({ ok: true })), cdp: makeFakeCdp() };
await handleRecordStart(manager, RECORD_START_V3, deps);
tabsApi.activate(5);
chromeApi.tabsOnActivated.emit({ tabId: 5, windowId: AGENT_WINDOW_ID });
await settleWait();
tabsApi.goTo(5, START_URL, "Same URL in second tab");
chromeApi.webNavigationOnCommitted.emit({
tabId: 5,
frameId: 0,
url: START_URL,
transitionType: "typed",
transitionQualifiers: ["from_address_bar"],
} as unknown as chrome.webNavigation.WebNavigationTransitionCallbackDetails);
await settleWait();
const stopped = await handleRecordStop(manager, { session_id: "abcd" }, deps);
const trace = asTraceV3((stopped as RecordStopResult).trace);
expect(
trace.steps.filter((step) => step.op === "navigate" && step.to === START_URL),
).toHaveLength(1);
});
it("drops late input and frame navigation from an inactive tab", async () => {
const chromeApi = installChrome();
const manager = fakeManager();
const secondUrl = "https://example.com/second";
const tabsApi = makeMultiTabsApi([
{
id: TAB_ID,
windowId: AGENT_WINDOW_ID,
active: true,
status: "complete",
url: START_URL,
} as chrome.tabs.Tab,
{
id: 5,
windowId: AGENT_WINDOW_ID,
active: false,
status: "complete",
url: secondUrl,
} as chrome.tabs.Tab,
]);
let requestId = "";
const sendToTab = vi.fn(async (_tabId: number, message: unknown) => {
const typed = message as { type?: string; requestId?: string };
if (typed.type === RECORD_START && typed.requestId) requestId = typed.requestId;
return { ok: true };
});
const deps = { tabsApi, sendToTab, cdp: makeFakeCdp() };
await handleRecordStart(manager, RECORD_START_V3, deps);
attachRecordStepListener(deps);
tabsApi.activate(5);
chromeApi.tabsOnActivated.emit({ tabId: 5, windowId: AGENT_WINDOW_ID });
await settleWait();
const lateAck = runtimeOnMessageEmit(
chromeApi,
requestId,
{
op: "fill",
page_url: START_URL,
target: { role: "textbox", name: "Draft", tag: "input" },
value: "late",
},
TAB_ID,
false,
);
expect(lateAck).toHaveBeenCalledWith({ ok: true, sequence: 1 });
chromeApi.webNavigationOnCommitted.emit({
tabId: TAB_ID,
frameId: 0,
url: "https://example.com/late",
transitionType: "link",
transitionQualifiers: [],
} as unknown as chrome.webNavigation.WebNavigationTransitionCallbackDetails);
chromeApi.webNavigationOnCommitted.emit({
tabId: 5,
frameId: 7,
url: "https://frame.example/late",
transitionType: "link",
transitionQualifiers: [],
} as unknown as chrome.webNavigation.WebNavigationTransitionCallbackDetails);
await new Promise((resolve) => setTimeout(resolve, 50));
const stopped = await handleRecordStop(manager, { session_id: "abcd" }, deps);
const trace = asTraceV3((stopped as RecordStopResult).trace);
expect(trace.steps.map((step) => step.op)).toEqual(["switch_tab"]);
});
});
@@ -24,10 +24,14 @@ describe("lookupSnapshotRef", () => {
expect(lookupSnapshotRef(ctx, "@e3", 4)).toEqual({
backendNodeId: 1234,
refKey: "e3",
kind: "dom",
capabilities: ["interact", "screenshot"],
});
expect(lookupSnapshotRef(ctx, "e3", 4)).toEqual({
backendNodeId: 1234,
refKey: "e3",
kind: "dom",
capabilities: ["interact", "screenshot"],
});
});
@@ -53,6 +57,8 @@ describe("resolveSnapshotRef", () => {
tabId: 4,
frameId: "child-frame",
cdpSessionId: "child-session",
kind: "dom",
capabilities: ["interact", "screenshot"],
});
const expected = {
@@ -60,6 +66,8 @@ describe("resolveSnapshotRef", () => {
refKey: "e3",
frameId: "child-frame",
cdpSessionId: "child-session",
kind: "dom" as const,
capabilities: ["interact", "screenshot"] as const,
};
expect(lookupSnapshotRef(ctx, "@e3", 4)).toEqual(expected);
expect(resolveSnapshotRef(ctx, "@e3", 4)).toEqual(expected);
@@ -105,6 +113,28 @@ describe("resolveSnapshotRef", () => {
expect(resolveSnapshotRef(ctx, "@e3", 4)).toEqual({
backendNodeId: 1234,
refKey: "e3",
kind: "dom",
capabilities: ["interact", "screenshot"],
});
});
it("rejects an operation outside the ref's declared capabilities", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const ctx = await sm.start("aa11");
ctx.refStore.set("e3", 1234, {
tabId: 4,
kind: "surface",
capabilities: ["screenshot"],
});
expect(resolveSnapshotRef(ctx, "@e3", 4, "interact")).toMatchObject({
code: "permission_denied",
message: "ref @e3 does not support interact",
data: {
reason: "ref_capability_denied",
required_capability: "interact",
kind: "surface",
},
});
});
});
@@ -24,7 +24,7 @@ export interface BorrowResponseMessage {
allowed: boolean;
}
export const CONFIRMATION_TIMEOUT_MS = 5000;
export const CONFIRMATION_TIMEOUT_MS = 60_000;
const EXIT_ANIMATION_MS = 150;
/** Matches BorrowConfirmationOverlay progress ring/bar transition (duration-1000). */
export const PROGRESS_TRANSITION_MS = 1000;
@@ -480,6 +480,13 @@ export async function requestBorrowConfirmation(
settle(false);
}, BACKGROUND_TIMEOUT_MS);
// Bring the user window to the front so the overlay is visible even
// when the Agent Window has stolen focus. This is fire-and-forget: if
// it fails the overlay / OS notification still give the user a path.
void windowsApi.update(notificationAnchor.windowId, { focused: true }).catch((err) => {
console.debug("[bsk borrow] proactive focus of user window failed", err);
});
// Surface the OS notification *before* messaging any candidate so the
// user has a parallel signal even if every content script is missing.
// The Allow / Deny buttons let the user authorize explicitly when the
+79 -38
View File
@@ -4,6 +4,7 @@ import type { Transport } from "@/transport/transport";
import type {
ClickParams,
ConsoleParams,
DownloadParams,
EmulateParams,
EvaluateParams,
FillParams,
@@ -28,10 +29,12 @@ import type {
ScreenshotParams,
SelectParams,
SnapshotParams,
UploadParams,
WaitForNavigationParams,
} from "@/transport/types";
import { isRequestFrame } from "@/transport/types";
import { handleConsole } from "./console";
import { handleDownload } from "./download";
import { type EmulateCdpRunner, handleEmulate } from "./emulate";
import { handleEvaluate } from "./evaluate";
import { handleRequestHelp } from "./human-loop";
@@ -51,7 +54,12 @@ import {
handleScreenshot,
handleSnapshot,
} from "./observation";
import { handleRecordAwait, handleRecordStart, handleRecordStop } from "./record";
import {
handleRecordAwait,
handleRecordStart,
handleRecordStop,
type RecordRuntimeDeps,
} from "./record";
import {
handleSessionStart,
handleSessionStop,
@@ -74,6 +82,7 @@ import {
type TabReturnParams,
type TabSelectParams,
} from "./tabs";
import { handleUpload } from "./upload";
import { handleWaitForNavigation } from "./waits";
import { handleWindowResize, type WindowResizeParams } from "./window";
@@ -99,6 +108,7 @@ export interface DispatcherDeps {
transport: Transport;
sessions: SessionManager;
cdp?: DispatcherCdpRunner;
recording?: RecordRuntimeDeps;
/**
* Invoked whenever a dispatched RPC may have changed the live
* session set (currently `tool.session_start` and
@@ -134,6 +144,7 @@ export class ToolDispatcher {
private readonly transport: Transport;
private readonly sessions: SessionManager;
private readonly cdp?: DispatcherCdpRunner;
private readonly recording?: RecordRuntimeDeps;
private readonly onSessionsChanged?: () => void;
private readonly onBrowserControlResumed?: (sessionId: string) => void;
private readonly approveBorrow?: BorrowConfirmationApprover;
@@ -153,6 +164,7 @@ export class ToolDispatcher {
this.transport = deps.transport;
this.sessions = deps.sessions;
this.cdp = deps.cdp;
this.recording = deps.recording;
this.onSessionsChanged = deps.onSessionsChanged;
this.onBrowserControlResumed = deps.onBrowserControlResumed;
this.approveBorrow = deps.approveBorrow;
@@ -375,7 +387,11 @@ export class ToolDispatcher {
? {
cdp: this.cdp,
tabsApi: chromeTabsCaptureApi,
conditionalSurfaceProbe: !this.hasHoverLatchForScope(hoverScope),
// Active hover probing is opt-in. A held hover latch still
// suppresses it, because probing would move the cursor off
// the element the caller is deliberately holding.
conditionalSurfaceProbe:
params.probe_hover === true && !this.hasHoverLatchForScope(hoverScope),
hoverProbeBypassOverlay: bypassOverlay,
}
: undefined,
@@ -505,6 +521,40 @@ export class ToolDispatcher {
),
signal,
);
case "tool.upload":
return this.withHoverReleaseForRequest(
req.params as UploadParams,
() =>
this.cdp
? handleUpload(this.sessions, req.params as UploadParams, {
cdp: this.cdp,
tabsApi: chromeTabsApi,
signal,
bypassOverlay,
})
: Promise.resolve({
code: "unsupported",
message: "upload requires CDP",
} satisfies RpcError),
signal,
);
case "tool.download":
return this.withHoverReleaseForRequest(
req.params as DownloadParams,
() =>
this.cdp
? handleDownload(this.sessions, req.params as DownloadParams, {
cdp: this.cdp,
tabsApi: chromeTabsApi,
signal,
bypassOverlay,
})
: Promise.resolve({
code: "unsupported",
message: "download requires CDP",
} satisfies RpcError),
signal,
);
case "tool.evaluate":
return handleEvaluate(
this.sessions,
@@ -531,44 +581,26 @@ export class ToolDispatcher {
signal,
});
case "tool.record_start":
return handleRecordStart(this.sessions, req.params as RecordStartParams, {
tabsApi: chromeTabsApi,
sendToTab: (tabId, msg) => chrome.tabs.sendMessage(tabId, msg),
bypassOverlay: async (tabId, enabled) => {
try {
await chrome.tabs.sendMessage(tabId, {
type: OVERLAY_AUTOMATION_BYPASS,
enabled,
});
} catch {
// Content script may be unavailable on restricted pages.
}
},
...(this.cdp ? { cdp: this.cdp } : {}),
signal,
});
return this.recording
? handleRecordStart(this.sessions, req.params as RecordStartParams, {
...this.recording,
signal,
})
: recordingRuntimeUnavailable();
case "tool.record_stop":
return handleRecordStop(this.sessions, req.params as RecordStopParams, {
tabsApi: chromeTabsApi,
sendToTab: (tabId, msg) => chrome.tabs.sendMessage(tabId, msg),
bypassOverlay: async (tabId, enabled) => {
try {
await chrome.tabs.sendMessage(tabId, {
type: OVERLAY_AUTOMATION_BYPASS,
enabled,
});
} catch {
// Content script may be unavailable on restricted pages.
}
},
signal,
});
return this.recording
? handleRecordStop(this.sessions, req.params as RecordStopParams, {
...this.recording,
signal,
})
: recordingRuntimeUnavailable();
case "tool.record_await":
return handleRecordAwait(this.sessions, req.params as RecordAwaitParams, {
tabsApi: chromeTabsApi,
sendToTab: (tabId, msg) => chrome.tabs.sendMessage(tabId, msg),
signal,
});
return this.recording
? handleRecordAwait(this.sessions, req.params as RecordAwaitParams, {
...this.recording,
signal,
})
: recordingRuntimeUnavailable();
default:
return {
code: "unknown_method",
@@ -704,6 +736,13 @@ function isRpcError(v: unknown): v is RpcError {
);
}
function recordingRuntimeUnavailable(): RpcError {
return {
code: "protocol_error",
message: "recording runtime is unavailable",
};
}
function sessionIdForBrowserControlMethod(req: RequestFrame): string | null {
switch (req.method) {
case "tool.tab_create":
@@ -722,6 +761,8 @@ function sessionIdForBrowserControlMethod(req: RequestFrame): string | null {
case "tool.fill":
case "tool.press":
case "tool.select":
case "tool.upload":
case "tool.download":
case "tool.evaluate":
case "tool.observe":
case "tool.request_help":
@@ -0,0 +1,375 @@
// Order-independent coordinator for one browser download. CDP supplies the
// exact target/frame intent while chrome.downloads supplies the download id
// and filename routing hook; neither event is assumed to arrive first.
import type { CdpTarget } from "@/browser-driver/frame-graph";
import type { ClickResult, RpcError, TransferEffectState } from "@/transport/types";
import { transferError } from "./errors";
import { type CdpRunner, isRpcError } from "./shared";
const CORRELATION_GRACE_MS = 750;
const UNIQUE_SETTLE_MS = 50;
const CLEANUP_RECONCILE_MS = 1_000;
const CLEANUP_POLL_MS = 25;
const SIZE_POLL_MS = 250;
type DeterminingFilenameListener = (
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true;
interface ListenerEvent<T> {
addListener(listener: T): void;
removeListener(listener: T): void;
}
export interface DownloadsApi {
onCreated: ListenerEvent<(item: chrome.downloads.DownloadItem) => void>;
onChanged: ListenerEvent<(delta: chrome.downloads.DownloadDelta) => void>;
onDeterminingFilename: ListenerEvent<DeterminingFilenameListener>;
search(query: chrome.downloads.DownloadQuery): Promise<chrome.downloads.DownloadItem[]>;
cancel(downloadId: number): Promise<void>;
removeFile(downloadId: number): Promise<void>;
}
export const chromeDownloadsApi: DownloadsApi = {
get onCreated() {
return chrome.downloads.onCreated;
},
get onChanged() {
return chrome.downloads.onChanged;
},
get onDeterminingFilename() {
return chrome.downloads.onDeterminingFilename;
},
search: (query) => chrome.downloads.search(query),
cancel: (id) => chrome.downloads.cancel(id),
removeFile: (id) => chrome.downloads.removeFile(id),
};
export interface DownloadCaptureOptions {
cdp: CdpRunner;
target: CdpTarget;
expectedFrameId?: string;
downloads: DownloadsApi;
browserRelativeDir: string;
maxByteSize?: number;
timeoutMs: number;
signal?: AbortSignal;
trigger(): Promise<ClickResult | RpcError>;
}
export interface DownloadCaptureResult {
click: ClickResult;
item: chrome.downloads.DownloadItem;
}
interface DownloadIntent {
url: string;
suggestedFilename: string;
frameId?: string;
}
interface DownloadCandidate {
item: chrome.downloads.DownloadItem;
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void;
suggested: boolean;
graceTimer: ReturnType<typeof setTimeout>;
}
function safeBasename(filename: string): string {
const basename = filename.split(/[\\/]/).pop()?.trim();
return basename && basename !== "." && basename !== ".." ? basename : "download";
}
function sameTarget(source: { tabId?: number; sessionId?: string }, target: CdpTarget): boolean {
return source.tabId === target.tabId && source.sessionId === target.sessionId;
}
function matchesIntent(item: chrome.downloads.DownloadItem, intent: DownloadIntent): boolean {
const urlMatches = item.url === intent.url || item.finalUrl === intent.url;
return urlMatches && safeBasename(item.filename) === safeBasename(intent.suggestedFilename);
}
function knownSize(item: chrome.downloads.DownloadItem): number | undefined {
if (item.fileSize >= 0) return item.fileSize;
if (item.totalBytes >= 0) return item.totalBytes;
return undefined;
}
function captureError(
message: string,
effectState: TransferEffectState,
phase: string,
cleanupFailed = false,
): RpcError {
return transferError("cdp_failed", "download_capture_failed", message, {
effectState,
phase,
...(cleanupFailed ? { cleanupState: "failed" } : {}),
});
}
async function cleanupClaimedDownload(downloads: DownloadsApi, downloadId: number): Promise<void> {
const lookup = async () => (await downloads.search({ id: downloadId }))[0];
let item = await lookup();
if (!item || item.state === "interrupted") return;
if (item.state === "complete") {
await downloads.removeFile(downloadId);
return;
}
let cancelError: unknown;
try {
await downloads.cancel(downloadId);
} catch (error) {
cancelError = error;
}
const deadline = Date.now() + CLEANUP_RECONCILE_MS;
while (true) {
item = await lookup();
if (!item || item.state === "interrupted") return;
if (item.state === "complete") {
await downloads.removeFile(downloadId);
return;
}
if (Date.now() >= deadline) {
if (cancelError instanceof Error) throw cancelError;
throw new Error("download remained in progress after cancellation");
}
await new Promise((resolve) =>
setTimeout(resolve, Math.min(CLEANUP_POLL_MS, Math.max(1, deadline - Date.now()))),
);
}
}
export async function captureBrowserDownload(
options: DownloadCaptureOptions,
): Promise<DownloadCaptureResult | RpcError> {
let click: ClickResult | undefined;
let intent: DownloadIntent | undefined;
let capturedId: number | undefined;
let settled = false;
let succeeded = false;
let failureResult: RpcError | undefined;
let uniquenessTimer: ReturnType<typeof setTimeout> | undefined;
let operationTimer: ReturnType<typeof setTimeout> | undefined;
let sizePoll: ReturnType<typeof setInterval> | undefined;
const candidates = new Map<number, DownloadCandidate>();
const createdItems = new Map<number, chrome.downloads.DownloadItem>();
let resolveCompletion!: (item: chrome.downloads.DownloadItem) => void;
let rejectCompletion!: (error: Error) => void;
const completion = new Promise<chrome.downloads.DownloadItem>((resolve, reject) => {
resolveCompletion = resolve;
rejectCompletion = reject;
});
const fail = (error: Error) => {
if (settled) return;
settled = true;
rejectCompletion(error);
};
const complete = (item: chrome.downloads.DownloadItem) => {
if (settled) return;
const size = knownSize(item);
if (size !== undefined && options.maxByteSize !== undefined && size > options.maxByteSize) {
fail(new Error(`download exceeds transfer limit ${options.maxByteSize}`));
return;
}
settled = true;
resolveCompletion(item);
};
const suggestDefault = (candidate: DownloadCandidate) => {
if (candidate.suggested) return;
candidate.suggested = true;
candidate.suggest();
};
const matchingCandidates = (): DownloadCandidate[] => {
const currentIntent = intent;
return currentIntent
? [...candidates.values()].filter(
(candidate) => !candidate.suggested && matchesIntent(candidate.item, currentIntent),
)
: [];
};
const claimUnique = () => {
uniquenessTimer = undefined;
if (settled || capturedId !== undefined || !intent) return;
const matches = matchingCandidates();
if (matches.length !== 1) {
if (matches.length > 1) {
for (const candidate of matches) suggestDefault(candidate);
fail(new Error("download attribution is ambiguous"));
}
return;
}
const candidate = matches[0];
candidate.suggested = true;
clearTimeout(candidate.graceTimer);
capturedId = candidate.item.id;
candidate.suggest({
filename: `${options.browserRelativeDir}/${safeBasename(intent.suggestedFilename)}`,
conflictAction: "overwrite",
});
const size = knownSize(candidate.item);
if (size !== undefined && options.maxByteSize !== undefined && size > options.maxByteSize) {
fail(new Error(`download exceeds transfer limit ${options.maxByteSize}`));
return;
}
const created = createdItems.get(candidate.item.id);
if (created?.state === "interrupted") {
fail(new Error(created.error ?? "download interrupted"));
} else if (created?.state === "complete") {
complete(created);
}
};
const reconcile = () => {
if (settled || capturedId !== undefined || !intent) return;
const matches = matchingCandidates();
if (matches.length > 1) {
for (const candidate of matches) suggestDefault(candidate);
fail(new Error("download attribution is ambiguous"));
return;
}
if (matches.length === 1 && !uniquenessTimer) {
uniquenessTimer = setTimeout(claimUnique, UNIQUE_SETTLE_MS);
}
};
const determiningListener: DeterminingFilenameListener = (item, suggest) => {
const candidate: DownloadCandidate = {
item,
suggest,
suggested: false,
graceTimer: setTimeout(() => {
suggestDefault(candidate);
candidates.delete(item.id);
if (intent && matchesIntent(item, intent) && capturedId === undefined) {
fail(new Error("download correlation grace elapsed before unique attribution"));
}
}, CORRELATION_GRACE_MS),
};
candidates.set(item.id, candidate);
reconcile();
return true;
};
const createdListener = (item: chrome.downloads.DownloadItem) => {
createdItems.set(item.id, item);
if (capturedId !== item.id) return;
if (item.state === "interrupted") {
fail(new Error(item.error ?? "download interrupted"));
} else if (item.state === "complete") {
complete(item);
}
};
const changedListener = async (delta: chrome.downloads.DownloadDelta) => {
if (capturedId === undefined || delta.id !== capturedId || settled) return;
if (delta.state?.current === "interrupted" || delta.error?.current) {
fail(new Error(delta.error?.current ?? "download interrupted"));
return;
}
if (delta.state?.current === "complete") {
try {
const [item] = await options.downloads.search({ id: delta.id });
if (item) complete(item);
else fail(new Error("completed download disappeared"));
} catch (err) {
fail(err instanceof Error ? err : new Error(String(err)));
}
}
};
const onAbort = () => fail(new DOMException("aborted", "AbortError"));
const cdpSubscription = options.cdp.onEvent?.((source, method, raw) => {
if (method !== "Page.downloadWillBegin" || !sameTarget(source, options.target)) return;
const event = raw as { url?: unknown; suggestedFilename?: unknown; frameId?: unknown };
if (typeof event.url !== "string" || typeof event.suggestedFilename !== "string") return;
if (options.expectedFrameId && event.frameId !== options.expectedFrameId) {
fail(new Error("download originated from a different frame"));
return;
}
if (intent) {
fail(new Error("download trigger produced more than one browser download intent"));
return;
}
intent = {
url: event.url,
suggestedFilename: event.suggestedFilename,
...(typeof event.frameId === "string" ? { frameId: event.frameId } : {}),
};
reconcile();
});
if (!cdpSubscription) {
return captureError("CDP download intent subscription unavailable", "none", "arm");
}
options.downloads.onDeterminingFilename.addListener(determiningListener);
options.downloads.onCreated.addListener(createdListener);
options.downloads.onChanged.addListener(changedListener);
options.signal?.addEventListener("abort", onAbort, { once: true });
operationTimer = setTimeout(
() => fail(new Error("download did not complete before timeout")),
options.timeoutMs,
);
sizePoll = setInterval(() => {
if (capturedId === undefined || settled || options.maxByteSize === undefined) return;
void options.downloads
.search({ id: capturedId })
.then(([item]) => {
if (!item || settled) return;
if (item.bytesReceived > (options.maxByteSize as number)) {
fail(new Error(`download exceeds transfer limit ${options.maxByteSize}`));
}
})
.catch((err) => fail(err instanceof Error ? err : new Error(String(err))));
}, SIZE_POLL_MS);
try {
const triggered = await options.trigger();
if (isRpcError(triggered)) {
void completion.catch(() => undefined);
const effect: TransferEffectState =
capturedId !== undefined ? "committed" : intent ? "unknown" : "none";
failureResult = {
...triggered,
data: { ...triggered.data, effect_state: effect, phase: "trigger" },
};
return failureResult;
}
click = triggered;
const item = await completion;
succeeded = true;
return { click, item };
} catch (err) {
const effect: TransferEffectState =
capturedId !== undefined ? "committed" : click ? "unknown" : "none";
failureResult = captureError(
err instanceof Error ? err.message : String(err),
effect,
capturedId !== undefined ? "download" : "attribution",
);
return failureResult;
} finally {
settled = true;
if (operationTimer) clearTimeout(operationTimer);
if (uniquenessTimer) clearTimeout(uniquenessTimer);
if (sizePoll) clearInterval(sizePoll);
options.signal?.removeEventListener("abort", onAbort);
options.downloads.onDeterminingFilename.removeListener(determiningListener);
options.downloads.onCreated.removeListener(createdListener);
options.downloads.onChanged.removeListener(changedListener);
cdpSubscription.dispose();
for (const candidate of candidates.values()) {
clearTimeout(candidate.graceTimer);
if (candidate.item.id !== capturedId) suggestDefault(candidate);
}
if (!succeeded && capturedId !== undefined) {
try {
await cleanupClaimedDownload(options.downloads, capturedId);
} catch {
if (failureResult?.data) failureResult.data.cleanup_state = "failed";
}
}
}
}
+64
View File
@@ -0,0 +1,64 @@
// Download orchestration: validate session/tab ownership, then delegate one
// browser-global chrome.downloads transaction to download-capture.ts.
import type { SessionManager } from "@/session-manager/manager";
import type { DownloadParams, DownloadResult, RpcError } from "@/transport/types";
import { captureBrowserDownload, chromeDownloadsApi, type DownloadsApi } from "./download-capture";
import { clickResolvedTarget, type InteractionDeps, resolveActionTarget } from "./interaction";
import { enforceAgentWindow, isRpcError, lookupSession, resolveTargetTab } from "./shared";
let downloadActive = false;
export type { DownloadsApi } from "./download-capture";
export interface DownloadDeps extends InteractionDeps {
downloads?: DownloadsApi;
}
export async function handleDownload(
manager: SessionManager,
params: DownloadParams,
deps: DownloadDeps,
): Promise<DownloadResult | RpcError> {
if (downloadActive) return { code: "invalid_params", message: "another bsk download is active" };
downloadActive = true;
try {
const ctx = lookupSession(manager, params, "download");
if (isRpcError(ctx)) return ctx;
const target = await resolveTargetTab(manager, ctx, params.tab_id, deps.tabsApi);
if (isRpcError(target)) return target;
const denied = enforceAgentWindow(ctx, target, "download");
if (denied) return denied;
if (!params.browser_relative_dir) {
return { code: "invalid_params", message: "download requires a daemon capability directory" };
}
const address = await resolveActionTarget(deps.cdp, ctx, target, params, "download");
if (isRpcError(address)) return address;
const capture = await captureBrowserDownload({
cdp: deps.cdp,
target: address.cdpTarget,
downloads: deps.downloads ?? chromeDownloadsApi,
browserRelativeDir: params.browser_relative_dir,
maxByteSize: params.max_byte_size,
timeoutMs: params.timeout_ms ?? 120_000,
signal: deps.signal,
expectedFrameId: address.frameId,
trigger: () => clickResolvedTarget(ctx, address, {}, deps),
});
if (isRpcError(capture)) return capture;
const { click, item } = capture;
return {
tab_id: target.tabId,
used_ref: click.used_ref,
used_selector: click.used_selector,
suggested_filename: item.filename.split(/[\\/]/).pop() ?? "download",
byte_size: item.fileSize >= 0 ? item.fileSize : item.totalBytes,
mime: item.mime || undefined,
danger: item.danger,
browser_path: item.filename,
};
} finally {
downloadActive = false;
}
}
+27 -1
View File
@@ -2,10 +2,23 @@
// rendering. Extension handlers attach reasons here; human-facing copy
// lives in bsk-cli `render_error.rs`.
import type { ErrorCode, RpcError, RpcErrorData, RpcErrorReason } from "@/transport/types";
import type {
ErrorCode,
RpcError,
RpcErrorData,
RpcErrorReason,
TransferCleanupState,
TransferEffectState,
} from "@/transport/types";
export type { RpcErrorData, RpcErrorReason };
export interface TransferErrorOptions {
effectState: TransferEffectState;
phase: string;
cleanupState?: TransferCleanupState;
}
export function rpcError(
code: ErrorCode,
reason: RpcErrorReason,
@@ -15,3 +28,16 @@ export function rpcError(
const data: RpcErrorData = { reason, ...extra };
return { code, message, data };
}
export function transferError(
code: ErrorCode,
reason: RpcErrorReason,
message: string,
options: TransferErrorOptions,
): RpcError {
return rpcError(code, reason, message, {
effect_state: options.effectState,
phase: options.phase,
...(options.cleanupState ? { cleanup_state: options.cleanupState } : {}),
});
}
@@ -0,0 +1,528 @@
// One upload transaction with an explicit browser-side commit boundary.
// Chrome interception prevents a native chooser from escaping automation;
// chooser events and a frame-scoped DOM probe are independent input-location
// signals, so event delivery is not required for standard file inputs.
import type { ClickResult, RpcError, TransferEffectState } from "@/transport/types";
import { transferError } from "./errors";
import type { ResolvedActionTarget } from "./interaction";
import { type CdpRunner, isRpcError, sendToCdpTarget } from "./shared";
const CLEANUP_TIMEOUT_MS = 1_000;
const DETACH_TIMEOUT_MS = 500;
const ACTIVATION_GRACE_MS = 1_000;
const PROBE_INTERVAL_MS = 20;
type UploadPhase =
| "arm_interception"
| "arm_input_probe"
| "trigger"
| "resolve_input"
| "set_files"
| "cleanup";
interface RuntimeReply {
result?: { value?: unknown; objectId?: string };
exceptionDetails?: { text?: string; exception?: { description?: string } };
}
interface ChooserEvent {
frameId?: string;
backendNodeId?: number;
mode?: "selectSingle" | "selectMultiple";
}
export interface FileInputTransactionOptions {
cdp: CdpRunner;
actionTarget: ResolvedActionTarget;
files: string[];
timeoutMs: number;
signal?: AbortSignal;
trigger(): Promise<ClickResult | RpcError>;
}
export interface FileInputTransactionResult {
click: ClickResult;
multiple: boolean;
}
class BoundedWaitError extends Error {
constructor(
readonly kind: "timeout" | "aborted",
message: string,
) {
super(message);
}
}
function remainingMs(deadline: number): number {
return Math.max(0, deadline - Date.now());
}
async function waitBounded<T>(
operation: () => Promise<T>,
deadline: number,
signal: AbortSignal | undefined,
timeoutMessage: string,
): Promise<T> {
const remaining = remainingMs(deadline);
if (signal?.aborted) throw new BoundedWaitError("aborted", "upload transaction aborted");
if (remaining === 0) throw new BoundedWaitError("timeout", timeoutMessage);
let timer: ReturnType<typeof setTimeout> | undefined;
let onAbort: (() => void) | undefined;
const boundary = new Promise<never>((_resolve, reject) => {
timer = setTimeout(() => reject(new BoundedWaitError("timeout", timeoutMessage)), remaining);
if (signal) {
onAbort = () => reject(new BoundedWaitError("aborted", "upload transaction aborted"));
signal.addEventListener("abort", onAbort, { once: true });
}
});
try {
const work = Promise.resolve().then(() => {
if (signal?.aborted) throw new BoundedWaitError("aborted", "upload transaction aborted");
if (remainingMs(deadline) === 0) throw new BoundedWaitError("timeout", timeoutMessage);
return operation();
});
return await Promise.race([work, boundary]);
} finally {
if (timer) clearTimeout(timer);
if (signal && onAbort) signal.removeEventListener("abort", onAbort);
}
}
function runtimeError(reply: RuntimeReply, fallback: string): Error | null {
if (!reply.exceptionDetails) return null;
return new Error(
reply.exceptionDetails.exception?.description ?? reply.exceptionDetails.text ?? fallback,
);
}
function transferFailure(
code: RpcError["code"],
reason: "file_input_probe_failed" | "file_input_not_activated" | "set_file_input_failed",
message: string,
effectState: TransferEffectState,
phase: UploadPhase,
): RpcError {
return transferError(code, reason, message, { effectState, phase });
}
function enrichFailure(
error: RpcError,
effectState: TransferEffectState,
phase: UploadPhase,
): RpcError {
return {
...error,
data: { ...error.data, effect_state: effectState, phase },
};
}
function sameTarget(
source: { tabId?: number; sessionId?: string },
target: { tabId: number; sessionId?: string },
): boolean {
return source.tabId === target.tabId && source.sessionId === target.sessionId;
}
async function delay(ms: number, signal?: AbortSignal): Promise<void> {
await waitBounded(
() => new Promise((resolve) => setTimeout(resolve, ms)),
Date.now() + ms + 1,
signal,
"upload activation probe timed out",
);
}
function waitErrorCode(error: unknown): RpcError["code"] {
if (!(error instanceof BoundedWaitError)) return "cdp_failed";
return error.kind === "aborted" ? "cancelled" : "timeout";
}
async function callOnTrigger<T = RuntimeReply>(
options: FileInputTransactionOptions,
objectId: string,
functionDeclaration: string,
args: unknown[],
returnByValue: boolean,
): Promise<T> {
return sendToCdpTarget<T>(options.cdp, options.actionTarget.cdpTarget, "Runtime.callFunctionOn", {
objectId,
functionDeclaration,
arguments: args.map((value) => ({ value })),
returnByValue,
awaitPromise: false,
});
}
export async function uploadThroughActivatedFileInput(
options: FileInputTransactionOptions,
): Promise<FileInputTransactionResult | RpcError> {
const deadline = Date.now() + options.timeoutMs;
const target = options.actionTarget.cdpTarget;
const objectGroup = `bsk-upload-${crypto.randomUUID()}`;
const stateKey = `__bskUpload_${crypto.randomUUID().replaceAll("-", "")}`;
const chooserEvents: ChooserEvent[] = [];
let interceptionState: "not_started" | "possibly_active" | "active" = "not_started";
let probeState: "not_started" | "possibly_active" | "active" = "not_started";
let triggerObjectId: string | undefined;
let outcome: FileInputTransactionResult | RpcError = transferFailure(
"protocol_error",
"file_input_probe_failed",
"upload transaction ended without an outcome",
"none",
"cleanup",
);
const chooserSubscription = options.cdp.onEvent?.((source, method, raw) => {
if (method !== "Page.fileChooserOpened" || !sameTarget(source, target)) return;
const event = raw as ChooserEvent;
chooserEvents.push(event);
});
try {
try {
interceptionState = "possibly_active";
await waitBounded(
() =>
sendToCdpTarget(options.cdp, target, "Page.setInterceptFileChooserDialog", {
enabled: true,
}),
deadline,
options.signal,
"arming native file chooser interception timed out",
);
interceptionState = "active";
} catch (err) {
outcome = transferFailure(
waitErrorCode(err),
"file_input_probe_failed",
err instanceof Error ? err.message : String(err),
"none",
"arm_interception",
);
return outcome;
}
try {
const resolved = await waitBounded(
() =>
sendToCdpTarget<{ object?: { objectId?: string } }>(
options.cdp,
target,
"DOM.resolveNode",
{
backendNodeId: options.actionTarget.backendNodeId,
objectGroup,
},
),
deadline,
options.signal,
"resolving upload trigger timed out",
);
triggerObjectId = resolved.object?.objectId;
if (!triggerObjectId) throw new Error("DOM.resolveNode returned no trigger objectId");
probeState = "possibly_active";
const armed = await waitBounded(
() =>
callOnTrigger<RuntimeReply>(
options,
triggerObjectId as string,
`function(key) {
const doc = this.ownerDocument;
const owner = doc.defaultView;
if (!owner) return false;
const state = { inputs: [], listener: null };
Object.defineProperty(owner, key, { value: state, configurable: true });
state.listener = event => {
const path = typeof event.composedPath === "function" ? event.composedPath() : [];
const candidate = path[0] || event.target;
if (candidate && candidate.nodeType === 1 &&
candidate.localName === "input" && candidate.type === "file") {
if (!state.inputs.includes(candidate)) state.inputs.push(candidate);
event.preventDefault();
}
};
doc.addEventListener("click", state.listener, true);
return true;
}`,
[stateKey],
true,
),
deadline,
options.signal,
"arming frame-scoped file input probe timed out",
);
const armError = runtimeError(armed, "failed to arm file input probe");
if (armError || armed.result?.value !== true) {
throw armError ?? new Error("file input probe did not arm");
}
probeState = "active";
} catch (err) {
outcome = transferFailure(
waitErrorCode(err),
"file_input_probe_failed",
err instanceof Error ? err.message : String(err),
"none",
"arm_input_probe",
);
return outcome;
}
let click: ClickResult | RpcError;
try {
click = await waitBounded(
options.trigger,
deadline,
options.signal,
"upload trigger timed out",
);
} catch (err) {
outcome = transferFailure(
waitErrorCode(err),
"file_input_probe_failed",
err instanceof Error ? err.message : String(err),
"none",
"trigger",
);
return outcome;
}
if (isRpcError(click)) {
outcome = enrichFailure(click, "none", "trigger");
return outcome;
}
try {
const activationDeadline = Math.min(deadline, Date.now() + ACTIVATION_GRACE_MS);
let summary: { count: number; multiple: boolean } = { count: 0, multiple: false };
while (Date.now() < activationDeadline) {
if (chooserEvents.length > 0) break;
const reply = await waitBounded(
() =>
callOnTrigger<RuntimeReply>(
options,
triggerObjectId as string,
`function(key) {
const state = this.ownerDocument.defaultView?.[key];
return state
? { count: state.inputs.length, multiple: state.inputs[0]?.multiple === true }
: { count: 0, multiple: false };
}`,
[stateKey],
true,
),
activationDeadline,
options.signal,
"inspecting activated file input timed out",
);
const summaryError = runtimeError(reply, "failed to inspect activated file input");
if (summaryError) throw summaryError;
const value = reply.result?.value as { count?: unknown; multiple?: unknown } | undefined;
summary = {
count: typeof value?.count === "number" ? value.count : 0,
multiple: value?.multiple === true,
};
if (summary.count > 0) break;
await delay(Math.min(PROBE_INTERVAL_MS, remainingMs(activationDeadline)), options.signal);
}
if (chooserEvents.length > 1) {
outcome = transferFailure(
"unsupported",
"file_input_not_activated",
"upload trigger activated more than one file chooser",
"none",
"resolve_input",
);
return outcome;
}
const chooser = chooserEvents[0];
let backendNodeId: number | undefined;
let multiple = summary.multiple;
if (chooser) {
if (options.actionTarget.frameId && chooser.frameId !== options.actionTarget.frameId) {
outcome = transferFailure(
"unsupported",
"file_input_not_activated",
"upload trigger activated a file chooser in a different frame",
"none",
"resolve_input",
);
return outcome;
}
if (typeof chooser.backendNodeId !== "number") {
outcome = transferFailure(
"unsupported",
"file_input_not_activated",
"upload trigger invoked a non-input file picker",
"none",
"resolve_input",
);
return outcome;
}
backendNodeId = chooser.backendNodeId;
multiple = chooser.mode === "selectMultiple";
} else {
if (summary.count !== 1) {
outcome = transferFailure(
"unsupported",
"file_input_not_activated",
summary.count === 0
? "upload trigger did not activate an input[type=file]"
: "upload trigger activated more than one input[type=file]",
"none",
"resolve_input",
);
return outcome;
}
const input = await waitBounded(
() =>
callOnTrigger<RuntimeReply>(
options,
triggerObjectId as string,
`function(key) { return this.ownerDocument.defaultView?.[key]?.inputs[0]; }`,
[stateKey],
false,
),
deadline,
options.signal,
"resolving activated file input timed out",
);
const inputError = runtimeError(input, "failed to resolve activated file input object");
if (inputError) throw inputError;
if (!input.result?.objectId) throw new Error("activated file input returned no objectId");
const described = await waitBounded(
() =>
sendToCdpTarget<{ node?: { backendNodeId?: number } }>(
options.cdp,
target,
"DOM.describeNode",
{ objectId: input.result?.objectId },
),
deadline,
options.signal,
"describing activated file input timed out",
);
backendNodeId = described.node?.backendNodeId;
if (typeof backendNodeId !== "number") {
throw new Error("DOM.describeNode returned no file input backendNodeId");
}
}
if (!multiple && options.files.length !== 1) {
outcome = enrichFailure(
{ code: "invalid_params", message: "file input accepts exactly one file" },
"none",
"resolve_input",
);
return outcome;
}
try {
await waitBounded(
() =>
sendToCdpTarget(options.cdp, target, "DOM.setFileInputFiles", {
files: options.files,
backendNodeId,
}),
deadline,
options.signal,
"setting file input files timed out",
);
} catch (err) {
outcome = transferFailure(
waitErrorCode(err),
"set_file_input_failed",
err instanceof Error ? err.message : String(err),
"unknown",
"set_files",
);
return outcome;
}
outcome = { click, multiple };
return outcome;
} catch (err) {
outcome = transferFailure(
waitErrorCode(err),
"file_input_probe_failed",
err instanceof Error ? err.message : String(err),
"none",
"resolve_input",
);
return outcome;
}
} finally {
chooserSubscription?.dispose();
const cleanupDeadline = Date.now() + CLEANUP_TIMEOUT_MS;
const cleanupTasks: Promise<boolean>[] = [];
const cleanup = async (operation: () => Promise<unknown>, message: string) => {
try {
await waitBounded(operation, cleanupDeadline, undefined, message);
return true;
} catch {
return false;
}
};
if (probeState !== "not_started" && triggerObjectId) {
cleanupTasks.push(
cleanup(
() =>
callOnTrigger(
options,
triggerObjectId as string,
`function(key) {
const owner = this.ownerDocument.defaultView;
const state = owner?.[key];
if (state?.listener) this.ownerDocument.removeEventListener("click", state.listener, true);
if (owner) delete owner[key];
}`,
[stateKey],
true,
),
"cleaning file input probe timed out",
),
);
}
if (interceptionState !== "not_started") {
cleanupTasks.push(
cleanup(
() =>
sendToCdpTarget(options.cdp, target, "Page.setInterceptFileChooserDialog", {
enabled: false,
cancel: true,
}),
"disabling file chooser interception timed out",
),
);
}
let cleanupFailed = (await Promise.all(cleanupTasks)).some((completed) => !completed);
if (!cleanupFailed) {
cleanupFailed = !(await cleanup(
() => sendToCdpTarget(options.cdp, target, "Runtime.releaseObjectGroup", { objectGroup }),
"releasing upload object group timed out",
));
}
const effect = isRpcError(outcome)
? (outcome.data?.effect_state as TransferEffectState | undefined)
: "committed";
if (effect === "unknown" || cleanupFailed) {
try {
await waitBounded(
() => options.cdp.detach?.(target.tabId) ?? Promise.resolve(),
Date.now() + DETACH_TIMEOUT_MS,
undefined,
"detaching upload target timed out",
);
} catch {
cleanupFailed = true;
}
}
if (cleanupFailed && isRpcError(outcome)) {
outcome.data = { ...outcome.data, cleanup_state: "failed" };
}
}
}
+40 -10
View File
@@ -40,6 +40,7 @@ import {
enforceAgentWindow,
isRpcError,
lookupSession,
type ResolvedTargetTab,
resolveTargetTab,
} from "./shared";
import { resolveSnapshotRef } from "./snapshot-ref";
@@ -56,6 +57,15 @@ export interface InteractionDeps {
keepOverlayBypassAfterHover?: boolean;
}
export interface ResolvedActionTarget {
tab: ResolvedTargetTab;
backendNodeId: number;
cdpTarget: CdpTarget;
frameId?: string;
usedRef?: string;
usedSelector?: string;
}
const DEFAULT_TIMEOUT_MS = 30_000;
const DEFAULT_HOVER_SETTLE_MS = 200;
@@ -128,7 +138,7 @@ async function wait(ms: number, signal?: AbortSignal): Promise<void> {
* `RpcError` if the caller supplied neither (or both), or if neither
* lookup matched.
*/
async function resolveBackendNode(
export async function resolveBackendNode(
cdp: CdpRunner,
ctx: SessionContext,
target: { tabId: number },
@@ -159,7 +169,7 @@ async function resolveBackendNode(
};
}
if (hasRef) {
const resolved = resolveSnapshotRef(ctx, params.ref as string, target.tabId);
const resolved = resolveSnapshotRef(ctx, params.ref as string, target.tabId, "interact");
if (isRpcError(resolved)) return resolved;
return {
backendNodeId: resolved.backendNodeId,
@@ -215,6 +225,17 @@ async function resolveBackendNode(
}
}
export async function resolveActionTarget(
cdp: CdpRunner,
ctx: SessionContext,
target: ResolvedTargetTab,
params: { ref?: string; selector?: string },
toolName: string,
): Promise<ResolvedActionTarget | RpcError> {
const node = await resolveBackendNode(cdp, ctx, target, params, toolName);
return isRpcError(node) ? node : { tab: target, ...node };
}
// ---------------------------------------------------------------------------
// tool.click
// ---------------------------------------------------------------------------
@@ -233,10 +254,19 @@ export async function handleClick(
if (isRpcError(target)) return target;
const denied = enforceAgentWindow(ctx, target, "click");
if (denied) return denied;
const dialogCursor = markDialogCursor(deps.cdp, target.tabId);
const resolved = await resolveActionTarget(deps.cdp, ctx, target, params, "click");
if (isRpcError(resolved)) return resolved;
return clickResolvedTarget(ctx, resolved, params, deps);
}
const node = await resolveBackendNode(deps.cdp, ctx, target, params, "click");
if (isRpcError(node)) return node;
export async function clickResolvedTarget(
ctx: SessionContext,
resolved: ResolvedActionTarget,
params: Pick<ClickParams, "button" | "click_count" | "modifiers">,
deps: InteractionDeps,
): Promise<ClickResult | RpcError> {
const { tab: target } = resolved;
const dialogCursor = markDialogCursor(deps.cdp, target.tabId);
if (throwIfAborted(deps.signal)) {
return { code: "cancelled", message: "click aborted" };
@@ -247,9 +277,9 @@ export async function handleClick(
deps.cdp,
target.tabId,
{
target: node.cdpTarget,
backendNodeId: node.backendNodeId,
...(node.frameId ? { frameId: node.frameId } : {}),
target: resolved.cdpTarget,
backendNodeId: resolved.backendNodeId,
...(resolved.frameId ? { frameId: resolved.frameId } : {}),
},
{ scrollIntoView: true },
);
@@ -337,8 +367,8 @@ export async function handleClick(
return attachDialogs(deps.cdp, target.tabId, dialogCursor, {
tab_id: target.tabId,
used_ref: node.usedRef,
used_selector: node.usedSelector,
used_ref: resolved.usedRef,
used_selector: resolved.usedSelector,
x: centre.x,
y: centre.y,
});
File diff suppressed because it is too large Load Diff
+181 -76
View File
@@ -18,8 +18,18 @@ import {
type RecordStepAck,
type RecordStopMessage,
} from "@/lib/record-bridge";
import {
type RecordFrameCoordinator,
type RecordingCaptureScope,
recordFrameCoordinator,
} from "@/lib/recording/frame-coordinator";
import { RecordingObservationRuntime } from "@/lib/recording/recording-runtime";
import { appendRecordedPayload, observeRecordedNavigation } from "@/lib/recording/step-buffer";
import {
appendRecordedPayload,
observeRecordedNavigation,
type RecordingStepBuffer,
} from "@/lib/recording/step-buffer";
import { RecordingTabCoordinator, type TabActivation } from "@/lib/recording/tab-coordinator";
import { buildTraceV2 } from "@/lib/recording/trace-reducer-v2";
import type { RecordingDraftStep } from "@/lib/recording/types";
import type { SessionManager } from "@/session-manager/manager";
@@ -48,7 +58,7 @@ import {
interface ActiveRecording {
requestId: string;
tabId: number;
tabs: RecordingTabCoordinator;
agentWindowId: number;
startUrl?: string;
purpose?: string;
@@ -56,14 +66,12 @@ interface ActiveRecording {
startedAt: string;
startedAtMs: number;
traceVersion: 2 | 3;
supportsTabSwitchSteps: boolean;
finishPromise: Promise<RecordedTrace>;
resolveFinish: (trace: RecordedTrace) => void;
rejectFinish: (err: Error) => void;
settled: boolean;
finishAttempt: Promise<RecordedTrace | null> | null;
currentUrl?: string;
pendingNavigation: boolean;
pendingNavigationDeadline?: number;
observation: RecordingObservationRuntime | null;
stoppedBy: StopReason;
/** Navigation callbacks tracked from event receipt through action enqueue. */
@@ -79,8 +87,17 @@ interface ActiveRecording {
lastStepSequenceByProducer: Map<string, number>;
}
function enqueueRecordingAction(recording: ActiveRecording, task: () => Promise<void>): void {
recording.actionQueue = recording.actionQueue.then(task, task).catch(() => {});
function enqueueRecordingAction(
recording: ActiveRecording,
task: () => Promise<void>,
): Promise<void> {
const queued = recording.actionQueue.then(task, task);
recording.actionQueue = queued.catch(() => {});
return queued;
}
function isRecordingFinishing(recording: ActiveRecording): boolean {
return recording.settled || recording.finishAttempt !== null;
}
const recordings = new Map<string, ActiveRecording>();
@@ -163,9 +180,11 @@ async function sendRecordStartWithAck(
tabId: number,
msg: RecordStartMessage,
sendToTab: RecordDeps["sendToTab"],
cancelled?: () => boolean,
): Promise<void> {
let lastError: unknown;
for (let attempt = 0; attempt < RECORD_START_RETRIES; attempt += 1) {
if (cancelled?.()) throw new Error("record start cancelled");
try {
const response = await sendToTab(tabId, msg);
if (isRecordStartAck(response)) return;
@@ -173,6 +192,7 @@ async function sendRecordStartWithAck(
} catch (err) {
lastError = err;
}
if (cancelled?.()) throw new Error("record start cancelled");
if (attempt + 1 < RECORD_START_RETRIES) {
await sleep(RECORD_START_RETRY_DELAY_MS);
}
@@ -198,6 +218,7 @@ function buildTrace(recording: ActiveRecording): RecordedTrace {
startUrl: recording.startUrl,
stoppedBy: recording.stoppedBy,
bskVersion: BSK_TRACE_VERSION,
includeTabSwitches: recording.supportsTabSwitchSteps,
});
}
if (recording.traceVersion === 3) {
@@ -211,14 +232,46 @@ function buildTrace(recording: ActiveRecording): RecordedTrace {
});
}
function stepBufferFor(
recording: ActiveRecording,
tabId: number,
fallbackUrl?: string,
): RecordingStepBuffer {
return {
steps: recording.steps,
navigation: recording.tabs.navigation(tabId, fallbackUrl),
};
}
async function activateRecordingTab(
recording: ActiveRecording,
targetTabId: number,
): Promise<void> {
const previousTabId = recording.tabs.currentTabId;
if (targetTabId === previousTabId) return;
const transition = recording.observation
? await recording.observation.captureTabTransition(previousTabId, targetTabId)
: {};
const { targetUrl, ...stateLinks } = transition;
const draft: Extract<RecordingDraftStep, { op: "switch_tab" }> = {
op: "switch_tab",
...stateLinks,
};
recording.steps.push(draft);
recording.observation?.bindTabTransition(draft, recording.steps.length);
recording.tabs.commit(targetTabId, targetUrl);
}
async function processRecordedStep(
recording: ActiveRecording,
draftIndex: number,
tabId: number,
producerId?: string,
): Promise<void> {
if (!recording.observation) return;
try {
await recording.observation.processDraft(tabId, recording.steps, draftIndex);
await recording.observation.processDraft(tabId, recording.steps, draftIndex, producerId);
} catch (err) {
console.warn(`[bsk record] observation failed for step ${draftIndex + 1}`, err);
}
@@ -231,16 +284,25 @@ export interface RecordDeps {
msg: RecordStartMessage | RecordStopMessage | RecordCancelMessage,
): Promise<unknown>;
bypassOverlay?: (tabId: number, enabled: boolean) => Promise<void>;
frameCoordinator?: Pick<
RecordFrameCoordinator,
"begin" | "armTab" | "sourceFor" | "stop" | "cancel"
>;
cdp?: CdpRunner;
signal?: AbortSignal;
}
export type RecordRuntimeDeps = Omit<RecordDeps, "frameCoordinator" | "signal"> & {
frameCoordinator: NonNullable<RecordDeps["frameCoordinator"]>;
};
let defaultDeps: RecordDeps | null = null;
function getDefaultDeps(): RecordDeps {
if (!defaultDeps) {
defaultDeps = {
tabsApi: chromeTabsApi,
sendToTab: (tabId, msg) => chrome.tabs.sendMessage(tabId, msg),
frameCoordinator: recordFrameCoordinator,
};
}
return defaultDeps;
@@ -307,8 +369,15 @@ export function attachRecordStepListener(deps: RecordDeps = getDefaultDeps()): (
if (!isRecordStepMessage(message)) return false;
for (const recording of recordings.values()) {
if (recording.requestId !== message.requestId) continue;
const sourceTabId = sender.tab?.id ?? recording.tabId;
const producerKey = `${sourceTabId}:${message.producerId}`;
const source: RecordingCaptureScope | null | undefined = deps.frameCoordinator
? deps.frameCoordinator.sourceFor(message.requestId, message.producerId, sender)
: undefined;
if (deps.frameCoordinator && !source) return false;
const sourceTabId = source?.tabId ?? sender.tab?.id ?? recording.tabs.currentTabId;
const sourceWasActive = sender.tab?.active ?? sourceTabId === recording.tabs.activeTabId;
const producerKey = source
? `${source.tabId}:${source.documentId}:${source.producerId}`
: `${sourceTabId}:${message.producerId}`;
const expectedSequence = (recording.lastStepSequenceByProducer.get(producerKey) ?? 0) + 1;
if (message.sequence < expectedSequence) {
sendResponse({ ok: true, sequence: message.sequence });
@@ -325,11 +394,21 @@ export function attachRecordStepListener(deps: RecordDeps = getDefaultDeps()): (
recording.lastStepSequenceByProducer.set(producerKey, message.sequence);
enqueueRecordingAction(recording, async () => {
await recording.observation?.flushRedirects();
if (sourceTabId !== recording.tabs.currentTabId) {
if (!sourceWasActive) return;
if (sourceTabId !== recording.tabs.activeTabId)
recording.tabs.noteActivation(sourceTabId);
await activateRecordingTab(recording, sourceTabId);
}
await recording.observation?.flushRedirects(sourceTabId);
const targetHint = message.step.geometry ? { geometry: message.step.geometry } : undefined;
const draftIndex = appendRecordedPayload(recording, message.step, targetHint);
const draftIndex = appendRecordedPayload(
stepBufferFor(recording, sourceTabId, message.step.page_url),
message.step,
targetHint,
);
if (draftIndex !== null) {
await processRecordedStep(recording, draftIndex, sourceTabId);
await processRecordedStep(recording, draftIndex, sourceTabId, source?.producerId);
}
});
sendResponse({ ok: true, sequence: message.sequence });
@@ -359,7 +438,7 @@ export function attachRecordFinishListener(deps: RecordDeps = getDefaultDeps()):
function findRecordingByTabId(tabId: number): ActiveRecording | null {
for (const recording of recordings.values()) {
if (recording.tabId === tabId && !recording.settled) return recording;
if (recording.tabs.currentTabId === tabId && !recording.settled) return recording;
}
return null;
}
@@ -396,7 +475,7 @@ async function clearRearmTimersForRecording(
recording: ActiveRecording,
deps: RecordDeps,
): Promise<void> {
clearRearmTimer(recording.tabId);
clearRearmTimer(recording.tabs.currentTabId);
try {
const tabs = await deps.tabsApi.query({ windowId: recording.agentWindowId });
for (const tab of tabs) {
@@ -411,13 +490,16 @@ async function stopRecordingOnAllAgentTabs(
recording: ActiveRecording,
deps: RecordDeps,
): Promise<void> {
if (deps.frameCoordinator && !(await deps.frameCoordinator.stop(recording.requestId))) {
throw new Error("failed to flush one or more recording documents");
}
const stopMsg: RecordStopMessage = { type: RECORD_STOP, requestId: recording.requestId };
let tabIds = [recording.tabId];
let tabIds = [recording.tabs.currentTabId];
try {
const tabs = await deps.tabsApi.query({ windowId: recording.agentWindowId });
tabIds = [
...new Set([
recording.tabId,
recording.tabs.currentTabId,
...tabs.flatMap((tab) => (typeof tab.id === "number" ? [tab.id] : [])),
]),
];
@@ -428,11 +510,11 @@ async function stopRecordingOnAllAgentTabs(
for (const tabId of tabIds) {
try {
const response = await deps.sendToTab(tabId, stopMsg);
if (tabId === recording.tabId && !isRecordStartAck(response)) {
if (tabId === recording.tabs.currentTabId && !isRecordStartAck(response)) {
throw new Error("content script did not confirm recorded steps");
}
} catch {
if (tabId === recording.tabId) {
if (tabId === recording.tabs.currentTabId) {
throw new Error("failed to flush recorded steps");
}
}
@@ -450,22 +532,39 @@ async function rearmRecording(
recording: ActiveRecording,
targetTabId: number,
deps: RecordDeps,
activation?: TabActivation,
): Promise<boolean> {
// Do NOT toggle automation-bypass here: each retry used to increment the
// content-script counter, and a single stop decrement left the ControlOverlay
// stuck with pointer-events:none (page usable, Interrupt dead). RecordOverlay
// already hides the control chrome while activeRecord is set.
const isFinishing = () => isRecordingFinishing(recording);
for (let attempt = 0; attempt < RECORD_REARM_MAX_ATTEMPTS; attempt += 1) {
if (isFinishing()) return false;
const startMsg: RecordStartMessage = {
type: RECORD_START,
requestId: recording.requestId,
startedAtMs: recording.startedAtMs,
};
try {
await sendRecordStartWithAck(targetTabId, startMsg, deps.sendToTab);
recording.tabId = targetTabId;
if (deps.frameCoordinator) {
const frameStarted = await deps.frameCoordinator.armTab(recording.requestId, targetTabId);
if (!frameStarted) throw new Error("recording document did not start");
}
await sendRecordStartWithAck(targetTabId, startMsg, deps.sendToTab, isFinishing);
if (isFinishing()) return false;
if (activation) {
if (!recording.tabs.isLatest(activation)) return true;
await enqueueRecordingAction(recording, async () => {
if (isFinishing() || !recording.tabs.isLatest(activation)) {
return;
}
await activateRecordingTab(recording, targetTabId);
});
}
return true;
} catch {
if (isFinishing()) return false;
if (attempt + 1 < RECORD_REARM_MAX_ATTEMPTS) {
await sleep(RECORD_REARM_RETRY_DELAY_MS);
}
@@ -474,7 +573,7 @@ async function rearmRecording(
return false;
}
function scheduleRearmForTab(tabId: number, deps: RecordDeps): void {
function scheduleRearmForTab(tabId: number, deps: RecordDeps, activation?: TabActivation): void {
const existing = rearmTimers.get(tabId);
if (existing) clearTimeout(existing);
rearmTimers.set(
@@ -483,7 +582,7 @@ function scheduleRearmForTab(tabId: number, deps: RecordDeps): void {
rearmTimers.delete(tabId);
void (async () => {
const current = await findRecordingForTab(tabId, deps);
if (current) await rearmRecording(current, tabId, deps);
if (current) await rearmRecording(current, tabId, deps, activation);
})();
}, RECORD_REARM_DEBOUNCE_MS),
);
@@ -495,14 +594,20 @@ export function attachRecordTabListener(deps: RecordDeps = getDefaultDeps()): ()
const windowId = tab.windowId;
if (tabId === undefined || windowId === undefined) return;
for (const recording of recordings.values()) {
if (recording.settled || recording.agentWindowId !== windowId) continue;
if (isRecordingFinishing(recording) || recording.agentWindowId !== windowId) continue;
scheduleRearmForTab(tabId, deps);
return;
}
};
const onActivated = (activeInfo: chrome.tabs.TabActiveInfo) => {
scheduleRearmForTab(activeInfo.tabId, deps);
for (const recording of recordings.values()) {
if (isRecordingFinishing(recording) || recording.agentWindowId !== activeInfo.windowId)
continue;
const activation = recording.tabs.noteActivation(activeInfo.tabId);
scheduleRearmForTab(activeInfo.tabId, deps, activation);
return;
}
};
chrome.tabs.onCreated.addListener(onCreated);
@@ -522,54 +627,39 @@ export function attachRecordNavigationListener(deps: RecordDeps = getDefaultDeps
transitionQualifiers?: string[],
) => {
if (!url) return;
const direct = findRecordingByTabId(tabId);
const candidates = direct
? direct.acceptingNavigation
? [direct]
: []
: [...recordings.values()].filter(
(recording) => !recording.settled && recording.acceptingNavigation,
const candidates = [...recordings.values()].filter(
(recording) =>
!recording.settled && recording.acceptingNavigation && recording.tabs.activeTabId === tabId,
);
for (const recording of candidates) {
const queued = enqueueRecordingAction(recording, async () => {
if (tabId !== recording.tabs.currentTabId) {
await activateRecordingTab(recording, tabId);
}
const result = observeRecordedNavigation(
stepBufferFor(recording, tabId, url),
url,
causedByAction,
transitionType,
transitionQualifiers,
);
if (candidates.length === 0) return;
let resolveTracked!: () => void;
const tracked = new Promise<void>((resolve) => {
resolveTracked = resolve;
});
for (const candidate of candidates) candidate.navigationCallbacks.add(tracked);
void (async () => {
try {
const recording = await findRecordingForTab(tabId, deps);
if (!recording || !recording.acceptingNavigation || !candidates.includes(recording)) {
if (result.kind === "coalesce_redirect") {
recording.observation?.scheduleRedirect(tabId, recording.steps, result.url);
return;
}
enqueueRecordingAction(recording, async () => {
const result = observeRecordedNavigation(
recording,
url,
causedByAction,
transitionType,
transitionQualifiers,
);
if (result.kind === "coalesce_redirect") {
recording.observation?.scheduleRedirect(tabId, recording.steps, result.url);
return;
}
if (result.kind === "noop") return;
recording.observation?.clearRedirect(tabId);
if (result.kind === "appended") {
await processRecordedStep(recording, result.index, tabId);
} else {
recording.observation?.scheduleSettle(tabId, recording.steps, result.index);
}
});
} finally {
for (const candidate of candidates) candidate.navigationCallbacks.delete(tracked);
resolveTracked();
}
})();
if (result.kind === "noop") return;
recording.observation?.clearRedirect(tabId);
if (result.kind === "appended") {
await processRecordedStep(recording, result.index, tabId);
} else {
recording.observation?.scheduleSettle(tabId, recording.steps, result.index);
}
});
const tracked = queued.catch(() => {});
recording.navigationCallbacks.add(tracked);
void tracked.finally(() => recording.navigationCallbacks.delete(tracked));
}
};
const onMainFrameComplete = (tabId: number, url?: string) => {
void (async () => {
@@ -651,7 +741,8 @@ export function attachRecordQueryListener(deps: RecordDeps = getDefaultDeps()):
sendResponse({ active: false });
return;
}
await rearmRecording(recording, tabId, deps);
const activation = sender.tab?.active ? recording.tabs.noteActivation(tabId) : undefined;
await rearmRecording(recording, tabId, deps, activation);
sendResponse({
active: true,
requestId: recording.requestId,
@@ -716,7 +807,7 @@ async function finishRecordingAttempt(
if (!(await drainRecordingToStability(recording))) {
return null;
}
await recording.observation?.settleTrailing(recording.tabId, recording.steps);
await recording.observation?.settleTrailing(recording.tabs.currentTabId, recording.steps);
recording.settled = true;
recordings.delete(sessionId);
@@ -768,7 +859,7 @@ export async function handleRecordStart(
const redactValues = params.redact_values ?? false;
recordings.set(params.session_id, {
requestId,
tabId: target.tabId,
tabs: new RecordingTabCoordinator(target.tabId, navigateUrl),
agentWindowId: ctx.agentWindowId,
startUrl: navigateUrl,
...(params.purpose ? { purpose: params.purpose } : {}),
@@ -776,14 +867,12 @@ export async function handleRecordStart(
startedAt: new Date(startedAtMs).toISOString(),
startedAtMs,
traceVersion: traceVersionOrErr,
supportsTabSwitchSteps: params.supports_tab_switch_steps === true,
finishPromise,
resolveFinish,
rejectFinish,
settled: false,
finishAttempt: null,
currentUrl: navigateUrl,
pendingNavigation: false,
pendingNavigationDeadline: undefined,
observation:
traceVersionOrErr === 3 && deps.cdp
? new RecordingObservationRuntime({
@@ -799,12 +888,23 @@ export async function handleRecordStart(
actionQueue: Promise.resolve(),
lastStepSequenceByProducer: new Map(),
});
deps.frameCoordinator?.begin(requestId, startedAtMs, target.tabId, async (scope) => {
const recording = recordings.get(params.session_id);
if (!recording || recording.requestId !== requestId || recording.settled) return;
try {
await recording.observation?.refreshDocument(scope.tabId, scope.producerId);
} catch {
// The normal action-time capture remains available if a child Document
// is replaced while its readiness refresh is in flight.
}
});
// Observe navigations for the whole recording lifetime; attach before
// optional navigate so the destination load can rearm capture.
ensureBrowserObservationListeners(deps);
const abortPending = async (notifyContent: boolean) => {
recordings.get(params.session_id)?.observation?.cancel();
deps.frameCoordinator?.cancel(requestId);
recordings.delete(params.session_id);
releaseBrowserObservationListenersIfIdle();
if (notifyContent) {
@@ -886,7 +986,7 @@ export async function handleRecordStart(
return cancelledError();
}
active.startUrl = startUrl;
active.currentUrl = startUrl;
active.tabs.navigation(target.tabId, startUrl).currentUrl = startUrl;
if (isContentScriptRestrictedUrl(startUrl)) {
await abortPending(false);
@@ -921,6 +1021,10 @@ export async function handleRecordStart(
const startMsg: RecordStartMessage = { type: RECORD_START, requestId, startedAtMs };
try {
if (deps.frameCoordinator) {
const frameStarted = await deps.frameCoordinator.armTab(requestId, target.tabId);
if (!frameStarted) throw new Error("top recording document did not start");
}
await sendRecordStartWithAck(target.tabId, startMsg, deps.sendToTab);
} catch {
await abortPending(true);
@@ -1040,6 +1144,7 @@ export function clearRecordingForSession(sessionId: string): void {
}
void clearRearmTimersForRecording(recording, getDefaultDeps());
if (!recording.settled) {
getDefaultDeps().frameCoordinator?.cancel(recording.requestId);
recording.observation?.cancel();
recording.settled = true;
recording.rejectFinish(new Error("recording cleared"));
+1
View File
@@ -53,6 +53,7 @@ export type { DialogCursor };
export interface CdpRunner {
send<T = unknown>(tabId: number, method: string, params?: object): Promise<T>;
sendToTarget?<T = unknown>(target: CdpTarget, method: string, params?: object): Promise<T>;
detach?(tabId: number): Promise<void>;
getFrameGraph?(tabId: number): Promise<CdpFrameGraph>;
ensureAttachedToUrl?(tabId: number, expectedUrl: string | undefined): Promise<void>;
trackSessionTab?(sessionId: string, tabId: number): void;
+17 -1
View File
@@ -2,8 +2,9 @@
// compound CDP node identity via the session RefStore, and build stable
// `ref_not_found` errors for hard-failure tool paths.
import type { Rect } from "@browser-skill/vom";
import type { SessionContext } from "@/session-manager/manager";
import { normaliseRef } from "@/session-manager/ref-store";
import { normaliseRef, type RefCapability, type RefTargetKind } from "@/session-manager/ref-store";
import type { RpcError } from "@/transport/types";
import { rpcError } from "./errors";
@@ -12,6 +13,9 @@ export interface SnapshotRefLookup {
refKey: string;
frameId?: string;
cdpSessionId?: string;
visibleRect?: Rect;
kind: RefTargetKind;
capabilities: RefCapability[];
}
function refEntryForTab(ctx: SessionContext, refKey: string, tabId: number) {
@@ -37,6 +41,9 @@ export function lookupSnapshotRef(
refKey,
...(entry.frameId ? { frameId: entry.frameId } : {}),
...(entry.cdpSessionId ? { cdpSessionId: entry.cdpSessionId } : {}),
...(entry.visibleRect ? { visibleRect: entry.visibleRect } : {}),
kind: entry.kind,
capabilities: entry.capabilities,
};
}
@@ -49,6 +56,7 @@ export function resolveSnapshotRef(
ctx: SessionContext,
ref: string,
tabId: number,
requiredCapability?: RefCapability,
): SnapshotRefLookup | RpcError {
const looked = lookupSnapshotRef(ctx, ref, tabId);
if (looked === null) {
@@ -58,5 +66,13 @@ export function resolveSnapshotRef(
`ref ${ref} unknown for tab ${tabId} in session ${ctx.sessionId}`,
);
}
if (requiredCapability && !looked.capabilities.includes(requiredCapability)) {
return rpcError(
"permission_denied",
"ref_capability_denied",
`ref ${ref} does not support ${requiredCapability}`,
{ ref: looked.refKey, required_capability: requiredCapability, kind: looked.kind },
);
}
return looked;
}
+59
View File
@@ -0,0 +1,59 @@
// Upload orchestration: validate the session-scoped request, resolve its click
// target, then delegate the browser protocol transaction to the file-input
// transaction module.
import type { SessionManager } from "@/session-manager/manager";
import type { RpcError, UploadParams, UploadResult } from "@/transport/types";
import { uploadThroughActivatedFileInput } from "./file-input-transaction";
import { clickResolvedTarget, type InteractionDeps, resolveActionTarget } from "./interaction";
import {
type CdpRunner,
enforceAgentWindow,
isRpcError,
lookupSession,
resolveTargetTab,
} from "./shared";
const DEFAULT_TIMEOUT_MS = 120_000;
export interface UploadDeps extends InteractionDeps {
cdp: CdpRunner;
}
export async function handleUpload(
manager: SessionManager,
params: UploadParams,
deps: UploadDeps,
): Promise<UploadResult | RpcError> {
const ctx = lookupSession(manager, params, "upload");
if (isRpcError(ctx)) return ctx;
const target = await resolveTargetTab(manager, ctx, params.tab_id, deps.tabsApi);
if (isRpcError(target)) return target;
const denied = enforceAgentWindow(ctx, target, "upload");
if (denied) return denied;
if (
params.files.length === 0 ||
params.files.length > 20 ||
params.files.some((file) => !file.staged_path)
) {
return { code: "invalid_params", message: "upload requires daemon-staged files" };
}
const address = await resolveActionTarget(deps.cdp, ctx, target, params, "upload");
if (isRpcError(address)) return address;
const timeoutMs = params.timeout_ms ?? DEFAULT_TIMEOUT_MS;
const transaction = await uploadThroughActivatedFileInput({
cdp: deps.cdp,
actionTarget: address,
files: params.files.map((file) => file.staged_path as string),
timeoutMs,
signal: deps.signal,
trigger: () => clickResolvedTarget(ctx, address, {}, deps),
});
if (isRpcError(transaction)) return transaction;
return {
tab_id: target.tabId,
used_ref: transaction.click.used_ref,
used_selector: transaction.click.used_selector,
file_names: params.files.map((file) => file.name),
};
}
@@ -1,6 +1,7 @@
import { describe, expect, it, vi } from "vitest";
import { OVERLAY_HOST_MARKER_ATTR, OVERLAY_HOST_NAME } from "../../../lib/overlay-bridge";
import { captureViewModel, collectOverlayExcludedBackendIds } from "../capture";
import { captureViewModel, collectOverlayExcludedBackendIds, probeHoverSurfaces } from "../capture";
import { discoverRenderedSurfaces } from "../rendered-surfaces";
// Minimal but format-accurate captureSnapshot reply: a body with one
// fixed full-screen overlay div carrying a password input.
@@ -36,7 +37,7 @@ function fakeSnapshotReply() {
},
layout: {
nodeIndex: [1, 2, 3],
// styles columns follow requested computedStyles order: [position, pointer-events]
// Legacy fixture: omitted style columns fall back to visible defaults.
styles: [
[i("static"), i("auto")],
[i("fixed"), i("auto")],
@@ -251,11 +252,10 @@ describe("captureViewModel", () => {
expect(input?.attrs).toEqual({ type: "password" });
});
it("does not run conditional surface probes by default", async () => {
it("never hovers the page while capturing", async () => {
const cdp = makeCdp(fakeSnapshotReply());
const result = await captureViewModel(cdp, 4);
await captureViewModel(cdp, 4);
expect(result.surfaceProbes).toEqual([]);
expect(cdp.send).not.toHaveBeenCalledWith(4, "Input.dispatchMouseEvent", expect.anything());
});
@@ -319,9 +319,10 @@ describe("captureViewModel", () => {
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
const result = await captureViewModel(cdp, 4, { conditionalSurfaceProbe: true });
const captured = await captureViewModel(cdp, 4);
const surfaceProbes = await probeHoverSurfaces(cdp, 4, captured.nodes);
expect(result.surfaceProbes).toEqual([
expect(surfaceProbes).toEqual([
{
triggerBackendNodeId: 12,
triggerPoint: { x: 956, y: 32 },
@@ -378,9 +379,10 @@ describe("captureViewModel", () => {
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
const result = await captureViewModel(cdp, 4, { conditionalSurfaceProbe: true });
const captured = await captureViewModel(cdp, 4);
const surfaceProbes = await probeHoverSurfaces(cdp, 4, captured.nodes);
expect(result.surfaceProbes).toEqual([
expect(surfaceProbes).toEqual([
expect.objectContaining({
triggerBackendNodeId: 12,
subItems: ["My profile", "Sign out"],
@@ -422,7 +424,8 @@ describe("captureViewModel", () => {
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
await captureViewModel(cdp, 4, { conditionalSurfaceProbe: true });
const captured = await captureViewModel(cdp, 4);
await probeHoverSurfaces(cdp, 4, captured.nodes);
expect(hoverMoves).toBe(1);
});
@@ -682,6 +685,84 @@ describe("captureViewModel", () => {
expect(nodes.find((n) => n.backendNodeId === 11)?.cursor).toBe("auto");
});
it("captures painted visibility as DOM fallback evidence", async () => {
const S = ["html", "body", "button", "static", "auto", "visible", "hidden", "1", "0"];
const i = (value: string) => S.indexOf(value);
const snapshot = {
strings: S,
documents: [
{
nodes: {
parentIndex: [-1, 0, 1, 1, 1],
nodeName: [i("html"), i("body"), i("button"), i("button"), i("button")],
backendNodeId: [10, 11, 12, 13, 14],
attributes: [[], [], [], [], []],
},
layout: {
nodeIndex: [1, 2, 3, 4],
styles: [
[i("static"), i("auto"), i("auto"), i("visible"), i("1")],
[i("static"), i("auto"), i("auto"), i("visible"), i("1")],
[i("static"), i("auto"), i("auto"), i("hidden"), i("1")],
[i("static"), i("auto"), i("auto"), i("visible"), i("0")],
],
bounds: [
[0, 0, 1000, 800],
[10, 10, 100, 30],
[10, 50, 100, 30],
[10, 90, 100, 30],
],
paintOrders: [0, 1, 2, 3],
},
},
],
};
const { nodes } = await captureViewModel(makeCdp(snapshot), 4);
expect(nodes.find((node) => node.backendNodeId === 12)?.rendered).toBe(true);
expect(nodes.find((node) => node.backendNodeId === 13)?.rendered).toBe(false);
expect(nodes.find((node) => node.backendNodeId === 14)?.rendered).toBe(false);
});
it("propagates fully transparent ancestor suppression to painted descendants", async () => {
const S = ["html", "body", "div", "canvas", "static", "auto", "visible", "1", "0"];
const i = (value: string) => S.indexOf(value);
const snapshot = {
strings: S,
documents: [
{
nodes: {
parentIndex: [-1, 0, 1, 2],
nodeName: [i("html"), i("body"), i("div"), i("canvas")],
backendNodeId: [10, 11, 12, 13],
attributes: [[], [], [], []],
},
layout: {
nodeIndex: [1, 2, 3],
styles: [
[i("static"), i("auto"), i("auto"), i("visible"), i("1")],
[i("static"), i("auto"), i("auto"), i("visible"), i("0")],
[i("static"), i("auto"), i("auto"), i("visible"), i("1")],
],
bounds: [
[0, 0, 1000, 800],
[10, 10, 200, 100],
[10, 10, 200, 100],
],
paintOrders: [0, 1, 2],
},
},
],
};
const { nodes } = await captureViewModel(makeCdp(snapshot), 4);
const canvas = nodes.find((node) => node.backendNodeId === 13);
expect(canvas?.rendered).toBe(true);
expect(canvas?.visuallySuppressed).toBe(true);
});
it("subtracts scroll offset to make rects viewport-relative", async () => {
const cdp = {
send: vi.fn(async (_t: number, method: string) => {
@@ -1097,6 +1178,97 @@ describe("captureViewModel", () => {
expect(input?.rect).toEqual({ x: 16, y: 28, w: 40, h: 20 });
});
it("keeps same-origin iframe canvas discovery in top-level CSS coordinates at dpr 1 and 2", async () => {
const captureAtDpr = async (dpr: number) => {
const S = ["html", "body", "iframe", "canvas", "static", "auto"];
const i = (s: string) => S.indexOf(s);
const scale = (values: number[]) => values.map((value) => value * dpr);
const snapshot = {
strings: S,
documents: [
{
frameId: "main",
nodes: {
parentIndex: [-1, 0, 1],
nodeName: [i("html"), i("body"), i("iframe")],
backendNodeId: [10, 11, 13],
attributes: [[], [], []],
contentDocumentIndex: { index: [2], value: [1] },
},
layout: {
nodeIndex: [1, 2],
styles: [
[i("static"), i("auto")],
[i("static"), i("auto")],
],
bounds: [scale([0, 0, 1000, 800]), scale([100, 150, 300, 200])],
paintOrders: [0, 1],
},
},
{
frameId: "child",
nodes: {
parentIndex: [-1, 0],
nodeName: [i("body"), i("canvas")],
backendNodeId: [20, 21],
attributes: [[], []],
},
layout: {
nodeIndex: [0, 1],
styles: [
[i("static"), i("auto")],
[i("static"), i("auto")],
],
bounds: [scale([0, 0, 300, 200]), scale([20, 30, 120, 60])],
paintOrders: [0, 1],
},
},
],
};
const cdp = {
send: vi.fn(async (_tabId: number, method: string) => {
if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") return snapshot;
if (method === "Page.getLayoutMetrics") {
return {
cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 },
layoutViewport: { clientWidth: 1000 * dpr, clientHeight: 800 * dpr },
};
}
throw new Error(method);
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
const captured = await captureViewModel(cdp, 4);
const childNodes = captured.frameNodes?.get("child") ?? [];
return discoverRenderedSurfaces(
[
{
frameId: "child",
contextScopeId: "child",
parentFrameId: "main",
ownerBackendNodeId: 13,
target: { tabId: 4 },
domNodes: childNodes,
axNodes: [],
},
],
captured.viewport,
);
};
const atDpr1 = await captureAtDpr(1);
const atDpr2 = await captureAtDpr(2);
expect(atDpr1).toEqual([
expect.objectContaining({
backendNodeId: 21,
frameId: "child",
visibleRect: { x: 120, y: 180, w: 120, h: 60 },
}),
]);
expect(atDpr2).toEqual(atDpr1);
});
it("normalizes bounds then subtracts CSS scroll at dpr>1", async () => {
const S = ["html", "body", "div", "position", "fixed", "static", "pointer-events", "auto"];
const i = (s: string) => S.indexOf(s);
@@ -132,4 +132,67 @@ describe("captureFrameData", () => {
]),
);
});
it("keeps OOPIF semantics when viewport projection is unavailable", async () => {
const owner = { ...ownerNode(10, 50), rect: null, localRect: null };
const captured: CapturedViewModel = {
nodes: [owner],
viewport: { width: 1000, height: 800 },
iframeNodes: new Map(),
frameNodes: new Map([["main", [owner]]]),
rootFrameId: "main",
excludedBackendNodeIds: new Set(),
};
const cdp: CdpRunner = {
send: vi.fn(async (_tabId, method) => {
if (method === "Accessibility.enable") return {};
if (method === "Accessibility.getFullAXTree") return { nodes: [] };
if (method === "DOM.getBoxModel") throw new Error("owner geometry unavailable");
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1000, clientHeight: 800 } };
}
throw new Error(`unexpected root ${method}`);
}) as CdpRunner["send"],
sendToTarget: vi.fn(async (_target, method) => {
if (method === "Page.getLayoutMetrics") throw new Error("viewport unavailable");
if (method === "DOMSnapshot.enable" || method === "Accessibility.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") return childSnapshot("child", 101);
if (method === "Accessibility.getFullAXTree") {
return {
nodes: [
{
nodeId: "button",
backendDOMNodeId: 101,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Continue" },
},
],
};
}
throw new Error(`unexpected ${method}`);
}) as unknown as NonNullable<CdpRunner["sendToTarget"]>,
getFrameGraph: vi.fn(async () => ({
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "child",
parentFrameId: "main",
ownerBackendNodeId: 10,
target: { tabId: 4, sessionId: "child-session" },
},
],
})),
};
const documents = await captureFrameData(cdp, 4, captured);
const childDocument = documents.find((document) => document.frameId === "child");
expect(childDocument?.axNodes).toEqual([
expect.objectContaining({ backendDOMNodeId: 101, frameId: "child" }),
]);
expect(childDocument?.domNodes.find((node) => node.backendNodeId === 101)).toEqual(
expect.objectContaining({ rect: null, localRect: { x: 10, y: 20, w: 100, h: 40 } }),
);
});
});
@@ -0,0 +1,84 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { ProbeBudget, resetOverlayBypassDepth, withOverlayBypass } from "../hover-perception";
afterEach(() => {
resetOverlayBypassDepth();
});
describe("ProbeBudget", () => {
it("refuses a candidate that cannot finish inside the budget", () => {
const budget = new ProbeBudget(500);
expect(budget.canAfford(400)).toBe(true);
expect(budget.canAfford(600)).toBe(false);
});
it("stops admitting candidates once the remaining budget is smaller than one", async () => {
const budget = new ProbeBudget(60);
await new Promise((resolve) => setTimeout(resolve, 40));
// A top-of-loop `elapsed > total` check would still admit this candidate
// and then overshoot by its full cost.
expect(budget.canAfford(50)).toBe(false);
});
});
describe("withOverlayBypass", () => {
it("toggles the overlay once around nested probe phases", async () => {
const bypass = vi.fn(async () => {});
await withOverlayBypass(bypass, 4, async () => {
await withOverlayBypass(bypass, 4, async () => {
expect(bypass).toHaveBeenCalledTimes(1);
});
expect(bypass).toHaveBeenCalledTimes(1);
});
expect(bypass.mock.calls).toEqual([
[4, true],
[4, false],
]);
});
it("restores the overlay when the probe throws", async () => {
const bypass = vi.fn(async () => {});
await expect(
withOverlayBypass(bypass, 4, async () => {
throw new Error("probe exploded");
}),
).rejects.toThrow("probe exploded");
expect(bypass).toHaveBeenLastCalledWith(4, false);
});
it("reports a failed restore instead of swallowing it", async () => {
const onRestoreFailure = vi.fn();
const bypass = vi.fn(async (_tabId: number, enabled: boolean) => {
if (!enabled) throw new Error("tab is gone");
});
await withOverlayBypass(bypass, 4, async () => undefined, { onRestoreFailure });
expect(onRestoreFailure).toHaveBeenCalledTimes(1);
});
it("does not pin the overlay off for the tab when a restore fails", async () => {
const failing = vi.fn(async (_tabId: number, enabled: boolean) => {
if (!enabled) throw new Error("tab is gone");
});
await withOverlayBypass(failing, 4, async () => undefined, { onRestoreFailure: () => {} });
const bypass = vi.fn(async () => {});
await withOverlayBypass(bypass, 4, async () => undefined);
expect(bypass.mock.calls).toEqual([
[4, true],
[4, false],
]);
});
it("runs the probe untouched when no bypass is wired up", async () => {
await expect(withOverlayBypass(undefined, 4, async () => "done")).resolves.toBe("done");
});
});
@@ -0,0 +1,214 @@
import { describe, expect, it, vi } from "vitest";
import type { CdpTarget } from "@/browser-driver/frame-graph";
import type { CdpRunner } from "../../shared";
import type { CapturedNode } from "../capture";
import type { FrameDocument, FrameOwnedAxNode } from "../frame-document";
import { probeTooltipNames } from "../name-enrichment";
import { buildSemanticGraph, resolveSemanticGraph } from "../semantic-graph";
import { stableIdentifierName } from "../semantic-graph/name-evidence";
import { frameBackendKey } from "../semantic-graph/types";
function document(
target: CdpTarget,
backendNodeId: number,
accessibleName?: string,
frameId = "child",
x = 10,
): FrameDocument<FrameOwnedAxNode> {
const node: CapturedNode = {
backendNodeId,
parentBackendNodeId: null,
frameId,
tag: "button",
attrs: {
id: "toolInsertRecord",
"data-test-id": "toolInsertRecord",
...(accessibleName ? { "aria-label": accessibleName } : {}),
},
rect: { x, y: 20, w: 40, h: 30 },
localRect: { x: 2, y: 3, w: 40, h: 30 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
cursor: "pointer",
};
return {
frameId,
parentFrameId: "main",
contextScopeId: "child",
target,
axNodes: [],
domNodes: [node],
url: "https://example.com/frame",
};
}
function semantics(documents: FrameDocument<FrameOwnedAxNode>[]) {
return resolveSemanticGraph(
buildSemanticGraph({
documents,
viewport: { width: 800, height: 600 },
rootFrameId: documents[0]?.frameId,
}),
{ identifierFallback: false },
);
}
describe("tooltip name enrichment", () => {
it("uses the owning frame target and upgrades an identifier name from a tooltip", async () => {
const target = { tabId: 7, sessionId: "oopif-session" };
let hovered = false;
const cdp: CdpRunner = {
send: vi.fn(async (_tabId, method, params) => {
if (method === "Input.dispatchMouseEvent") {
hovered = (params as { x?: number }).x === 30;
}
return {};
}) as CdpRunner["send"],
sendToTarget: vi.fn(async (_target, method) => {
if (method === "Page.createIsolatedWorld") return { executionContextId: 11 };
if (method === "DOM.resolveNode") return { object: { objectId: "button-501" } };
if (method === "Runtime.callFunctionOn") {
return { result: { value: hovered ? ["插入行"] : [] } };
}
return {};
}) as CdpRunner["sendToTarget"],
};
const documents = [document(target, 501)];
const graph = semantics(documents);
const names = await probeTooltipNames(cdp, 7, documents, graph);
expect(names.get(frameBackendKey("child", 501))).toBe("插入行");
expect(cdp.sendToTarget).toHaveBeenCalledWith(
target,
"Page.createIsolatedWorld",
expect.objectContaining({ frameId: "child" }),
);
const sendsBeforeCache = vi.mocked(cdp.send).mock.calls.length;
const targetSendsBeforeCache = vi.mocked(
cdp.sendToTarget as NonNullable<CdpRunner["sendToTarget"]>,
).mock.calls.length;
const cached = await probeTooltipNames(cdp, 7, documents, graph);
expect(cached.get(frameBackendKey("child", 501))).toBe("插入行");
expect(cdp.send).toHaveBeenCalledTimes(sendsBeforeCache);
expect(cdp.sendToTarget).toHaveBeenCalledTimes(targetSendsBeforeCache);
});
it("does not probe controls that already have a higher-confidence name", async () => {
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
sendToTarget: vi.fn(async () => ({ executionContextId: 12 })) as CdpRunner["sendToTarget"],
};
const documents = [document({ tabId: 8 }, 601, "Add a new record")];
const names = await probeTooltipNames(cdp, 8, documents, semantics(documents));
expect(names.size).toBe(0);
expect(cdp.send).not.toHaveBeenCalled();
expect(cdp.sendToTarget).not.toHaveBeenCalled();
});
it("does not collect supplemental names from form controls", async () => {
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
sendToTarget: vi.fn(async () => ({ executionContextId: 12 })) as CdpRunner["sendToTarget"],
};
const inputDocument = document({ tabId: 8 }, 602);
inputDocument.domNodes[0] = {
...inputDocument.domNodes[0],
tag: "input",
attrs: { type: "email" },
formValue: "user@example.com",
formDefaultValue: "",
formState: "filled",
};
const names = await probeTooltipNames(cdp, 8, [inputDocument], semantics([inputDocument]));
expect(names.size).toBe(0);
expect(cdp.send).not.toHaveBeenCalled();
expect(cdp.sendToTarget).not.toHaveBeenCalled();
});
it("remembers controls that revealed no tooltip so the miss is paid once", async () => {
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
sendToTarget: vi.fn(async (_target, method) => {
if (method === "Page.createIsolatedWorld") return { executionContextId: 31 };
if (method === "DOM.resolveNode") return { object: { objectId: "button-801" } };
if (method === "Runtime.callFunctionOn") return { result: { value: [] } };
return {};
}) as CdpRunner["sendToTarget"],
};
const documents = [
document({ tabId: 11, sessionId: "miss-session" }, 801, undefined, "miss-frame"),
];
const graph = semantics(documents);
const first = await probeTooltipNames(cdp, 11, documents, graph);
expect(first.size).toBe(0);
const hoversAfterFirst = vi
.mocked(cdp.send)
.mock.calls.filter(([, method]) => method === "Input.dispatchMouseEvent").length;
expect(hoversAfterFirst).toBeGreaterThan(0);
const second = await probeTooltipNames(cdp, 11, documents, graph);
expect(second.size).toBe(0);
expect(
vi.mocked(cdp.send).mock.calls.filter(([, method]) => method === "Input.dispatchMouseEvent")
.length,
).toBe(hoversAfterFirst);
});
it("keeps tooltip evidence isolated across sibling frame sessions", async () => {
let hoveredX = -10;
const cdp: CdpRunner = {
send: vi.fn(async (_tabId, method, params) => {
if (method === "Input.dispatchMouseEvent") {
hoveredX = (params as { x: number }).x;
}
return {};
}) as CdpRunner["send"],
sendToTarget: vi.fn(async (target, method, params) => {
if (method === "Page.createIsolatedWorld") {
return { executionContextId: target.sessionId === "left-session" ? 21 : 22 };
}
if (method === "DOM.resolveNode") {
return { object: { objectId: target.sessionId === "left-session" ? "left" : "right" } };
}
if (method === "Runtime.callFunctionOn") {
const objectId = (params as { objectId: string }).objectId;
const value =
objectId === "left" && hoveredX === 30
? ["Left action"]
: objectId === "right" && hoveredX === 130
? ["Right action"]
: [];
return { result: { value } };
}
return {};
}) as CdpRunner["sendToTarget"],
};
const documents = [
document({ tabId: 9, sessionId: "left-session" }, 701, undefined, "left", 10),
document({ tabId: 9, sessionId: "right-session" }, 701, undefined, "right", 110),
];
const names = await probeTooltipNames(cdp, 9, documents, semantics(documents));
expect(names.get(frameBackendKey("left", 701))).toBe("Left action");
expect(names.get(frameBackendKey("right", 701))).toBe("Right action");
});
});
describe("stable identifier evidence", () => {
it("accepts readable action identifiers and rejects framework-generated ids", () => {
expect(stableIdentifierName({ id: "toolInsertRecord" })).toBe("Insert record");
expect(stableIdentifierName({ id: "reactivateAccount" })).toBe("Reactivate account");
expect(stableIdentifierName({ id: "headlessui-menu-button-r1" })).toBeUndefined();
expect(stableIdentifierName({ id: "7f15a3bc91" })).toBeUndefined();
});
});
@@ -0,0 +1,82 @@
import { describe, expect, it } from "vitest";
import {
RECORD_DOCUMENT_ATTRIBUTE,
recordingDocumentMarkerValue,
} from "@/shared/recording-document-identity";
import { projectRecordSafeObservation } from "../record-safe-observation";
describe("projectRecordSafeObservation", () => {
it("projects the opaque recording identity without exposing captured attributes", () => {
const recordingDocumentId = "123e4567-e89b-42d3-a456-426614174000";
const result = projectRecordSafeObservation({
rootFrameId: "child",
frameDocuments: [
{
frameId: "child",
target: { tabId: 4, sessionId: "oopif-session" },
contextScopeId: "scope-1",
axNodes: [],
domNodes: [
{
backendNodeId: 1,
parentBackendNodeId: null,
tag: "html",
attrs: {
[RECORD_DOCUMENT_ATTRIBUTE]: recordingDocumentMarkerValue(recordingDocumentId),
value: "user@example.com",
},
rect: null,
paintOrder: 0,
position: "static",
pointerEvents: "auto",
},
],
},
],
rendered: { text: '@vom 1\nRootWebArea "Example"', refs: [], truncated: false },
});
expect(result.frames).toEqual([
{
frameId: "child",
target: { tabId: 4, sessionId: "oopif-session" },
recordingDocumentId,
},
]);
const dumped = JSON.stringify(result);
expect(dumped).not.toContain("user@example.com");
expect(dumped).not.toContain("attrs");
expect(dumped).not.toContain("domNodes");
expect(dumped).not.toContain("axNodes");
});
it("does not project a page-controlled marker value as recording identity", () => {
const result = projectRecordSafeObservation({
rootFrameId: "root",
frameDocuments: [
{
frameId: "root",
target: { tabId: 4 },
contextScopeId: "scope-1",
axNodes: [],
domNodes: [
{
backendNodeId: 1,
parentBackendNodeId: null,
tag: "html",
attrs: { [RECORD_DOCUMENT_ATTRIBUTE]: "bsk:user@example.com" },
rect: null,
paintOrder: 0,
position: "static",
pointerEvents: "auto",
},
],
},
],
rendered: { text: '@vom 1\nRootWebArea "Example"', refs: [], truncated: false },
});
expect(result.frames).toEqual([{ frameId: "root", target: { tabId: 4 } }]);
expect(JSON.stringify(result)).not.toContain("user@example.com");
});
});
@@ -0,0 +1,256 @@
import type { VomScene } from "@browser-skill/vom";
import { describe, expect, it } from "vitest";
import type { CapturedNode } from "../capture";
import type { CapturedFrameDocument } from "../frame-capture";
import {
clusterRenderedSurfaces,
discoverRenderedSurfaces,
projectRenderedSurfaces,
} from "../rendered-surfaces";
import type { SemanticAxNode } from "../semantic-graph";
function domNode(backendNodeId: number, overrides: Partial<CapturedNode> = {}): CapturedNode {
return {
backendNodeId,
parentBackendNodeId: null,
frameId: "main",
ownerFrameBackendNodeId: null,
tag: "canvas",
attrs: {},
rect: { x: 10, y: 20, w: 800, h: 500 },
localRect: { x: 10, y: 20, w: 800, h: 500 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
rendered: true,
...overrides,
};
}
function document(
domNodes: CapturedNode[],
axNodes: SemanticAxNode[] = [],
): CapturedFrameDocument<SemanticAxNode> {
return {
frameId: domNodes[0]?.frameId ?? "main",
contextScopeId: domNodes[0]?.frameId ?? "main",
target: { tabId: 4 },
domNodes,
axNodes,
};
}
describe("rendered surface discovery", () => {
it("finds prominent rendered canvases and excludes hidden descendants", () => {
const visible = domNode(2, { parentBackendNodeId: 1 });
const hidden = domNode(3, { parentBackendNodeId: 4 });
const surfaces = discoverRenderedSurfaces(
[
document([
domNode(1, { tag: "main", rect: null, localRect: null }),
visible,
domNode(4, { tag: "section", attrs: { "aria-hidden": "true" }, rect: null }),
hidden,
]),
],
{ width: 1280, height: 720 },
);
expect(surfaces).toHaveLength(1);
expect(surfaces[0]).toMatchObject({ backendNodeId: 2 });
});
it("keeps tiny canvases as individually addressable exact discoveries", () => {
const surfaces = discoverRenderedSurfaces(
[document([domNode(2, { rect: { x: 1, y: 1, w: 8, h: 8 } })])],
{ width: 1280, height: 720 },
);
expect(surfaces).toEqual([expect.objectContaining({ backendNodeId: 2 })]);
});
it("keeps a surface when the canvas only has partial native accessibility content", () => {
const axNodes = [
{
nodeId: "canvas",
backendDOMNodeId: 2,
role: { type: "role", value: "presentation" },
childIds: ["button"],
frameId: "main",
},
{
nodeId: "button",
backendDOMNodeId: 3,
role: { type: "role", value: "button" },
frameId: "main",
},
] as SemanticAxNode[];
expect(
discoverRenderedSurfaces([document([domNode(2)], axNodes)], { width: 1280, height: 720 }),
).toEqual([expect.objectContaining({ backendNodeId: 2 })]);
});
it("discovers unnamed canvases without guessing labels from engineering attributes", () => {
const canvas = domNode(2, {
attrs: {
id: "must-not-be-a-label",
class: "must-not-be-a-label",
"data-testid": "must-not-be-a-label",
},
});
expect(discoverRenderedSurfaces([document([canvas])], { width: 1280, height: 720 })).toEqual([
expect.not.objectContaining({
label: expect.anything(),
}),
]);
});
it("treats blank explicit Canvas names as unnamed", () => {
expect(
discoverRenderedSurfaces(
[document([domNode(2, { attrs: { "aria-label": " ", title: "\n\t" } })])],
{ width: 1280, height: 720 },
),
).toEqual([expect.not.objectContaining({ label: expect.anything() })]);
});
it("excludes canvases inside BrowserSkill-owned DOM subtrees", () => {
const owner = domNode(1, { tag: "div", rect: null, localRect: null });
const canvas = domNode(2, { parentBackendNodeId: 1 });
expect(
discoverRenderedSurfaces(
[document([owner, canvas])],
{ width: 1280, height: 720 },
new Set([1]),
),
).toEqual([]);
});
it("excludes canvases suppressed by a fully transparent ancestor", () => {
const owner = domNode(1, {
tag: "div",
rect: { x: 0, y: 0, w: 900, h: 600 },
visuallySuppressed: true,
});
const canvas = domNode(2, { parentBackendNodeId: 1 });
expect(
discoverRenderedSurfaces([document([owner, canvas])], { width: 1280, height: 720 }),
).toEqual([]);
});
it("clips canvases to overflow ancestors and drops fully clipped canvases", () => {
const owner = domNode(1, {
tag: "div",
rect: { x: 10, y: 20, w: 100, h: 70 },
overflowX: "hidden",
overflowY: "hidden",
});
const partial = domNode(2, {
parentBackendNodeId: 1,
rect: { x: 55, y: 30, w: 140, h: 60 },
});
const fullyClipped = domNode(3, {
parentBackendNodeId: 1,
rect: { x: 140, y: 30, w: 80, h: 50 },
});
expect(
discoverRenderedSurfaces([document([owner, partial, fullyClipped])], {
width: 1280,
height: 720,
}),
).toEqual([
expect.objectContaining({
backendNodeId: 2,
visibleRect: { x: 55, y: 30, w: 55, h: 60 },
}),
]);
});
});
describe("rendered surface clustering and projection", () => {
it("collapses overlapping canvas layers and attaches them to the nearest retained parent", () => {
const body = domNode(1, { tag: "body", rect: null, localRect: null });
const surfaces = discoverRenderedSurfaces(
[
document([
body,
domNode(2, { parentBackendNodeId: 1, paintOrder: 1 }),
domNode(3, { parentBackendNodeId: 1, paintOrder: 2 }),
]),
],
{ width: 1280, height: 720 },
);
const groups = clusterRenderedSurfaces(surfaces);
const scene: VomScene = {
viewport: { width: 1280, height: 720 },
nodes: [
{
id: 10,
parentId: null,
backendNodeId: 1,
frameId: "main",
tag: "body",
role: "RootWebArea",
rect: null,
paintOrder: 0,
position: "static",
pointerEvents: "auto",
},
],
};
expect(groups).toHaveLength(1);
expect(groups[0]?.members).toHaveLength(2);
expect(groups[0]?.label).toBeUndefined();
expect(
projectRenderedSurfaces(
scene,
[document([body, ...surfaces.map((s) => domNode(s.backendNodeId))])],
groups,
),
).toMatchObject({
visualSurfaces: [expect.objectContaining({ parentId: 10, backendNodeId: 3, memberCount: 2 })],
});
});
it("does not merge a contained canvas with a larger independent canvas", () => {
const groups = clusterRenderedSurfaces([
...discoverRenderedSurfaces(
[
document([
domNode(1, { tag: "main", rect: null, localRect: null }),
domNode(2, { parentBackendNodeId: 1 }),
domNode(3, {
parentBackendNodeId: 1,
rect: { x: 30, y: 40, w: 200, h: 100 },
}),
]),
],
{ width: 1280, height: 720 },
),
]);
expect(groups).toHaveLength(2);
});
it("does not merge coincident canvases owned by different containers", () => {
const groups = clusterRenderedSurfaces(
discoverRenderedSurfaces(
[
document([
domNode(1, { tag: "section", rect: null, localRect: null }),
domNode(2, { tag: "section", rect: null, localRect: null }),
domNode(3, { parentBackendNodeId: 1 }),
domNode(4, { parentBackendNodeId: 2 }),
]),
],
{ width: 1280, height: 720 },
),
);
expect(groups).toHaveLength(2);
});
});
@@ -0,0 +1,782 @@
import { isVomReferenceNode, renderVom } from "@browser-skill/vom";
import { describe, expect, it } from "vitest";
import type { CapturedNode } from "../capture";
import type { FrameDocument } from "../frame-document";
import { buildSemanticVomScene, type SemanticAxNode } from "../semantic-graph";
function dom(
backendNodeId: number,
parentBackendNodeId: number | null,
tag: string,
attrs: Record<string, string> = {},
textContent?: string,
): CapturedNode {
return {
backendNodeId,
parentBackendNodeId,
tag,
attrs,
rect: { x: 0, y: 0, w: 100, h: 30 },
paintOrder: backendNodeId,
position: "static",
pointerEvents: "auto",
...(textContent ? { textContent } : {}),
};
}
function document(
frameId: string,
axNodes: SemanticAxNode[],
domNodes: CapturedNode[],
frame?: Pick<FrameDocument<SemanticAxNode>, "parentFrameId" | "ownerBackendNodeId">,
): FrameDocument<SemanticAxNode> {
return {
frameId,
contextScopeId: frameId,
target: { tabId: 1 },
axNodes,
domNodes,
...frame,
};
}
describe("semantic VOM graph", () => {
it("preserves backend-less AX structure without making it referenceable", () => {
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["toolbar"],
},
{
nodeId: "toolbar",
parentId: "root",
role: { type: "role", value: "toolbar" },
name: { type: "computedString", value: "Formatting" },
childIds: ["bold", "italic"],
},
{
nodeId: "bold",
parentId: "toolbar",
backendDOMNodeId: 2,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Bold" },
},
{
nodeId: "italic",
parentId: "toolbar",
backendDOMNodeId: 3,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Italic" },
},
],
[dom(1, null, "body"), dom(2, 1, "button"), dom(3, 1, "button")],
),
],
});
const toolbar = scene.nodes.find((node) => node.role === "toolbar");
const buttons = scene.nodes.filter((node) => node.role === "button");
expect(toolbar).toEqual(expect.objectContaining({ name: "Formatting", referenceable: false }));
expect(buttons.map((node) => node.parentId)).toEqual([toolbar?.id, toolbar?.id]);
expect(toolbar && isVomReferenceNode(toolbar)).toBe(false);
expect(renderVom(scene).refs.map((ref) => ref.backendNodeId)).toEqual([2, 3]);
});
it("resolves missing AX names through one DOM naming pipeline", () => {
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["labelled", "titled", "text"],
},
{
nodeId: "labelled",
parentId: "root",
backendDOMNodeId: 2,
role: { type: "role", value: "button" },
},
{
nodeId: "titled",
parentId: "root",
backendDOMNodeId: 3,
role: { type: "role", value: "button" },
},
{
nodeId: "text",
parentId: "root",
backendDOMNodeId: 4,
role: { type: "role", value: "button" },
},
],
[
dom(1, null, "body"),
dom(2, 1, "button", { "aria-label": "Add row" }),
dom(3, 1, "button", { title: "Settings" }),
dom(4, 1, "button"),
dom(5, 4, "span", {}, "Save"),
],
),
],
});
expect(scene.nodes.filter((node) => node.role === "button").map((node) => node.name)).toEqual([
"Add row",
"Settings",
"Save",
]);
});
it("aggregates AX text at the semantic text frontier", () => {
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["covered", "inline-only", "repeated", "labelled"],
},
{
nodeId: "covered",
parentId: "root",
backendDOMNodeId: 2,
role: { type: "role", value: "paragraph" },
childIds: ["covered-static"],
},
{
nodeId: "covered-static",
parentId: "covered",
role: { type: "role", value: "StaticText" },
name: { type: "computedString", value: "Covered text" },
childIds: ["covered-inline"],
},
{
nodeId: "covered-inline",
parentId: "covered-static",
role: { type: "role", value: "InlineTextBox" },
name: { type: "computedString", value: "Covered text" },
},
{
nodeId: "inline-only",
parentId: "root",
backendDOMNodeId: 3,
role: { type: "role", value: "paragraph" },
childIds: ["inline-leaf"],
},
{
nodeId: "inline-leaf",
parentId: "inline-only",
role: { type: "role", value: "InlineTextBox" },
name: { type: "computedString", value: "Inline fallback" },
},
{
nodeId: "repeated",
parentId: "root",
backendDOMNodeId: 4,
role: { type: "role", value: "paragraph" },
childIds: ["first-go", "second-go"],
},
{
nodeId: "first-go",
parentId: "repeated",
role: { type: "role", value: "StaticText" },
name: { type: "computedString", value: "Go" },
},
{
nodeId: "second-go",
parentId: "repeated",
role: { type: "role", value: "StaticText" },
name: { type: "computedString", value: "Go" },
},
{
nodeId: "labelled",
parentId: "root",
backendDOMNodeId: 5,
role: { type: "role", value: "button" },
childIds: ["visible-label"],
},
{
nodeId: "visible-label",
parentId: "labelled",
role: { type: "role", value: "StaticText" },
name: { type: "computedString", value: "Visible text" },
},
],
[
dom(1, null, "body"),
dom(2, 1, "p"),
dom(3, 1, "p"),
dom(4, 1, "p"),
dom(5, 1, "button", { "aria-label": "Explicit action" }),
],
),
],
});
expect(scene.nodes.find((node) => node.backendNodeId === 2)?.name).toBe("Covered text");
expect(scene.nodes.find((node) => node.backendNodeId === 3)?.name).toBe("Inline fallback");
expect(scene.nodes.find((node) => node.backendNodeId === 4)?.name).toBe("Go Go");
expect(scene.nodes.find((node) => node.backendNodeId === 5)?.name).toBe("Explicit action");
expect(renderVom(scene).text).not.toContain("Covered text Covered text");
});
it("uses stable identifiers only as a final name fallback", () => {
const base = document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["fallback", "labelled", "container"],
},
{
nodeId: "fallback",
parentId: "root",
backendDOMNodeId: 2,
role: { type: "role", value: "button" },
},
{
nodeId: "labelled",
parentId: "root",
backendDOMNodeId: 3,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Localized name" },
},
{
nodeId: "container",
parentId: "root",
backendDOMNodeId: 4,
role: { type: "role", value: "generic" },
},
],
[
dom(1, null, "body"),
dom(2, 1, "button", { "data-test-id": "toolInsertRecord" }),
dom(3, 1, "button", { id: "toolIgnoredFallback" }),
dom(4, 1, "div", { id: "toolbarContainer" }),
],
);
const staticScene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [base],
});
const enrichedScene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [base],
supplementalNames: new Map([["main\u00002", "插入行"]]),
});
expect(staticScene.nodes.find((node) => node.backendNodeId === 2)?.name).toBe("Insert record");
expect(staticScene.nodes.find((node) => node.backendNodeId === 3)?.name).toBe("Localized name");
expect(staticScene.nodes.find((node) => node.backendNodeId === 4)?.name).toBeUndefined();
expect(enrichedScene.nodes.find((node) => node.backendNodeId === 2)?.name).toBe("插入行");
expect(enrichedScene.nodes.find((node) => node.backendNodeId === 3)?.name).toBe(
"Localized name",
);
});
it("isolates equal backend node ids in sibling frames", () => {
const scene = buildSemanticVomScene({
viewport: { width: 1000, height: 800 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["owner-a", "owner-b"],
},
{
nodeId: "owner-a",
parentId: "root",
backendDOMNodeId: 10,
role: { type: "role", value: "Iframe" },
},
{
nodeId: "owner-b",
parentId: "root",
backendDOMNodeId: 20,
role: { type: "role", value: "Iframe" },
},
],
[dom(1, null, "body"), dom(10, 1, "iframe"), dom(20, 1, "iframe")],
),
document(
"frame-a",
[
{
nodeId: "button-a",
backendDOMNodeId: 5,
role: { type: "role", value: "button" },
childIds: ["text-a"],
},
{
nodeId: "text-a",
parentId: "button-a",
role: { type: "role", value: "StaticText" },
name: { type: "computedString", value: "Accept" },
childIds: ["inline-a"],
},
{
nodeId: "inline-a",
parentId: "text-a",
role: { type: "role", value: "InlineTextBox" },
name: { type: "computedString", value: "Accept" },
},
],
[dom(5, null, "button")],
{ parentFrameId: "main", ownerBackendNodeId: 10 },
),
document(
"frame-b",
[
{
nodeId: "button-b",
backendDOMNodeId: 5,
role: { type: "role", value: "button" },
childIds: ["text-b"],
},
{
nodeId: "text-b",
parentId: "button-b",
role: { type: "role", value: "StaticText" },
name: { type: "computedString", value: "Decline" },
childIds: ["inline-b"],
},
{
nodeId: "inline-b",
parentId: "text-b",
role: { type: "role", value: "InlineTextBox" },
name: { type: "computedString", value: "Decline" },
},
],
[dom(5, null, "button")],
{ parentFrameId: "main", ownerBackendNodeId: 20 },
),
],
});
const frameButtons = scene.nodes.filter((node) => node.backendNodeId === 5);
const owners = new Map(
scene.nodes
.filter((node) => node.backendNodeId === 10 || node.backendNodeId === 20)
.map((node) => [node.backendNodeId, node.id]),
);
expect(frameButtons.map((node) => node.frameId)).toEqual(["frame-a", "frame-b"]);
expect(frameButtons.map((node) => node.name)).toEqual(["Accept", "Decline"]);
expect(frameButtons.map((node) => node.parentId)).toEqual([owners.get(10), owners.get(20)]);
expect(renderVom(scene).refs.map((ref) => [ref.frameId, ref.backendNodeId])).toEqual([
["frame-a", 5],
["frame-b", 5],
]);
});
it("requires render evidence before promoting DOM fallback semantics", () => {
const hiddenOwner = {
...dom(10, 1, "iframe", { hidden: "" }),
rect: null,
localRect: null,
rendered: false,
};
const hiddenButton = {
...dom(2, 1, "button", { "aria-label": "Hidden action" }),
rendered: false,
};
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
},
],
[dom(1, null, "body"), hiddenButton, hiddenOwner],
),
document("hidden-frame", [], [dom(20, null, "button", {}, "Frame action")], {
parentFrameId: "main",
ownerBackendNodeId: 10,
}),
],
});
expect(scene.nodes.some((node) => node.backendNodeId === 2)).toBe(false);
expect(scene.nodes.some((node) => node.backendNodeId === 10)).toBe(false);
expect(scene.nodes.some((node) => node.frameId === "hidden-frame")).toBe(false);
expect(renderVom(scene).refs).toEqual([]);
});
it("retains rendered DOM fallback controls when AX semantics are unavailable", () => {
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[],
[dom(1, null, "body"), { ...dom(2, 1, "button", {}, "Save"), rendered: true }],
),
],
});
expect(renderVom(scene).refs).toEqual([
expect.objectContaining({ backendNodeId: 2, role: "button", name: "Save" }),
]);
});
it("lets explicit DOM structure override ignored AX metadata", () => {
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["toolbar"],
},
{
nodeId: "toolbar",
parentId: "root",
backendDOMNodeId: 2,
ignored: true,
role: { type: "role", value: "generic" },
childIds: ["wrapper"],
},
{
nodeId: "wrapper",
parentId: "toolbar",
backendDOMNodeId: 5,
role: { type: "role", value: "generic" },
childIds: ["bold", "italic"],
},
{
nodeId: "bold",
parentId: "wrapper",
backendDOMNodeId: 3,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Bold" },
},
{
nodeId: "italic",
parentId: "wrapper",
backendDOMNodeId: 4,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Italic" },
},
],
[
dom(1, null, "body"),
dom(2, 1, "div", { role: "toolbar" }),
dom(5, 2, "div"),
dom(3, 5, "button"),
dom(4, 5, "button"),
],
),
],
});
const toolbar = scene.nodes.find((node) => node.backendNodeId === 2);
const buttons = scene.nodes.filter((node) => [3, 4].includes(node.backendNodeId ?? -1));
expect(toolbar).toEqual(expect.objectContaining({ role: "toolbar" }));
expect(toolbar && isVomReferenceNode(toolbar)).toBe(false);
expect(scene.nodes.some((node) => node.role === "group")).toBe(false);
expect(buttons.map((node) => node.parentId)).toEqual([toolbar?.id, toolbar?.id]);
});
it("infers an unnamed group only from independent interaction branches", () => {
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["wrapper"],
},
{
nodeId: "wrapper",
parentId: "root",
backendDOMNodeId: 2,
role: { type: "role", value: "generic" },
childIds: ["first", "second"],
},
{
nodeId: "first",
parentId: "wrapper",
backendDOMNodeId: 3,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "First" },
},
{
nodeId: "second",
parentId: "wrapper",
backendDOMNodeId: 4,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Second" },
},
],
[dom(1, null, "body"), dom(2, 1, "div"), dom(3, 2, "button"), dom(4, 2, "button")],
),
],
});
const group = scene.nodes.find((node) => node.backendNodeId === 2);
const buttons = scene.nodes.filter((node) => [3, 4].includes(node.backendNodeId ?? -1));
expect(group).toEqual(expect.objectContaining({ role: "group" }));
expect(group?.name).toBeUndefined();
expect(group && isVomReferenceNode(group)).toBe(false);
expect(buttons.map((node) => node.parentId)).toEqual([group?.id, group?.id]);
const rendered = renderVom(scene);
const lines = rendered.text.split("\n");
const groupLine = lines.findIndex((line) => line.trim() === "group");
expect(groupLine).toBeGreaterThanOrEqual(0);
for (const ref of rendered.refs) {
expect(ref.line).toBeGreaterThan(groupLine);
expect(lines[ref.line]).toMatch(/^\s+@e\d+ button/);
}
});
it("never promotes a form value into its semantic name through supplemental evidence", () => {
const secret = "user@example.com";
const input = {
...dom(2, 1, "input", { type: "email" }),
formValue: secret,
formDefaultValue: "",
formState: "filled" as const,
};
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
supplementalNames: new Map([["main\u00002", secret]]),
identifierFallback: false,
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["email"],
},
{
nodeId: "email",
parentId: "root",
backendDOMNodeId: 2,
role: { type: "role", value: "textbox" },
value: { value: secret },
},
],
[dom(1, null, "body"), input],
),
],
});
const field = scene.nodes.find((node) => node.backendNodeId === 2);
expect(field?.name).toBeUndefined();
const rendered = renderVom(scene, { redactValues: true });
expect(rendered.text).toContain("•••");
expect(JSON.stringify(rendered)).not.toContain(secret);
});
it("keeps uncertain single-branch wrappers flat", () => {
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["wrapper"],
},
{
nodeId: "wrapper",
parentId: "root",
backendDOMNodeId: 2,
role: { type: "role", value: "generic" },
childIds: ["button"],
},
{
nodeId: "button",
parentId: "wrapper",
backendDOMNodeId: 3,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Only action" },
},
],
[dom(1, null, "body"), dom(2, 1, "div"), dom(3, 2, "button")],
),
],
});
expect(scene.nodes.some((node) => node.role === "group")).toBe(false);
expect(scene.nodes.find((node) => node.backendNodeId === 3)?.parentId).toBe(
scene.nodes.find((node) => node.backendNodeId === 1)?.id,
);
});
it("does not infer across mixed content boundaries", () => {
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["wrapper"],
},
{
nodeId: "wrapper",
parentId: "root",
backendDOMNodeId: 2,
role: { type: "role", value: "generic" },
childIds: ["first", "second", "paragraph"],
},
{
nodeId: "first",
parentId: "wrapper",
backendDOMNodeId: 3,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "First" },
},
{
nodeId: "second",
parentId: "wrapper",
backendDOMNodeId: 4,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Second" },
},
{
nodeId: "paragraph",
parentId: "wrapper",
backendDOMNodeId: 5,
role: { type: "role", value: "paragraph" },
},
],
[
dom(1, null, "body"),
dom(2, 1, "div"),
dom(3, 2, "button"),
dom(4, 2, "button"),
dom(5, 2, "p", {}, "Description"),
],
),
],
});
expect(scene.nodes.some((node) => node.role === "group")).toBe(false);
});
it("keeps only the innermost reliable inferred group", () => {
const scene = buildSemanticVomScene({
viewport: { width: 800, height: 600 },
rootFrameId: "main",
documents: [
document(
"main",
[
{
nodeId: "root",
backendDOMNodeId: 1,
role: { type: "role", value: "RootWebArea" },
childIds: ["outer"],
},
{
nodeId: "outer",
parentId: "root",
backendDOMNodeId: 2,
role: { type: "role", value: "generic" },
childIds: ["inner", "third"],
},
{
nodeId: "inner",
parentId: "outer",
backendDOMNodeId: 3,
role: { type: "role", value: "generic" },
childIds: ["first", "second"],
},
{
nodeId: "first",
parentId: "inner",
backendDOMNodeId: 4,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "First" },
},
{
nodeId: "second",
parentId: "inner",
backendDOMNodeId: 5,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Second" },
},
{
nodeId: "third",
parentId: "outer",
backendDOMNodeId: 6,
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Third" },
},
],
[
dom(1, null, "body"),
dom(2, 1, "div"),
dom(3, 2, "div"),
dom(4, 3, "button"),
dom(5, 3, "button"),
dom(6, 2, "button"),
],
),
],
});
const groups = scene.nodes.filter((node) => node.role === "group");
expect(groups.map((node) => node.backendNodeId)).toEqual([3]);
expect(scene.nodes.find((node) => node.backendNodeId === 6)?.parentId).not.toBe(groups[0]?.id);
});
});
+136 -98
View File
@@ -1,16 +1,25 @@
// CDP capture adapter: DOMSnapshot.captureSnapshot + Page.getLayoutMetrics
// → CapturedNode[] + Viewport. This is the ONLY VOM module that touches
// raw CDP. The captureSnapshot reply is columnar (parallel arrays + a
// shared string table); we request exactly three computed styles so the
// `styles` columns are [position, pointer-events, cursor] in that order.
// shared string table); requested computed styles are decoded through
// STYLE_COL so capture and visibility policy share one explicit contract.
import type { Rect, Viewport } from "@browser-skill/vom";
import { evaluateHoverTrigger } from "@/lib/hover-trigger-policy";
import { isOverlayHostNode, OVERLAY_HOST_SELECTOR } from "../../lib/overlay-bridge";
import { childFrameProjection, type GeometryProjection, projectRectToViewport } from "../geometry";
import type { CdpRunner } from "../shared";
import { clearHover, ProbeBudget, waitForHover } from "./hover-perception";
const REQUESTED_STYLES = ["position", "pointer-events", "cursor"] as const;
const REQUESTED_STYLES = [
"position",
"pointer-events",
"cursor",
"visibility",
"opacity",
"overflow-x",
"overflow-y",
] as const;
const STYLE_COL = Object.fromEntries(
REQUESTED_STYLES.map((name, index) => [name, index]),
) as Record<(typeof REQUESTED_STYLES)[number], number>;
@@ -38,6 +47,16 @@ export interface CapturedNode {
* `textContent`: the live parser always sets it, hand-built fixtures may not.
*/
cursor?: string;
overflowX?: string;
overflowY?: string;
/** True when this node or one of its DOM ancestors makes the subtree fully transparent/hidden. */
visuallySuppressed?: boolean;
/**
* Whether the live DOM snapshot provides a painted, non-hidden box for this
* node. Semantic resolution uses this only for DOM fallback nodes; AX-backed
* nodes remain authoritative even when they are outside the viewport.
*/
rendered?: boolean;
textContent?: string;
formState?: "empty" | "filled" | "default";
formValue?: string;
@@ -64,14 +83,11 @@ export interface CapturedViewModel {
/** Explicit DOMSnapshot frame ancestry; never inferred from backend node ids. */
frameParentIds?: Map<string, string>;
rootFrameId?: string;
surfaceProbes?: CapturedSurfaceProbe[];
/** Backend node ids belonging to the agent overlay host + its shadow subtree. */
excludedBackendNodeIds: Set<number>;
}
export interface CaptureViewModelOptions {
conditionalSurfaceProbe?: boolean;
hoverProbeBypassOverlay?: (tabId: number, enabled: boolean) => Promise<void>;
signal?: AbortSignal;
}
@@ -361,7 +377,16 @@ function collectBackendIdsFromDomNode(node: CdpDomNode | undefined, out: Set<num
}
}
const MAX_HOVER_PROBE_MS = 2_000;
/**
* Ceiling for the whole hover-surface phase.
*
* The previous 2000 was only a floor: the loop checked elapsed time at the top,
* so a candidate could start with 1ms left and still run its two settle
* windows, pushing real cost to ~2.6s. This is that true ceiling, now actually
* enforced by an up-front affordability check, so the same number of candidates
* get probed under a bound that no longer lies.
*/
const MAX_HOVER_PROBE_MS = 2_600;
const MAX_HOVER_TRIGGERS = 6;
const MAX_HOVER_SURFACES = 3;
const HOVER_SETTLE_MS = 300;
@@ -460,36 +485,6 @@ function hoverStateExpression(): string {
})()`;
}
async function wait(ms: number, signal?: AbortSignal): Promise<void> {
throwIfAborted(signal);
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => {
signal?.removeEventListener("abort", onAbort);
resolve();
}, ms);
const onAbort = () => {
clearTimeout(timer);
signal?.removeEventListener("abort", onAbort);
reject(captureAbortError());
};
signal?.addEventListener("abort", onAbort, { once: true });
});
}
async function clearHover(cdp: CdpRunner, tabId: number): Promise<void> {
await cdp
.send(tabId, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: -10,
y: -10,
})
.catch(() =>
cdp.send(tabId, "Input.dispatchMouseEvent", { type: "mouseMoved", x: 0, y: 0 }).catch(() => {
// best effort
}),
);
}
function capturedText(node: CapturedNode): string | undefined {
const value =
node.attrs["aria-label"] ?? node.attrs.title ?? node.attrs.alt ?? node.textContent ?? "";
@@ -637,13 +632,35 @@ function confidenceForHover(
return "low";
}
async function probeHoverSurfaces(
/**
* One candidate costs two settle windows (baseline + post-hover) plus a few
* CDP round trips. Used to decide whether the next candidate still fits the
* budget before paying for it.
*/
const HOVER_CANDIDATE_COST_MS = HOVER_SETTLE_MS * 2;
export interface HoverSurfaceProbeOptions {
signal?: AbortSignal;
}
/**
* Hovers a bounded set of likely menu triggers and reports the sub-items each
* one reveals.
*
* Runs after DOM *and* accessibility capture. Hovering can open menus and
* change layout, so probing between the two captures would leave the DOM half
* of an observation describing the page before the change and the AX half
* describing it after.
*
* The caller owns the overlay bypass span (see `withOverlayBypass`).
*/
export async function probeHoverSurfaces(
cdp: CdpRunner,
tabId: number,
nodes: CapturedNode[],
options: CaptureViewModelOptions,
options: HoverSurfaceProbeOptions = {},
): Promise<CapturedSurfaceProbe[]> {
const started = Date.now();
const budget = new ProbeBudget(MAX_HOVER_PROBE_MS);
try {
throwIfAborted(options.signal);
const cssScan = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
@@ -658,60 +675,54 @@ async function probeHoverSurfaces(
const results: CapturedSurfaceProbe[] = [];
const seen = new Set<number>();
throwIfAborted(options.signal);
await options.hoverProbeBypassOverlay?.(tabId, true).catch(() => undefined);
try {
for (const candidate of candidates.slice(0, MAX_HOVER_TRIGGERS)) {
throwIfAborted(options.signal);
for (const candidate of candidates.slice(0, MAX_HOVER_TRIGGERS)) {
if (!budget.canAfford(HOVER_CANDIDATE_COST_MS)) break;
if (results.length >= MAX_HOVER_SURFACES) break;
if (seen.has(candidate.backendNodeId)) continue;
try {
await clearHover(cdp, tabId);
throwIfAborted(options.signal);
if (Date.now() - started > MAX_HOVER_PROBE_MS) break;
if (results.length >= MAX_HOVER_SURFACES) break;
if (seen.has(candidate.backendNodeId)) continue;
try {
await clearHover(cdp, tabId);
throwIfAborted(options.signal);
await wait(HOVER_SETTLE_MS, options.signal);
const baselineReply = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: hoverStateExpression(),
returnByValue: true,
});
throwIfAborted(options.signal);
const baselineItems = runtimeValue<HoverRuntimeItem[]>(baselineReply) ?? [];
await waitForHover(HOVER_SETTLE_MS, options.signal);
const baselineReply = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: hoverStateExpression(),
returnByValue: true,
});
throwIfAborted(options.signal);
const baselineItems = runtimeValue<HoverRuntimeItem[]>(baselineReply) ?? [];
throwIfAborted(options.signal);
await cdp.send(tabId, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: candidate.x,
y: candidate.y,
});
throwIfAborted(options.signal);
await wait(HOVER_SETTLE_MS, options.signal);
const collected = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: hoverStateExpression(),
returnByValue: true,
});
throwIfAborted(options.signal);
const subItems = diffHoverItems(
baselineItems,
runtimeValue<HoverRuntimeItem[]>(collected) ?? [],
);
if (subItems.length === 0) continue;
seen.add(candidate.backendNodeId);
results.push({
triggerBackendNodeId: candidate.backendNodeId,
triggerPoint: { x: candidate.x, y: candidate.y },
triggerAction: "hover",
subItems,
confidence: confidenceForHover(candidate, subItems),
});
} catch (error) {
if (isAbortError(error)) throw error;
continue;
} finally {
await clearHover(cdp, tabId);
}
throwIfAborted(options.signal);
await cdp.send(tabId, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: candidate.x,
y: candidate.y,
});
throwIfAborted(options.signal);
await waitForHover(HOVER_SETTLE_MS, options.signal);
const collected = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: hoverStateExpression(),
returnByValue: true,
});
throwIfAborted(options.signal);
const subItems = diffHoverItems(
baselineItems,
runtimeValue<HoverRuntimeItem[]>(collected) ?? [],
);
if (subItems.length === 0) continue;
seen.add(candidate.backendNodeId);
results.push({
triggerBackendNodeId: candidate.backendNodeId,
triggerPoint: { x: candidate.x, y: candidate.y },
triggerAction: "hover",
subItems,
confidence: confidenceForHover(candidate, subItems),
});
} catch (error) {
if (isAbortError(error)) throw error;
continue;
} finally {
await clearHover(cdp, tabId);
}
} finally {
await options.hoverProbeBypassOverlay?.(tabId, false).catch(() => undefined);
}
return results;
} catch (err) {
@@ -817,6 +828,10 @@ function parseDocumentNodes(
const posCol = STYLE_COL.position;
const peCol = STYLE_COL["pointer-events"];
const cursorCol = STYLE_COL.cursor;
const visibilityCol = STYLE_COL.visibility;
const opacityCol = STYLE_COL.opacity;
const overflowXCol = STYLE_COL["overflow-x"];
const overflowYCol = STYLE_COL["overflow-y"];
const inputValues = sparseIndexMap(dn.inputValue);
const textValues = sparseIndexMap(dn.textValue);
const checkedInputs = new Set(dn.inputChecked?.index ?? []);
@@ -842,6 +857,7 @@ function parseDocumentNodes(
// Build CapturedNodes.
const nodes: CapturedNode[] = [];
const visuallySuppressedByNodeIndex = new Array<boolean>(count).fill(false);
for (let n = 0; n < count; n++) {
if (excluded[n]) continue;
const backendNodeId = dn.backendNodeId[n];
@@ -861,6 +877,11 @@ function parseDocumentNodes(
let position = "static";
let pointerEvents = "auto";
let cursor = "auto";
let overflowX = "visible";
let overflowY = "visible";
let rendered = false;
let visuallySuppressed =
parentIdx >= 0 ? visuallySuppressedByNodeIndex[parentIdx] === true : false;
const li = layoutByNode.get(n);
if (li !== undefined) {
const b = dl?.bounds?.[li];
@@ -881,7 +902,21 @@ function parseDocumentNodes(
position = str(strings, styleRow[posCol]) || "static";
pointerEvents = str(strings, styleRow[peCol]) || "auto";
cursor = str(strings, styleRow[cursorCol]) || "auto";
const visibility = str(strings, styleRow[visibilityCol]) || "visible";
const opacity = str(strings, styleRow[opacityCol]) || "1";
overflowX = str(strings, styleRow[overflowXCol]) || "visible";
overflowY = str(strings, styleRow[overflowYCol]) || "visible";
visuallySuppressed ||=
visibility === "hidden" ||
visibility === "collapse" ||
(Number.parseFloat(opacity) || 0) <= 0;
rendered =
localRect !== null &&
visibility !== "hidden" &&
visibility !== "collapse" &&
(Number.parseFloat(opacity) || 0) > 0;
}
visuallySuppressedByNodeIndex[n] = visuallySuppressed;
// Skip non-element nodes (#text, #cdata-section, etc.) — they carry no
// geometry and are never queried by callers.
@@ -916,6 +951,10 @@ function parseDocumentNodes(
position,
pointerEvents,
cursor,
overflowX,
overflowY,
visuallySuppressed,
rendered,
textContent,
...(formValue !== undefined ? { formValue } : {}),
...(tag === "input" || tag === "textarea"
@@ -1020,7 +1059,13 @@ export async function captureViewModel(
options: CaptureViewModelOptions = {},
): Promise<CapturedViewModel> {
throwIfAborted(options.signal);
const metrics = await cdp.send<LayoutMetricsReply>(tabId, "Page.getLayoutMetrics", {});
let metrics: LayoutMetricsReply = {};
try {
metrics = await cdp.send<LayoutMetricsReply>(tabId, "Page.getLayoutMetrics", {});
} catch (error) {
if (isAbortError(error)) throw error;
console.debug("[bsk capture] layout metrics unavailable", error);
}
throwIfAborted(options.signal);
const dpr = devicePixelRatio(metrics);
const vpSrc = metrics.cssLayoutViewport ?? metrics.layoutViewport ?? {};
@@ -1048,7 +1093,6 @@ export async function captureViewModel(
nodes: [],
viewport,
iframeNodes: new Map(),
surfaceProbes: [],
excludedBackendNodeIds: new Set(),
};
}
@@ -1077,11 +1121,6 @@ export async function captureViewModel(
await enrichFormControlStates(cdp, tabId, [nodes, ...iframeNodes.values()], options.signal);
throwIfAborted(options.signal);
const surfaceProbes = options.conditionalSurfaceProbe
? await probeHoverSurfaces(cdp, tabId, nodes, options)
: [];
throwIfAborted(options.signal);
const frameNodes = new Map<string, CapturedNode[]>();
if (topContext.frameId) frameNodes.set(topContext.frameId, nodes);
for (const iframeFrameNodes of iframeNodes.values()) {
@@ -1102,7 +1141,6 @@ export async function captureViewModel(
frameOwnerBackendNodeIds: frameParsed.frameOwnerBackendNodeIds,
frameParentIds: frameParsed.frameParentIds,
...(topContext.frameId ? { rootFrameId: topContext.frameId } : {}),
surfaceProbes,
excludedBackendNodeIds,
};
}
+24 -22
View File
@@ -33,25 +33,19 @@ async function discoverFrameGraph(cdp: CdpRunner, tabId: number): Promise<CdpFra
function transformFrameNodes(
nodes: CapturedNode[],
projection: GeometryProjection,
projection: GeometryProjection | null,
): CapturedNode[] {
return nodes.map((node) => ({
...node,
rect: (() => {
const bounds = projectRectToViewport(node.rect, projection);
return bounds ? { x: bounds.x, y: bounds.y, w: bounds.width, h: bounds.height } : null;
})(),
rect: projection
? (() => {
const bounds = projectRectToViewport(node.rect, projection);
return bounds ? { x: bounds.x, y: bounds.y, w: bounds.width, h: bounds.height } : null;
})()
: null,
}));
}
function frameOwnerNode(captured: CapturedViewModel, frame: CdpFrame): CapturedNode | undefined {
if (frame.ownerBackendNodeId === undefined) return undefined;
const parentNodes = frame.parentFrameId
? captured.frameNodes?.get(frame.parentFrameId)
: captured.nodes;
return parentNodes?.find((node) => node.backendNodeId === frame.ownerBackendNodeId);
}
async function captureMissingFrameDocuments(
cdp: CdpRunner,
tabId: number,
@@ -65,14 +59,20 @@ async function captureMissingFrameDocuments(
for (const frame of graph.frames) {
if (!frame.target.sessionId || captured.frameNodes.has(frame.frameId)) continue;
const owner = frameOwnerNode(captured, frame);
if (!owner?.rect || frame.ownerBackendNodeId === undefined) continue;
try {
const child = await captureViewModel(cdpRunnerForTarget(cdp, frame.target), tabId, {
signal,
});
const projection = await resolveFrameProjection(cdp, graph, frame.frameId);
if (!projection) continue;
let projection: GeometryProjection | null = null;
try {
projection = await resolveFrameProjection(cdp, graph, frame.frameId);
} catch (err) {
if (isAbortError(err)) throw err;
console.debug("[bsk observation] child frame geometry projection failed", {
frameId: frame.frameId,
err,
});
}
const rootFrameId = child.rootFrameId ?? frame.frameId;
const childFrames = child.frameNodes ?? new Map<string, CapturedNode[]>();
if (!childFrames.has(rootFrameId)) childFrames.set(rootFrameId, child.nodes);
@@ -85,11 +85,13 @@ async function captureMissingFrameDocuments(
for (const [childFrameId, nodes] of childFrames) {
captured.frameNodes.set(childFrameId, transformFrameNodes(nodes, projection));
}
captured.iframeNodes.set(
frame.ownerBackendNodeId,
captured.frameNodes.get(frame.frameId) ?? [],
);
captured.frameOwnerBackendNodeIds.set(frame.frameId, frame.ownerBackendNodeId);
if (frame.ownerBackendNodeId !== undefined) {
captured.iframeNodes.set(
frame.ownerBackendNodeId,
captured.frameNodes.get(frame.frameId) ?? [],
);
captured.frameOwnerBackendNodeIds.set(frame.frameId, frame.ownerBackendNodeId);
}
if (frame.parentFrameId) captured.frameParentIds.set(frame.frameId, frame.parentFrameId);
for (const [childFrameId, ownerBackendNodeId] of child.frameOwnerBackendNodeIds ?? []) {
captured.frameOwnerBackendNodeIds.set(childFrameId, ownerBackendNodeId);
@@ -0,0 +1,125 @@
import type { CdpRunner } from "../shared";
function abortError(): Error {
const error = new Error("hover perception aborted");
error.name = "AbortError";
return error;
}
export async function waitForHover(ms: number, signal?: AbortSignal): Promise<void> {
if (signal?.aborted) throw abortError();
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => {
signal?.removeEventListener("abort", onAbort);
resolve();
}, ms);
const onAbort = () => {
clearTimeout(timer);
signal?.removeEventListener("abort", onAbort);
reject(abortError());
};
signal?.addEventListener("abort", onAbort, { once: true });
});
}
export async function clearHover(cdp: CdpRunner, tabId: number): Promise<void> {
await cdp
.send(tabId, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: -10,
y: -10,
})
.catch(() =>
cdp.send(tabId, "Input.dispatchMouseEvent", { type: "mouseMoved", x: 0, y: 0 }).catch(() => {
// Best effort cleanup.
}),
);
}
/**
* Wall-clock budget for a probe phase.
*
* Callers ask whether the *next* candidate still fits before starting it,
* rather than checking elapsed time at the top of the loop. A top-of-loop
* check lets a candidate start with 1ms of budget left and then run to
* completion, overshooting by a full settle window.
*/
export class ProbeBudget {
private readonly startedAt = Date.now();
constructor(private readonly totalMs: number) {}
get elapsedMs(): number {
return Date.now() - this.startedAt;
}
/** True when a candidate costing `estimatedMs` still fits in the budget. */
canAfford(estimatedMs: number): boolean {
return this.elapsedMs + estimatedMs <= this.totalMs;
}
}
export type OverlayBypass = (tabId: number, enabled: boolean) => Promise<void>;
export interface OverlayBypassScopeOptions {
/**
* Called when the overlay could not be restored. Defaults to `console.error`
* so a stuck-bypassed overlay is never silent; tests inject a spy.
*/
onRestoreFailure?: (error: unknown) => void;
}
const overlayBypassDepth = new Map<number, number>();
/**
* Runs `body` with the agent overlay bypassed, restoring it afterwards.
*
* Reference counted per tab so nested probe phases share a single bypass span
* instead of toggling the overlay once per phase. Restore failures are
* reported rather than swallowed, and the depth is released even when restore
* throws so a failure cannot pin the overlay off for the tab's lifetime.
*/
export async function withOverlayBypass<T>(
bypass: OverlayBypass | undefined,
tabId: number,
body: () => Promise<T>,
options: OverlayBypassScopeOptions = {},
): Promise<T> {
if (!bypass) return body();
const depth = overlayBypassDepth.get(tabId) ?? 0;
overlayBypassDepth.set(tabId, depth + 1);
if (depth === 0) {
try {
await bypass(tabId, true);
} catch (error) {
// Failing to hide the overlay only risks probing our own UI, so continue.
console.debug("[bsk hover] overlay bypass could not be enabled", error);
}
}
try {
return await body();
} finally {
const current = overlayBypassDepth.get(tabId) ?? 1;
if (current <= 1) {
overlayBypassDepth.delete(tabId);
try {
await bypass(tabId, false);
} catch (error) {
(options.onRestoreFailure ?? defaultRestoreFailure)(error);
}
} else {
overlayBypassDepth.set(tabId, current - 1);
}
}
}
function defaultRestoreFailure(error: unknown): void {
console.error("[bsk hover] agent overlay could not be restored after probing", error);
}
/** Test seam: drops any bypass depth left behind by a crashed probe. */
export function resetOverlayBypassDepth(): void {
overlayBypassDepth.clear();
}
@@ -0,0 +1,307 @@
import { isVomReferenceNode } from "@browser-skill/vom";
import { type CdpRunner, sendToCdpTarget } from "../shared";
import type { FrameDocument, FrameOwnedAxNode } from "./frame-document";
import { clearHover, ProbeBudget, waitForHover } from "./hover-perception";
import { stableIdentifierName } from "./semantic-graph/name-evidence";
import { frameBackendKey, type ResolvedSemanticGraph } from "./semantic-graph/types";
const MAX_TOOLTIP_CANDIDATES = 8;
const MAX_TOOLTIP_PROBE_MS = 2_400;
const TOOLTIP_SETTLE_MS = 250;
const CACHE_TTL_MS = 5 * 60_000;
/**
* Misses expire sooner than hits: a control that gains a tooltip once the app
* finishes booting should still be discoverable, but not at the cost of
* re-probing every unlabelled control on every observation.
*/
const MISS_CACHE_TTL_MS = 60_000;
const MAX_CACHE_ENTRIES = 500;
const FORM_CONTROL_TAGS = new Set(["input", "select", "textarea"]);
interface TooltipCacheEntry {
/** `null` records "probed, revealed no tooltip" so the miss is not re-paid. */
name: string | null;
expiresAt: number;
}
interface TooltipCandidate<T extends FrameOwnedAxNode> {
document: FrameDocument<T>;
backendNodeId: number;
x: number;
y: number;
stableIdentifier: boolean;
}
export interface TooltipNameProbeOptions {
signal?: AbortSignal;
}
const tooltipCache = new Map<string, TooltipCacheEntry>();
function cacheKey<T extends FrameOwnedAxNode>(
document: FrameDocument<T>,
backendNodeId: number,
): string {
return `${document.target.tabId}\u0000${document.frameId}\u0000${document.url ?? ""}\u0000${backendNodeId}`;
}
/** Returns the cache entry, or `undefined` when this control was never probed. */
function cachedEntry<T extends FrameOwnedAxNode>(
document: FrameDocument<T>,
backendNodeId: number,
): TooltipCacheEntry | undefined {
const key = cacheKey(document, backendNodeId);
const entry = tooltipCache.get(key);
if (!entry) return undefined;
if (entry.expiresAt <= Date.now()) {
tooltipCache.delete(key);
return undefined;
}
return entry;
}
function storeName<T extends FrameOwnedAxNode>(
document: FrameDocument<T>,
backendNodeId: number,
name: string | null,
): void {
if (tooltipCache.size >= MAX_CACHE_ENTRIES) {
const oldest = tooltipCache.keys().next().value;
if (oldest !== undefined) tooltipCache.delete(oldest);
}
tooltipCache.set(cacheKey(document, backendNodeId), {
name,
expiresAt: Date.now() + (name === null ? MISS_CACHE_TTL_MS : CACHE_TTL_MS),
});
}
function tooltipCollectorFunction(): string {
return `function() {
const ownRoot = this && typeof this.getRootNode === "function" ? this.getRootNode() : null;
const roots = ownRoot && ownRoot !== document ? [ownRoot, document] : [document];
const items = [];
const seen = new Set();
for (const root of roots) {
for (const el of Array.from(root.querySelectorAll("[role='tooltip']"))) {
const style = getComputedStyle(el);
const rect = el.getBoundingClientRect();
if (rect.width <= 0 || rect.height <= 0) continue;
if (style.display === "none" || style.visibility === "hidden" || style.opacity === "0") continue;
const text = String(el.getAttribute("aria-label") || el.getAttribute("title") || el.textContent || "")
.replace(/\\s+/g, " ")
.trim();
if (!text || text.length > 160 || seen.has(text)) continue;
seen.add(text);
items.push(text);
}
}
return items.slice(0, 20);
}`;
}
function cleanTooltipDiff(before: string[], after: string[]): string | undefined {
const existing = new Set(before.map((value) => value.toLowerCase()));
const added = [...new Set(after.map((value) => value.replace(/\s+/g, " ").trim()))].filter(
(value) => value && !existing.has(value.toLowerCase()),
);
return added.length === 1 ? added[0] : undefined;
}
function candidates<T extends FrameOwnedAxNode>(
documents: FrameDocument<T>[],
graph: ResolvedSemanticGraph,
): TooltipCandidate<T>[] {
const documentByFrame = new Map(documents.map((document) => [document.frameId, document]));
const byFrame = new Map<string, TooltipCandidate<T>[]>();
for (const node of graph.nodes.values()) {
if (
node.vom.name ||
FORM_CONTROL_TAGS.has(node.dom?.tag.toLowerCase() ?? "") ||
!node.referenceable ||
node.backendNodeId === undefined ||
!isVomReferenceNode({
...node.vom,
id: 0,
parentId: null,
backendNodeId: node.backendNodeId,
frameId: node.frameId,
referenceable: true,
})
) {
continue;
}
const document = documentByFrame.get(node.frameId);
const rect = node.dom?.rect;
if (!document || !rect || rect.w <= 0 || rect.h <= 0) continue;
if (
rect.x + rect.w <= 0 ||
rect.y + rect.h <= 0 ||
rect.x >= graph.viewport.width ||
rect.y >= graph.viewport.height
) {
continue;
}
if (
node.vom.disabled ||
node.vom.inert ||
node.vom.pointerEvents === "none" ||
node.vom.sensitive
) {
continue;
}
const frameCandidates = byFrame.get(node.frameId) ?? [];
frameCandidates.push({
document,
backendNodeId: node.backendNodeId,
x: (Math.max(0, rect.x) + Math.min(graph.viewport.width, rect.x + rect.w)) / 2,
y: (Math.max(0, rect.y) + Math.min(graph.viewport.height, rect.y + rect.h)) / 2,
stableIdentifier: stableIdentifierName(node.dom?.attrs ?? {}) !== undefined,
});
byFrame.set(node.frameId, frameCandidates);
}
const out: TooltipCandidate<T>[] = [];
for (const stableIdentifier of [true, false]) {
const queues = documents.map((document) =>
(byFrame.get(document.frameId) ?? []).filter(
(candidate) => candidate.stableIdentifier === stableIdentifier,
),
);
for (let index = 0; queues.some((queue) => index < queue.length); index += 1) {
for (const queue of queues) {
const candidate = queue[index];
if (candidate) out.push(candidate);
}
}
}
return out;
}
async function executionContextId<T extends FrameOwnedAxNode>(
cdp: CdpRunner,
document: FrameDocument<T>,
): Promise<number | undefined> {
const result = await sendToCdpTarget<{ executionContextId?: number }>(
cdp,
document.target,
"Page.createIsolatedWorld",
{
frameId: document.frameId,
worldName: "bsk-vom-tooltip",
grantUniveralAccess: false,
},
);
return result.executionContextId;
}
async function resolvedObjectId<T extends FrameOwnedAxNode>(
cdp: CdpRunner,
document: FrameDocument<T>,
contextId: number,
backendNodeId: number,
): Promise<string | undefined> {
const result = await sendToCdpTarget<{
object?: { objectId?: string };
}>(cdp, document.target, "DOM.resolveNode", {
backendNodeId,
executionContextId: contextId,
});
return result.object?.objectId;
}
async function tooltipTexts<T extends FrameOwnedAxNode>(
cdp: CdpRunner,
document: FrameDocument<T>,
objectId: string,
): Promise<string[]> {
const result = await sendToCdpTarget<{
result?: { value?: string[] };
}>(cdp, document.target, "Runtime.callFunctionOn", {
functionDeclaration: tooltipCollectorFunction(),
objectId,
returnByValue: true,
});
return result.result?.value ?? [];
}
async function releaseObject<T extends FrameOwnedAxNode>(
cdp: CdpRunner,
document: FrameDocument<T>,
objectId: string,
): Promise<void> {
await sendToCdpTarget(cdp, document.target, "Runtime.releaseObject", { objectId }).catch(
() => undefined,
);
}
export async function probeTooltipNames<T extends FrameOwnedAxNode>(
cdp: CdpRunner,
tabId: number,
documents: FrameDocument<T>[],
graph: ResolvedSemanticGraph,
options: TooltipNameProbeOptions = {},
): Promise<Map<string, string>> {
const names = new Map<string, string>();
const pending: TooltipCandidate<T>[] = [];
for (const candidate of candidates(documents, graph)) {
const cached = cachedEntry(candidate.document, candidate.backendNodeId);
if (cached) {
// A cached miss is still a decision: skip the control instead of re-probing.
if (cached.name !== null) {
names.set(
frameBackendKey(candidate.document.frameId, candidate.backendNodeId),
cached.name,
);
}
} else if (pending.length < MAX_TOOLTIP_CANDIDATES) {
pending.push(candidate);
}
}
if (pending.length === 0) return names;
const budget = new ProbeBudget(MAX_TOOLTIP_PROBE_MS);
const contexts = new Map<string, number>();
try {
for (const candidate of pending) {
if (options.signal?.aborted) throw new DOMException("observation aborted", "AbortError");
if (!budget.canAfford(TOOLTIP_SETTLE_MS)) break;
let objectId: string | undefined;
try {
let contextId = contexts.get(candidate.document.frameId);
if (contextId === undefined) {
contextId = await executionContextId(cdp, candidate.document);
if (contextId === undefined) continue;
contexts.set(candidate.document.frameId, contextId);
}
objectId = await resolvedObjectId(
cdp,
candidate.document,
contextId,
candidate.backendNodeId,
);
if (!objectId) continue;
await clearHover(cdp, tabId);
const before = await tooltipTexts(cdp, candidate.document, objectId);
await cdp.send(tabId, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: candidate.x,
y: candidate.y,
});
await waitForHover(TOOLTIP_SETTLE_MS, options.signal);
const after = await tooltipTexts(cdp, candidate.document, objectId);
const name = cleanTooltipDiff(before, after);
storeName(candidate.document, candidate.backendNodeId, name ?? null);
if (!name) continue;
names.set(frameBackendKey(candidate.document.frameId, candidate.backendNodeId), name);
} catch (error) {
if (error instanceof Error && error.name === "AbortError") throw error;
} finally {
if (objectId) await releaseObject(cdp, candidate.document, objectId);
await clearHover(cdp, tabId);
}
}
} finally {
await clearHover(cdp, tabId);
}
return names;
}
@@ -1,5 +1,6 @@
import type { Rect, RenderedRef, VomResult } from "@browser-skill/vom";
import type { CdpTarget } from "@/browser-driver/frame-graph";
import { readRecordingDocumentIdentity } from "@/shared/recording-document-identity";
import type { CapturedSurfaceProbe } from "./capture";
import type { CapturedFrameDocument, FrameAxNode } from "./frame-capture";
@@ -9,6 +10,8 @@ export interface CaptureVomFrame {
target: CdpTarget;
parentFrameId?: string;
ownerBackendNodeId?: number;
/** BrowserSkill-generated identity for the recording agent in this Document. */
recordingDocumentId?: string;
}
/** Allowlisted geometry used to match a recorded action to a rendered ref. */
@@ -34,17 +37,36 @@ export interface CaptureVomObservationResult {
frames: CaptureVomFrame[];
matchNodes: CaptureVomMatchNode[];
surfaceProbes?: CapturedSurfaceProbe[];
/**
* Whether this observation actively hovered the page, and whether that
* revealed content the static tree does not contain. Lets callers judge how
* far the page may have drifted from the returned snapshot.
*/
hoverProbe?: HoverProbeReport;
}
export interface HoverProbeReport {
performed: boolean;
revealedContent: boolean;
}
function projectFrames(documents: CapturedFrameDocument<FrameAxNode>[]): CaptureVomFrame[] {
return documents.map((document) => ({
frameId: document.frameId,
target: document.target,
...(document.parentFrameId ? { parentFrameId: document.parentFrameId } : {}),
...(document.ownerBackendNodeId !== undefined
? { ownerBackendNodeId: document.ownerBackendNodeId }
: {}),
}));
return documents.map((document) => {
let recordingDocumentId: string | undefined;
for (const node of document.domNodes) {
recordingDocumentId = readRecordingDocumentIdentity(node.attrs);
if (recordingDocumentId) break;
}
return {
frameId: document.frameId,
target: document.target,
...(document.parentFrameId ? { parentFrameId: document.parentFrameId } : {}),
...(document.ownerBackendNodeId !== undefined
? { ownerBackendNodeId: document.ownerBackendNodeId }
: {}),
...(recordingDocumentId ? { recordingDocumentId } : {}),
};
});
}
function projectMatchNodes(documents: CapturedFrameDocument<FrameAxNode>[]): CaptureVomMatchNode[] {
@@ -64,6 +86,7 @@ export function projectRecordSafeObservation(input: {
frameDocuments: CapturedFrameDocument<FrameAxNode>[];
rendered: VomResult;
surfaceProbes?: CapturedSurfaceProbe[];
hoverProbe?: HoverProbeReport;
}): CaptureVomObservationResult {
return {
text: input.rendered.text,
@@ -73,5 +96,6 @@ export function projectRecordSafeObservation(input: {
frames: projectFrames(input.frameDocuments),
matchNodes: projectMatchNodes(input.frameDocuments),
surfaceProbes: input.surfaceProbes,
...(input.hoverProbe ? { hoverProbe: input.hoverProbe } : {}),
};
}
@@ -0,0 +1,65 @@
import type { Rect } from "@browser-skill/vom";
import type { RenderedSurface, RenderedSurfaceGroup } from "./types";
const STACK_IOU_RATIO = 0.9;
const STACK_SIZE_RATIO = 0.9;
function area(rect: Rect): number {
return Math.max(0, rect.w) * Math.max(0, rect.h);
}
function intersectionArea(a: Rect, b: Rect): number {
const width = Math.max(0, Math.min(a.x + a.w, b.x + b.w) - Math.max(a.x, b.x));
const height = Math.max(0, Math.min(a.y + a.h, b.y + b.h) - Math.max(a.y, b.y));
return width * height;
}
function sameVisualStack(a: RenderedSurface, b: RenderedSurface): boolean {
if (a.frameId !== b.frameId) return false;
if (a.parentBackendNodeId !== b.parentBackendNodeId) return false;
const aArea = area(a.visibleRect);
const bArea = area(b.visibleRect);
const intersection = intersectionArea(a.visibleRect, b.visibleRect);
const union = aArea + bArea - intersection;
const widthRatio =
Math.min(a.visibleRect.w, b.visibleRect.w) / Math.max(a.visibleRect.w, b.visibleRect.w);
const heightRatio =
Math.min(a.visibleRect.h, b.visibleRect.h) / Math.max(a.visibleRect.h, b.visibleRect.h);
return (
union > 0 &&
intersection / union >= STACK_IOU_RATIO &&
widthRatio >= STACK_SIZE_RATIO &&
heightRatio >= STACK_SIZE_RATIO
);
}
function toGroup(members: RenderedSurface[]): RenderedSurfaceGroup {
const representative = [...members].sort(
(a, b) => area(b.visibleRect) - area(a.visibleRect) || b.paintOrder - a.paintOrder,
)[0];
return {
frameId: representative.frameId,
members,
representative,
label: members.map((surface) => surface.label).find((label) => label !== undefined),
};
}
export function clusterRenderedSurfaces(surfaces: RenderedSurface[]): RenderedSurfaceGroup[] {
const groups: RenderedSurface[][] = [];
for (const surface of surfaces) {
const matching = groups.find((group) =>
group.some((member) => sameVisualStack(member, surface)),
);
if (matching) matching.push(surface);
else groups.push([surface]);
}
return groups
.map(toGroup)
.sort(
(a, b) =>
a.frameId.localeCompare(b.frameId) ||
a.representative.visibleRect.y - b.representative.visibleRect.y ||
a.representative.visibleRect.x - b.representative.visibleRect.x,
);
}
@@ -0,0 +1,113 @@
import type { Rect, Viewport } from "@browser-skill/vom";
import type { CapturedNode } from "../capture";
import type { CapturedFrameDocument } from "../frame-capture";
import type { SemanticAxNode } from "../semantic-graph";
import type { RenderedSurface } from "./types";
function clean(value: string | undefined): string | undefined {
const normalized = value?.replace(/\s+/g, " ").trim();
return normalized || undefined;
}
function intersectViewport(rect: Rect, viewport: Viewport): Rect | null {
return intersectRect(rect, { x: 0, y: 0, w: viewport.width, h: viewport.height });
}
function intersectRect(a: Rect, b: Rect): Rect | null {
const x = Math.max(a.x, b.x);
const y = Math.max(a.y, b.y);
const right = Math.min(a.x + a.w, b.x + b.w);
const bottom = Math.min(a.y + a.h, b.y + b.h);
if (right <= x || bottom <= y) return null;
return { x, y, w: right - x, h: bottom - y };
}
const CLIPPING_OVERFLOW = new Set(["auto", "clip", "hidden", "scroll"]);
function clipToOverflowAncestors(
node: CapturedNode,
rect: Rect,
nodesByBackend: ReadonlyMap<number, CapturedNode>,
): Rect | null {
let clipped: Rect | null = rect;
let parentBackendNodeId = node.parentBackendNodeId;
let guard = 0;
while (clipped && parentBackendNodeId !== null && guard <= nodesByBackend.size) {
const parent = nodesByBackend.get(parentBackendNodeId);
if (!parent) break;
if (parent.rect) {
const clipX = CLIPPING_OVERFLOW.has(parent.overflowX ?? "visible");
const clipY = CLIPPING_OVERFLOW.has(parent.overflowY ?? "visible");
if (clipX || clipY) {
const clipRect = {
x: clipX ? parent.rect.x : clipped.x,
y: clipY ? parent.rect.y : clipped.y,
w: clipX ? parent.rect.w : clipped.w,
h: clipY ? parent.rect.h : clipped.h,
};
clipped = intersectRect(clipped, clipRect);
}
}
parentBackendNodeId = parent.parentBackendNodeId;
guard += 1;
}
return clipped;
}
function excludedByDomPolicy(
node: CapturedNode,
nodesByBackend: Map<number, CapturedNode>,
excludedBackendNodeIds: ReadonlySet<number>,
): boolean {
let current: CapturedNode | undefined = node;
let guard = 0;
while (current && guard <= nodesByBackend.size) {
if (excludedBackendNodeIds.has(current.backendNodeId)) return true;
if (current.visuallySuppressed === true) return true;
const attrs = current.attrs;
if (
Object.prototype.hasOwnProperty.call(attrs, "hidden") ||
Object.prototype.hasOwnProperty.call(attrs, "inert") ||
(attrs["aria-hidden"] ?? "").toLowerCase() === "true"
) {
return true;
}
current =
current.parentBackendNodeId === null
? undefined
: nodesByBackend.get(current.parentBackendNodeId);
guard += 1;
}
return false;
}
export function discoverRenderedSurfaces(
documents: CapturedFrameDocument<SemanticAxNode>[],
viewport: Viewport,
excludedBackendNodeIds: ReadonlySet<number> = new Set(),
): RenderedSurface[] {
const surfaces: RenderedSurface[] = [];
for (const document of documents) {
const nodesByBackend = new Map(document.domNodes.map((node) => [node.backendNodeId, node]));
for (const node of document.domNodes) {
if (node.tag.toLowerCase() !== "canvas" || node.rendered !== true || !node.rect) continue;
if (excludedByDomPolicy(node, nodesByBackend, excludedBackendNodeIds)) continue;
const viewportRect = intersectViewport(node.rect, viewport);
const visibleRect = viewportRect
? clipToOverflowAncestors(node, viewportRect, nodesByBackend)
: null;
if (!visibleRect) continue;
const label = clean(node.attrs["aria-label"] ?? node.attrs.title);
surfaces.push({
renderingKind: "canvas",
frameId: document.frameId,
backendNodeId: node.backendNodeId,
parentBackendNodeId: node.parentBackendNodeId,
visibleRect,
paintOrder: node.paintOrder,
...(label ? { label } : {}),
});
}
}
return surfaces;
}
@@ -0,0 +1,4 @@
export { clusterRenderedSurfaces } from "./cluster";
export { discoverRenderedSurfaces } from "./discovery";
export { projectRenderedSurfaces } from "./project";
export type { RenderedSurface, RenderedSurfaceGroup } from "./types";
@@ -0,0 +1,100 @@
import type { VomScene, VomVisualSurface } from "@browser-skill/vom";
import type { CapturedFrameDocument } from "../frame-capture";
import type { SemanticAxNode } from "../semantic-graph";
import type { RenderedSurfaceGroup } from "./types";
function frameBackendKey(frameId: string, backendNodeId: number): string {
return `${frameId}\u0000${backendNodeId}`;
}
interface ProjectionIndex {
sceneByBackend: ReadonlyMap<string, number>;
documentByFrame: ReadonlyMap<string, CapturedFrameDocument<SemanticAxNode>>;
domByFrame: ReadonlyMap<
string,
ReadonlyMap<number, CapturedFrameDocument<SemanticAxNode>["domNodes"][number]>
>;
rootSceneIdByFrame: ReadonlyMap<string, number>;
}
function nearestSceneParent(index: ProjectionIndex, group: RenderedSurfaceGroup): number | null {
const document = index.documentByFrame.get(group.frameId);
const domByBackend = index.domByFrame.get(group.frameId) ?? new Map();
let parentBackendNodeId = group.representative.parentBackendNodeId;
let guard = 0;
while (parentBackendNodeId !== null && guard <= domByBackend.size) {
const sceneId = index.sceneByBackend.get(frameBackendKey(group.frameId, parentBackendNodeId));
if (sceneId !== undefined) return sceneId;
parentBackendNodeId = domByBackend.get(parentBackendNodeId)?.parentBackendNodeId ?? null;
guard += 1;
}
if (document?.parentFrameId && document.ownerBackendNodeId !== undefined) {
const owner = index.sceneByBackend.get(
frameBackendKey(document.parentFrameId, document.ownerBackendNodeId),
);
if (owner !== undefined) return owner;
}
return index.rootSceneIdByFrame.get(group.frameId) ?? null;
}
function buildProjectionIndex(
scene: VomScene,
documents: CapturedFrameDocument<SemanticAxNode>[],
): ProjectionIndex {
const rootSceneIdByFrame = new Map<string, number>();
for (const node of scene.nodes) {
if (
node.frameId &&
!rootSceneIdByFrame.has(node.frameId) &&
["rootwebarea", "webarea"].includes(node.role?.toLowerCase() ?? "")
) {
rootSceneIdByFrame.set(node.frameId, node.id);
}
}
return {
sceneByBackend: new Map(
scene.nodes.flatMap((node) =>
node.backendNodeId === undefined
? []
: [[frameBackendKey(node.frameId ?? "", node.backendNodeId), node.id] as const],
),
),
documentByFrame: new Map(documents.map((document) => [document.frameId, document])),
domByFrame: new Map(
documents.map((document) => [
document.frameId,
new Map(document.domNodes.map((node) => [node.backendNodeId, node])),
]),
),
rootSceneIdByFrame,
};
}
export function projectRenderedSurfaces(
scene: VomScene,
documents: CapturedFrameDocument<SemanticAxNode>[],
groups: RenderedSurfaceGroup[],
): VomScene {
const index = buildProjectionIndex(scene, documents);
const visualSurfaces: VomVisualSurface[] = [];
for (const group of groups) {
const parentId = nearestSceneParent(index, group);
visualSurfaces.push({
parentId,
backendNodeId: group.representative.backendNodeId,
frameId: group.frameId,
renderingKind: "canvas",
visibleRect: group.representative.visibleRect,
...(group.label ? { label: group.label } : {}),
memberCount: group.members.length,
});
}
return {
...scene,
...(visualSurfaces.length > 0 ? { visualSurfaces } : {}),
};
}
@@ -0,0 +1,18 @@
import type { Rect } from "@browser-skill/vom";
export interface RenderedSurface {
renderingKind: "canvas";
frameId: string;
backendNodeId: number;
parentBackendNodeId: number | null;
visibleRect: Rect;
paintOrder: number;
label?: string;
}
export interface RenderedSurfaceGroup {
frameId: string;
members: RenderedSurface[];
representative: RenderedSurface;
label?: string;
}
@@ -0,0 +1,154 @@
import type { Viewport } from "@browser-skill/vom";
import type { FrameDocument } from "../frame-document";
import {
frameAxKey,
frameBackendKey,
type SemanticAxNode,
type SemanticFrame,
type SemanticGraph,
type SemanticGraphNode,
type SemanticNodeId,
} from "./types";
export interface BuildSemanticGraphInput {
documents: FrameDocument<SemanticAxNode>[];
viewport: Viewport;
rootFrameId?: string;
excludedBackendNodeIds?: ReadonlySet<number>;
}
function domNodeId(frameId: string, backendNodeId: number): SemanticNodeId {
return JSON.stringify([frameId, "dom", backendNodeId]);
}
function axNodeId(frameId: string, nodeId: string): SemanticNodeId {
return JSON.stringify([frameId, "ax", nodeId]);
}
export function buildSemanticGraph(input: BuildSemanticGraphInput): SemanticGraph {
const rootFrameId =
input.rootFrameId ??
input.documents.find((document) => !document.parentFrameId)?.frameId ??
input.documents[0]?.frameId ??
"root";
const frames = new Map<string, SemanticFrame>();
const nodes = new Map<SemanticNodeId, SemanticGraphNode>();
const nodeIdsByFrame = new Map<string, SemanticNodeId[]>();
const nodeByFrameBackend = new Map<string, SemanticNodeId>();
const nodeByFrameAx = new Map<string, SemanticNodeId>();
for (const [frameOrder, document] of input.documents.entries()) {
frames.set(document.frameId, {
frameId: document.frameId,
...(document.parentFrameId ? { parentFrameId: document.parentFrameId } : {}),
...(document.ownerBackendNodeId !== undefined
? { ownerBackendNodeId: document.ownerBackendNodeId }
: {}),
contextScopeId: document.contextScopeId,
target: document.target,
...(document.url ? { url: document.url } : {}),
order: frameOrder,
});
const frameNodeIds: SemanticNodeId[] = [];
nodeIdsByFrame.set(document.frameId, frameNodeIds);
for (const [index, dom] of document.domNodes.entries()) {
const id = domNodeId(document.frameId, dom.backendNodeId);
const node: SemanticGraphNode = {
id,
frameId: document.frameId,
backendNodeId: dom.backendNodeId,
dom,
sourceOrder: index,
excluded:
document.frameId === rootFrameId &&
input.excludedBackendNodeIds?.has(dom.backendNodeId) === true,
};
nodes.set(id, node);
frameNodeIds.push(id);
nodeByFrameBackend.set(frameBackendKey(document.frameId, dom.backendNodeId), id);
}
for (const [index, ax] of document.axNodes.entries()) {
const joinedId =
typeof ax.backendDOMNodeId === "number"
? nodeByFrameBackend.get(frameBackendKey(document.frameId, ax.backendDOMNodeId))
: undefined;
const id = joinedId ?? axNodeId(document.frameId, ax.nodeId);
const existing = nodes.get(id);
if (existing) {
existing.ax = ax;
existing.axNodeId = ax.nodeId;
} else {
nodes.set(id, {
id,
frameId: document.frameId,
...(typeof ax.backendDOMNodeId === "number"
? { backendNodeId: ax.backendDOMNodeId }
: {}),
axNodeId: ax.nodeId,
ax,
sourceOrder: document.domNodes.length + index,
excluded:
document.frameId === rootFrameId &&
typeof ax.backendDOMNodeId === "number" &&
input.excludedBackendNodeIds?.has(ax.backendDOMNodeId) === true,
});
frameNodeIds.push(id);
}
nodeByFrameAx.set(frameAxKey(document.frameId, ax.nodeId), id);
}
}
for (const document of input.documents) {
for (const dom of document.domNodes) {
const id = nodeByFrameBackend.get(frameBackendKey(document.frameId, dom.backendNodeId));
const node = id ? nodes.get(id) : undefined;
if (!node || dom.parentBackendNodeId === null) continue;
node.domParentId = nodeByFrameBackend.get(
frameBackendKey(document.frameId, dom.parentBackendNodeId),
);
}
for (const ax of document.axNodes) {
const id = nodeByFrameAx.get(frameAxKey(document.frameId, ax.nodeId));
const node = id ? nodes.get(id) : undefined;
if (!node || !ax.parentId) continue;
node.axParentId = nodeByFrameAx.get(frameAxKey(document.frameId, ax.parentId));
}
}
const axChildren = new Map<SemanticNodeId, SemanticNodeId[]>();
const excludedQueue: SemanticNodeId[] = [];
for (const node of nodes.values()) {
if (node.axParentId) {
const children = axChildren.get(node.axParentId) ?? [];
children.push(node.id);
axChildren.set(node.axParentId, children);
}
if (node.excluded) excludedQueue.push(node.id);
}
for (let index = 0; index < excludedQueue.length; index += 1) {
for (const childId of axChildren.get(excludedQueue[index]) ?? []) {
const child = nodes.get(childId);
if (!child || child.excluded) continue;
child.excluded = true;
excludedQueue.push(childId);
}
}
for (const frame of frames.values()) {
if (!frame.parentFrameId || frame.ownerBackendNodeId === undefined) continue;
frame.ownerNodeId = nodeByFrameBackend.get(
frameBackendKey(frame.parentFrameId, frame.ownerBackendNodeId),
);
}
return {
viewport: input.viewport,
rootFrameId,
frames,
nodes,
nodeIdsByFrame,
nodeByFrameBackend,
};
}
@@ -0,0 +1,59 @@
import type { Viewport, VomScene } from "@browser-skill/vom";
import type { FrameDocument } from "../frame-document";
import { buildSemanticGraph as buildGraph } from "./build";
import { projectSemanticGraph as projectGraph } from "./project";
import { resolveSemanticGraph as resolveGraph } from "./resolve";
import { normalizeSemanticStructure as normalizeStructure } from "./structure";
import type {
ResolvedSemanticGraph,
SemanticAxNode,
SemanticGraph,
StructuredSemanticGraph,
} from "./types";
export type { SemanticAxNode } from "./types";
export interface SemanticVomInput {
documents: FrameDocument<SemanticAxNode>[];
viewport: Viewport;
rootFrameId?: string;
excludedBackendNodeIds?: ReadonlySet<number>;
}
export function buildSemanticGraph(input: SemanticVomInput): SemanticGraph {
return buildGraph(input);
}
export function resolveSemanticGraph(
graph: SemanticGraph,
options: {
supplementalNames?: ReadonlyMap<string, string>;
identifierFallback?: boolean;
} = {},
): ResolvedSemanticGraph {
return resolveGraph(graph, options);
}
export function normalizeSemanticStructure(graph: ResolvedSemanticGraph): StructuredSemanticGraph {
return normalizeStructure(graph);
}
export function projectSemanticGraph(graph: StructuredSemanticGraph): VomScene {
return projectGraph(graph);
}
export function buildSemanticVomScene(
input: SemanticVomInput & {
supplementalNames?: ReadonlyMap<string, string>;
identifierFallback?: boolean;
},
): VomScene {
return projectSemanticGraph(
normalizeSemanticStructure(
resolveSemanticGraph(buildSemanticGraph(input), {
supplementalNames: input.supplementalNames,
identifierFallback: input.identifierFallback,
}),
),
);
}
@@ -0,0 +1,34 @@
const IDENTIFIER_ATTRS = ["data-testid", "data-test-id", "id"] as const;
const GENERIC_TOKENS = new Set(["btn", "button", "control", "icon", "item", "test", "tool"]);
const GENERATED_PREFIXES = new Set(["ember", "headlessui", "mui", "radix", "react", "rc"]);
function identifierTokens(value: string): string[] {
return value
.replace(/([a-z0-9])([A-Z])/g, "$1 $2")
.split(/[^a-zA-Z0-9]+/)
.map((token) => token.toLowerCase())
.filter(Boolean);
}
function readableIdentifier(value: string | undefined): string | undefined {
const raw = value?.trim();
if (!raw || raw.length < 3 || raw.length > 80) return undefined;
if (/^[a-f0-9]{8,}$/i.test(raw)) return undefined;
const tokens = identifierTokens(raw);
if (GENERATED_PREFIXES.has(tokens[0])) return undefined;
while (tokens.length > 1 && GENERIC_TOKENS.has(tokens[0])) tokens.shift();
while (tokens.length > 1 && GENERIC_TOKENS.has(tokens[tokens.length - 1])) tokens.pop();
if (tokens.length === 0 || !tokens.some((token) => /[a-z]{3,}/.test(token))) return undefined;
return tokens
.map((token, index) => (index === 0 ? `${token[0].toUpperCase()}${token.slice(1)}` : token))
.join(" ");
}
export function stableIdentifierName(attrs: Readonly<Record<string, string>>): string | undefined {
for (const name of IDENTIFIER_ATTRS) {
const label = readableIdentifier(attrs[name]);
if (label) return label;
}
return undefined;
}
@@ -0,0 +1,85 @@
import type { VomNode, VomScene } from "@browser-skill/vom";
import type { SemanticNodeId, StructuredSemanticGraph, StructuredSemanticNode } from "./types";
function retained(node: StructuredSemanticNode): boolean {
return node.structure.kind === "keep" || node.structure.kind === "inferred-group";
}
function connectedFrameIds(graph: StructuredSemanticGraph): Set<string> {
const connected = new Set<string>([graph.rootFrameId]);
const childrenByParent = new Map<string, string[]>();
for (const frame of graph.frames.values()) {
if (!frame.parentFrameId) continue;
const children = childrenByParent.get(frame.parentFrameId) ?? [];
children.push(frame.frameId);
childrenByParent.set(frame.parentFrameId, children);
}
const queue = [graph.rootFrameId];
for (let index = 0; index < queue.length; index += 1) {
for (const frameId of childrenByParent.get(queue[index]) ?? []) {
if (connected.has(frameId)) continue;
const frame = graph.frames.get(frameId);
if (!frame?.ownerNodeId) continue;
const owner = graph.nodes.get(frame.ownerNodeId);
if (!owner || !retained(owner)) continue;
connected.add(frame.frameId);
queue.push(frame.frameId);
}
}
return connected;
}
function domAncestors(
graph: StructuredSemanticGraph,
node: StructuredSemanticNode,
numericId: ReadonlyMap<SemanticNodeId, number>,
): number[] {
const ancestors: number[] = [];
const seen = new Set<SemanticNodeId>();
let current = node.domParentId;
while (current && !seen.has(current)) {
seen.add(current);
const id = numericId.get(current);
if (id !== undefined) ancestors.push(id);
current = graph.nodes.get(current)?.domParentId;
}
return ancestors;
}
export function projectSemanticGraph(graph: StructuredSemanticGraph): VomScene {
const connectedFrames = connectedFrameIds(graph);
const orderedNodes = [...graph.frames.values()]
.filter((frame) => connectedFrames.has(frame.frameId))
.sort((a, b) => a.order - b.order)
.flatMap((frame) =>
(graph.nodeIdsByFrame.get(frame.frameId) ?? [])
.map((id) => graph.nodes.get(id))
.filter((node): node is StructuredSemanticNode => node !== undefined && retained(node))
.sort((a, b) => a.sourceOrder - b.sourceOrder),
);
const numericId = new Map<SemanticNodeId, number>();
orderedNodes.forEach((node, index) => numericId.set(node.id, index + 1));
const nodes: VomNode[] = orderedNodes.map((node) => ({
...node.vom,
id: numericId.get(node.id) as number,
parentId: node.semanticParentId ? (numericId.get(node.semanticParentId) ?? null) : null,
...(node.backendNodeId !== undefined ? { backendNodeId: node.backendNodeId } : {}),
frameId: node.frameId,
contextScopeId: graph.frames.get(node.frameId)?.contextScopeId ?? node.frameId,
referenceable: node.referenceable,
...(node.dom
? {
domParentId:
node.domParentId === undefined ? null : (numericId.get(node.domParentId) ?? null),
domAncestorIds: domAncestors(graph, node, numericId),
}
: {}),
}));
return {
viewport: graph.viewport,
nodes,
rootFrameId: graph.rootFrameId,
};
}
@@ -0,0 +1,521 @@
import type { VomNode } from "@browser-skill/vom";
import { stableIdentifierName } from "./name-evidence";
import type {
ResolvedSemanticGraph,
ResolvedSemanticNode,
SemanticAxNode,
SemanticGraph,
SemanticGraphNode,
SemanticNodeId,
} from "./types";
import { frameBackendKey } from "./types";
const FORM_TAGS = new Set(["input", "textarea", "select"]);
const NATIVE_CONTROL_TAGS = new Set(["button", "input", "select", "textarea"]);
const TEXT_AX_ROLES = new Set(["statictext", "inlinetextbox", "labeltext", "text"]);
const INTERACTIVE_ROLES = new Set([
"button",
"checkbox",
"combobox",
"link",
"listbox",
"menuitem",
"menuitemcheckbox",
"menuitemradio",
"option",
"radio",
"searchbox",
"slider",
"spinbutton",
"switch",
"tab",
"textbox",
"treeitem",
]);
const SENSITIVE_INPUT_TYPES = new Set([
"password",
"credit-card",
"one-time-code",
"current-password",
"new-password",
]);
function clean(value: string | undefined): string | undefined {
const normalized = value?.replace(/\s+/g, " ").trim();
return normalized || undefined;
}
function axValue(field?: { value?: string | number | boolean }): string | undefined {
return field?.value === undefined ? undefined : clean(String(field.value));
}
function axProperty(node: SemanticAxNode | undefined, name: string): string | undefined {
return axValue(node?.properties?.find((property) => property.name === name)?.value);
}
function normalizeRole(value: string | undefined): string | undefined {
const role = clean(value)?.toLowerCase();
return role || undefined;
}
function nativeRole(tag: string, attrs: Record<string, string>): string | undefined {
if (tag === "button") return "button";
if (tag === "textarea") return "textbox";
if (tag === "select") return attrs.multiple === undefined ? "combobox" : "listbox";
if (tag === "a" && attrs.href) return "link";
if (tag === "iframe") return "Iframe";
if (tag === "dialog") return "dialog";
if (tag === "img") return "img";
if (tag === "table") return "table";
if (tag === "tr") return "row";
if (tag === "th") return "columnheader";
if (tag === "td") return "cell";
if (tag === "ul" || tag === "ol") return "list";
if (tag === "li") return "listitem";
if (/^h[1-6]$/.test(tag)) return "heading";
if (tag === "nav") return "navigation";
if (tag === "main") return "main";
if (tag === "form") return "form";
if (tag === "input") {
const type = (attrs.type ?? "text").toLowerCase();
if (type === "hidden") return undefined;
if (type === "checkbox") return "checkbox";
if (type === "radio") return "radio";
if (type === "range") return "slider";
if (type === "number") return "spinbutton";
if (type === "search") return "searchbox";
if (["button", "submit", "reset", "image"].includes(type)) return "button";
return "textbox";
}
return undefined;
}
function domRole(tag: string, attrs: Record<string, string>): string | undefined {
return normalizeRole(attrs.role) ?? nativeRole(tag, attrs);
}
function resolvedRole(node: SemanticGraphNode): {
value?: string;
source: ResolvedSemanticNode["roleSource"];
} {
const rawAxRole = clean(axValue(node.ax?.role));
const axRole = normalizeRole(rawAxRole);
const attrs = node.dom?.attrs ?? {};
const explicitDomRole = normalizeRole(attrs.role);
const nativeDomRole = nativeRole(node.dom?.tag.toLowerCase() ?? "", attrs);
const usableAxRole = node.ax?.ignored !== true;
if (usableAxRole && axRole && !["generic", "none", "presentation"].includes(axRole)) {
return { value: rawAxRole, source: "ax" };
}
if (explicitDomRole) return { value: explicitDomRole, source: "dom-explicit" };
if (nativeDomRole) return { value: nativeDomRole, source: "dom-native" };
if (rawAxRole) {
return { value: rawAxRole, source: usableAxRole ? "ax" : "ax-ignored" };
}
return { source: "none" };
}
function sensitive(node: SemanticGraphNode): boolean {
const attrs = node.dom?.attrs ?? {};
const type = (attrs.type ?? "").toLowerCase();
const autocomplete = (attrs.autocomplete ?? "").toLowerCase();
return (
type === "password" ||
autocomplete.startsWith("cc-") ||
SENSITIVE_INPUT_TYPES.has(autocomplete) ||
SENSITIVE_INPUT_TYPES.has(axProperty(node.ax, "inputType") ?? "")
);
}
function inputState(
node: SemanticGraphNode,
role: string | undefined,
value: string | undefined,
isSensitive: boolean,
): VomNode["inputState"] {
if (node.dom?.formState) return node.dom.formState;
const tag = node.dom?.tag.toLowerCase() ?? "";
if (!FORM_TAGS.has(tag) && !["textbox", "searchbox"].includes(role ?? "")) return undefined;
if (isSensitive) return value ? "filled" : "empty";
if (node.dom?.formValue !== undefined) {
if (node.dom.formValue === "") return "empty";
return node.dom.formValue === (node.dom.formDefaultValue ?? "") ? "default" : "filled";
}
return value ? "filled" : "empty";
}
function externalHrefHost(
href: string | undefined,
pageUrl: string | undefined,
): string | undefined {
if (!href || !pageUrl) return undefined;
try {
const page = new URL(pageUrl);
const target = new URL(href, page);
return target.origin === page.origin ? undefined : target.hostname;
} catch {
return undefined;
}
}
interface ResolveIndexes {
domChildren: Map<SemanticNodeId, SemanticNodeId[]>;
axChildren: Map<SemanticNodeId, SemanticNodeId[]>;
domIdIndex: Map<string, SemanticNodeId>;
labelForIndex: Map<string, SemanticNodeId[]>;
domText: (nodeId: SemanticNodeId) => string | undefined;
axText: (nodeId: SemanticNodeId) => string | undefined;
hasNativeDescendant: (nodeId: SemanticNodeId) => boolean;
}
function buildIndexes(graph: SemanticGraph): ResolveIndexes {
const domChildren = new Map<SemanticNodeId, SemanticNodeId[]>();
const axChildren = new Map<SemanticNodeId, SemanticNodeId[]>();
const domIdIndex = new Map<string, SemanticNodeId>();
const labelForIndex = new Map<string, SemanticNodeId[]>();
for (const node of graph.nodes.values()) {
if (node.domParentId) {
const children = domChildren.get(node.domParentId) ?? [];
children.push(node.id);
domChildren.set(node.domParentId, children);
}
if (node.axParentId) {
const children = axChildren.get(node.axParentId) ?? [];
children.push(node.id);
axChildren.set(node.axParentId, children);
}
const id = clean(node.dom?.attrs.id);
if (id) domIdIndex.set(`${node.frameId}\u0000${id}`, node.id);
const targetId = clean(node.dom?.attrs.for);
if (node.dom?.tag.toLowerCase() === "label" && targetId) {
const key = `${node.frameId}\u0000${targetId}`;
const labels = labelForIndex.get(key) ?? [];
labels.push(node.id);
labelForIndex.set(key, labels);
}
}
for (const [parentId, children] of axChildren) {
const order = graph.nodes.get(parentId)?.ax?.childIds ?? [];
const orderByAxId = new Map(order.map((nodeId, index) => [nodeId, index]));
children.sort((a, b) => {
const aIndex = orderByAxId.get(graph.nodes.get(a)?.axNodeId ?? "") ?? -1;
const bIndex = orderByAxId.get(graph.nodes.get(b)?.axNodeId ?? "") ?? -1;
return (
(aIndex < 0 ? Number.MAX_SAFE_INTEGER : aIndex) -
(bIndex < 0 ? Number.MAX_SAFE_INTEGER : bIndex)
);
});
}
const textMemo = new Map<SemanticNodeId, string | undefined>();
const resolvingText = new Set<SemanticNodeId>();
const domText = (nodeId: SemanticNodeId): string | undefined => {
if (textMemo.has(nodeId)) return textMemo.get(nodeId);
if (resolvingText.has(nodeId)) return undefined;
resolvingText.add(nodeId);
const node = graph.nodes.get(nodeId);
const parts: string[] = [];
const own = clean(node?.dom?.textContent);
if (own) parts.push(own);
for (const childId of domChildren.get(nodeId) ?? []) {
const child = graph.nodes.get(childId);
const childRole = child
? domRole(child.dom?.tag.toLowerCase() ?? "", child.dom?.attrs ?? {})
: undefined;
if (childRole && INTERACTIVE_ROLES.has(childRole.toLowerCase())) continue;
const childText = domText(childId);
if (childText) parts.push(childText);
if (parts.join(" ").length >= 240) break;
}
resolvingText.delete(nodeId);
const result = clean(parts.join(" ").slice(0, 240));
textMemo.set(nodeId, result);
return result;
};
const axTextMemo = new Map<SemanticNodeId, string | undefined>();
const resolvingAxText = new Set<SemanticNodeId>();
const axText = (nodeId: SemanticNodeId): string | undefined => {
if (axTextMemo.has(nodeId)) return axTextMemo.get(nodeId);
if (resolvingAxText.has(nodeId)) return undefined;
resolvingAxText.add(nodeId);
const complete = (value: string | undefined): string | undefined => {
resolvingAxText.delete(nodeId);
axTextMemo.set(nodeId, value);
return value;
};
const node = graph.nodes.get(nodeId);
const role = normalizeRole(axValue(node?.ax?.role)) ?? "";
if (TEXT_AX_ROLES.has(role)) {
const own = clean(axValue(node?.ax?.name) ?? axValue(node?.ax?.value));
if (own) return complete(own);
}
const parts: string[] = [];
for (const childId of axChildren.get(nodeId) ?? []) {
const child = graph.nodes.get(childId);
const childRole = normalizeRole(axValue(child?.ax?.role)) ?? "";
if (INTERACTIVE_ROLES.has(childRole)) continue;
const childText = axText(childId);
if (childText) parts.push(childText);
if (parts.join(" ").length >= 240) break;
}
return complete(clean(parts.join(" ").slice(0, 240)));
};
const nativeMemo = new Map<SemanticNodeId, boolean>();
const hasNativeDescendant = (nodeId: SemanticNodeId): boolean => {
const cached = nativeMemo.get(nodeId);
if (cached !== undefined) return cached;
nativeMemo.set(nodeId, false);
for (const childId of domChildren.get(nodeId) ?? []) {
const child = graph.nodes.get(childId);
const childRole =
(child?.ax?.ignored === true ? undefined : normalizeRole(axValue(child?.ax?.role))) ??
(child ? domRole(child.dom?.tag.toLowerCase() ?? "", child.dom?.attrs ?? {}) : undefined);
if (
(child?.dom && NATIVE_CONTROL_TAGS.has(child.dom.tag.toLowerCase())) ||
INTERACTIVE_ROLES.has(childRole ?? "") ||
hasNativeDescendant(childId)
) {
nativeMemo.set(nodeId, true);
return true;
}
}
return false;
};
return {
domChildren,
axChildren,
domIdIndex,
labelForIndex,
domText,
axText,
hasNativeDescendant,
};
}
function ariaLabelledText(node: SemanticGraphNode, indexes: ResolveIndexes): string | undefined {
const ids = (node.dom?.attrs["aria-labelledby"] ?? "").split(/\s+/).filter(Boolean);
const parts = ids.flatMap((id) => {
const target = indexes.domIdIndex.get(`${node.frameId}\u0000${id}`);
return target ? (indexes.domText(target) ?? []) : [];
});
return clean(parts.join(" "));
}
function nativeLabelText(
node: SemanticGraphNode,
graph: SemanticGraph,
indexes: ResolveIndexes,
): string | undefined {
const id = clean(node.dom?.attrs.id);
if (id) {
const labels = indexes.labelForIndex.get(`${node.frameId}\u0000${id}`) ?? [];
const explicit = clean(
labels
.map((labelId) => indexes.domText(labelId))
.filter(Boolean)
.join(" "),
);
if (explicit) return explicit;
}
let parentId = node.domParentId;
let guard = 0;
while (parentId && guard <= graph.nodes.size) {
const parent = graph.nodes.get(parentId);
if (!parent) break;
if (parent.dom?.tag.toLowerCase() === "label") return indexes.domText(parentId);
parentId = parent.domParentId;
guard += 1;
}
return undefined;
}
function precedingAxText(
node: SemanticGraphNode,
graph: SemanticGraph,
indexes: ResolveIndexes,
): string | undefined {
if (!node.axParentId) return undefined;
const siblings = indexes.axChildren.get(node.axParentId) ?? [];
const index = siblings.indexOf(node.id);
for (let offset = 1; offset <= 4 && index - offset >= 0; offset += 1) {
const sibling = graph.nodes.get(siblings[index - offset]);
const role = normalizeRole(axValue(sibling?.ax?.role)) ?? "";
if (INTERACTIVE_ROLES.has(role)) break;
if (!TEXT_AX_ROLES.has(role) && role !== "generic") continue;
const text = clean(axValue(sibling?.ax?.name) ?? axValue(sibling?.ax?.value))?.replace(
/[::]\s*$/,
"",
);
if (text && text.length <= 80) return text;
}
return undefined;
}
function precedingDomText(
node: SemanticGraphNode,
graph: SemanticGraph,
indexes: ResolveIndexes,
): string | undefined {
if (!node.domParentId) return undefined;
const siblings = indexes.domChildren.get(node.domParentId) ?? [];
const index = siblings.indexOf(node.id);
for (let offset = 1; offset <= 4 && index - offset >= 0; offset += 1) {
const sibling = graph.nodes.get(siblings[index - offset]);
const siblingRole = sibling
? domRole(sibling.dom?.tag.toLowerCase() ?? "", sibling.dom?.attrs ?? {})
: undefined;
if (siblingRole && INTERACTIVE_ROLES.has(siblingRole.toLowerCase())) break;
const text = indexes.domText(siblings[index - offset])?.replace(/[::]\s*$/, "");
if (text && text.length <= 80) return text;
}
return undefined;
}
function resolvedName(
node: SemanticGraphNode,
role: string | undefined,
graph: SemanticGraph,
indexes: ResolveIndexes,
supplementalNames: ReadonlyMap<string, string>,
identifierFallback: boolean,
): string | undefined {
const attrs = node.dom?.attrs ?? {};
const formControl = FORM_TAGS.has(node.dom?.tag.toLowerCase() ?? "");
const descendantText = indexes.domText(node.id);
const interactive =
INTERACTIVE_ROLES.has(role?.toLowerCase() ?? "") ||
NATIVE_CONTROL_TAGS.has(node.dom?.tag.toLowerCase() ?? "");
const candidates = [
node.ax?.ignored === true ? undefined : axValue(node.ax?.name),
ariaLabelledText(node, indexes),
clean(attrs["aria-label"]),
nativeLabelText(node, graph, indexes),
clean(attrs.title),
clean(attrs.alt),
node.ax?.ignored === true ? undefined : indexes.axText(node.id),
descendantText,
formControl ? precedingAxText(node, graph, indexes) : undefined,
formControl ? precedingDomText(node, graph, indexes) : undefined,
formControl ? clean(node.dom?.formPlaceholder ?? attrs.placeholder) : undefined,
formControl || !interactive || node.backendNodeId === undefined
? undefined
: clean(supplementalNames.get(frameBackendKey(node.frameId, node.backendNodeId))),
identifierFallback && interactive ? stableIdentifierName(attrs) : undefined,
];
let name = candidates.find((candidate) => candidate !== undefined);
const hasPopup = axProperty(node.ax, "hasPopup");
if (name && hasPopup && hasPopup !== "false") {
name =
axProperty(node.ax, "expanded") === "true" ? `${name} [expanded]` : `${name} [has-submenu]`;
}
if (!name && role?.toLowerCase() === "iframe") return clean(attrs.id);
return name;
}
function insideNative(node: SemanticGraphNode, graph: SemanticGraph): boolean {
let parentId = node.domParentId;
let guard = 0;
while (parentId && guard <= graph.nodes.size) {
const parent = graph.nodes.get(parentId);
if (!parent) break;
if (parent.dom && NATIVE_CONTROL_TAGS.has(parent.dom.tag.toLowerCase())) return true;
parentId = parent.domParentId;
guard += 1;
}
return false;
}
function nearbyText(
node: SemanticGraphNode,
graph: SemanticGraph,
indexes: ResolveIndexes,
): string | undefined {
if (!node.domParentId) return undefined;
const siblings = indexes.domChildren.get(node.domParentId) ?? [];
const index = siblings.indexOf(node.id);
if (index < 0) return undefined;
const texts = siblings
.slice(Math.max(0, index - 3), index + 4)
.filter((id) => id !== node.id)
.flatMap((id) => indexes.domText(id) ?? [])
.slice(0, 3);
return clean(texts.join(" "));
}
export function resolveSemanticGraph(
graph: SemanticGraph,
options: {
supplementalNames?: ReadonlyMap<string, string>;
identifierFallback?: boolean;
} = {},
): ResolvedSemanticGraph {
const indexes = buildIndexes(graph);
const supplementalNames = options.supplementalNames ?? new Map<string, string>();
const identifierFallback = options.identifierFallback ?? true;
const nodes = new Map<SemanticNodeId, ResolvedSemanticNode>();
for (const node of graph.nodes.values()) {
const roleResolution = resolvedRole(node);
const role = roleResolution.value;
const name = resolvedName(node, role, graph, indexes, supplementalNames, identifierFallback);
const isSensitive = sensitive(node);
const sourceValue = node.dom?.formValue ?? axValue(node.ax?.value);
const value = isSensitive ? undefined : sourceValue;
const attrs = { ...(node.dom?.attrs ?? {}) };
if (isSensitive) delete attrs.value;
const frame = graph.frames.get(node.frameId);
const tag = node.dom?.tag.toLowerCase() ?? "";
const vom: ResolvedSemanticNode["vom"] = {
tag,
rect: node.dom?.rect ?? null,
paintOrder: node.dom?.paintOrder ?? 0,
position: node.dom?.position || "static",
pointerEvents: node.dom?.pointerEvents || "auto",
...(node.dom?.cursor ? { cursor: node.dom.cursor } : {}),
...(role ? { role } : {}),
...(name ? { name } : {}),
...(value !== undefined ? { value } : {}),
...(clean(node.dom?.formPlaceholder ?? attrs.placeholder)
? { placeholder: clean(node.dom?.formPlaceholder ?? attrs.placeholder) }
: {}),
attrs,
...(clean(node.dom?.textContent) ? { text: clean(node.dom?.textContent) } : {}),
...(nearbyText(node, graph, indexes) ? { nearbyText: nearbyText(node, graph, indexes) } : {}),
inputState: inputState(node, role, sourceValue, isSensitive),
sensitive: isSensitive,
modal:
["dialog", "alertdialog"].includes(role?.toLowerCase() ?? "") ||
tag === "dialog" ||
(attrs["aria-modal"] ?? "").toLowerCase() === "true",
disabled:
Object.prototype.hasOwnProperty.call(attrs, "disabled") ||
(attrs["aria-disabled"] ?? "").toLowerCase() === "true",
inert: Object.prototype.hasOwnProperty.call(attrs, "inert"),
hasNativeDescendant: indexes.hasNativeDescendant(node.id),
insideNative: insideNative(node, graph),
...(role?.toLowerCase() === "link" ? { href: externalHrefHost(attrs.href, frame?.url) } : {}),
};
const selected = node.ax?.ignored === true ? undefined : axProperty(node.ax, "selected");
const expanded = node.ax?.ignored === true ? undefined : axProperty(node.ax, "expanded");
const controls = node.ax?.ignored === true ? undefined : axProperty(node.ax, "controls");
if (selected === "true" && !vom.attrs?.["aria-selected"]) {
vom.attrs = { ...(vom.attrs ?? {}), "aria-selected": "true" };
}
if (expanded === "true" && !vom.attrs?.["aria-expanded"]) {
vom.attrs = { ...(vom.attrs ?? {}), "aria-expanded": "true" };
}
if (controls && !vom.attrs?.["aria-controls"]) {
vom.attrs = { ...(vom.attrs ?? {}), "aria-controls": controls };
}
nodes.set(node.id, {
...node,
vom,
referenceable: node.backendNodeId !== undefined,
roleSource: roleResolution.source,
});
}
return { ...graph, nodes };
}
@@ -0,0 +1,456 @@
import {
applyVomInteractionRecovery,
isVomReferenceNode,
isVomStructuralRole,
type VomNode,
} from "@browser-skill/vom";
import type {
ResolvedSemanticGraph,
ResolvedSemanticNode,
SemanticNodeId,
StructureDecision,
StructuredSemanticGraph,
StructuredSemanticNode,
} from "./types";
const TRANSPARENT_ROLES = new Set(["", "generic", "none", "presentation", "inlinetextbox"]);
const ROOT_ROLES = new Set(["rootwebarea", "webarea"]);
const CONTENT_BOUNDARY_ROLES = new Set([
"article",
"dialog",
"document",
"feed",
"form",
"grid",
"heading",
"iframe",
"list",
"main",
"navigation",
"paragraph",
"region",
"section",
"table",
"tree",
]);
function hasMeaningfulRelations(node: ResolvedSemanticNode): boolean {
const attrs = node.vom.attrs ?? {};
return ["aria-controls", "aria-labelledby", "aria-owns", "aria-describedby"].some((name) =>
attrs[name]?.trim(),
);
}
function explicitStructural(node: ResolvedSemanticNode): boolean {
return (
isVomStructuralRole(node.vom.role) &&
node.roleSource !== "none" &&
node.roleSource !== "ax-ignored"
);
}
function hasUsableAxSemantics(node: ResolvedSemanticNode): boolean {
return node.ax !== undefined && node.ax.ignored !== true;
}
function isRenderedDomFallback(node: ResolvedSemanticNode): boolean {
if (!node.dom) return false;
const attrs = node.dom.attrs;
if (
Object.prototype.hasOwnProperty.call(attrs, "hidden") ||
Object.prototype.hasOwnProperty.call(attrs, "inert") ||
(attrs["aria-hidden"] ?? "").toLowerCase() === "true"
) {
return false;
}
return node.dom.rendered ?? (node.dom.localRect != null || node.dom.rect != null);
}
function initialParentId(
graph: ResolvedSemanticGraph,
node: ResolvedSemanticNode,
): SemanticNodeId | undefined {
let parentId = node.axParentId ?? node.domParentId;
if (node.axParentId && node.domParentId && node.axParentId !== node.domParentId) {
const axParent = graph.nodes.get(node.axParentId);
const axRole = axParent?.vom.role?.toLowerCase() ?? "";
const meaningfulAxParent =
axParent !== undefined &&
!ROOT_ROLES.has(axRole) &&
(explicitStructural(axParent) ||
!TRANSPARENT_ROLES.has(axRole) ||
Boolean(axParent.vom.name || axParent.vom.text || hasMeaningfulRelations(axParent)));
parentId = meaningfulAxParent ? node.axParentId : node.domParentId;
}
if (parentId) return parentId;
if (node.frameId === graph.rootFrameId) return undefined;
return graph.frames.get(node.frameId)?.ownerNodeId;
}
function orderedNodes(graph: ResolvedSemanticGraph): ResolvedSemanticNode[] {
return [...graph.frames.values()]
.sort((a, b) => a.order - b.order)
.flatMap((frame) =>
(graph.nodeIdsByFrame.get(frame.frameId) ?? [])
.map((id) => graph.nodes.get(id))
.filter((node): node is ResolvedSemanticNode => node !== undefined)
.sort((a, b) => a.sourceOrder - b.sourceOrder),
);
}
function recoveredNodes(
graph: ResolvedSemanticGraph,
nodes: ResolvedSemanticNode[],
): Map<SemanticNodeId, VomNode> {
const numericId = new Map(nodes.map((node, index) => [node.id, index + 1]));
const semanticId = new Map([...numericId].map(([id, numeric]) => [numeric, id]));
const provisional = nodes.map((node) => ({
...node.vom,
id: numericId.get(node.id) as number,
parentId: (() => {
const parent = initialParentId(graph, node);
return parent ? (numericId.get(parent) ?? null) : null;
})(),
...(node.backendNodeId !== undefined ? { backendNodeId: node.backendNodeId } : {}),
frameId: node.frameId,
contextScopeId: graph.frames.get(node.frameId)?.contextScopeId ?? node.frameId,
referenceable: node.referenceable,
}));
return new Map(
applyVomInteractionRecovery(provisional).flatMap((node) => {
const id = semanticId.get(node.id);
return id ? [[id, node] as const] : [];
}),
);
}
function redundantSourceNodes(
graph: ResolvedSemanticGraph,
nodes: ResolvedSemanticNode[],
recovered: ReadonlyMap<SemanticNodeId, VomNode>,
): Set<SemanticNodeId> {
const byKey = new Map<string, ResolvedSemanticNode[]>();
for (const node of nodes) {
const output = recovered.get(node.id) as VomNode;
const name = output.name?.trim().toLowerCase() ?? "";
if (!name || !isVomReferenceNode({ ...output, referenceable: true })) continue;
const key = `${node.frameId}\u0000${initialParentId(graph, node) ?? ""}\u0000${output.role?.toLowerCase() ?? ""}\u0000${name}\u0000${output.sensitive === true}`;
const matches = byKey.get(key) ?? [];
matches.push(node);
byKey.set(key, matches);
}
const redundant = new Set<SemanticNodeId>();
for (const matches of byKey.values()) {
const axOnly = matches.filter((node) => node.ax && !node.dom);
const domOnly = matches.filter((node) => node.dom && !node.ax);
if (matches.length === 2 && axOnly.length === 1 && domOnly.length === 1) {
redundant.add(domOnly[0].id);
}
}
return redundant;
}
function baseDecision(
graph: ResolvedSemanticGraph,
node: ResolvedSemanticNode,
output: VomNode,
frameOwnerIds: ReadonlySet<SemanticNodeId>,
redundant: ReadonlySet<SemanticNodeId>,
): StructureDecision {
if (node.excluded) return { kind: "excluded", reason: "excluded" };
if (redundant.has(node.id)) return { kind: "excluded", reason: "redundant-source" };
if (!hasUsableAxSemantics(node) && !isRenderedDomFallback(node)) {
return { kind: "excluded", reason: "non-rendered-dom-fallback" };
}
if (frameOwnerIds.has(node.id)) return { kind: "keep", reason: "frame-owner" };
if (node.frameId !== graph.rootFrameId && ROOT_ROLES.has(output.role?.toLowerCase() ?? "")) {
return { kind: "transparent", reason: "child-document-root" };
}
if (node.ax?.ignored === true && !node.dom) {
return { kind: "transparent", reason: "ax-ignored" };
}
const role = output.role?.toLowerCase() ?? "";
if (!TRANSPARENT_ROLES.has(role)) return { kind: "keep", reason: "semantic" };
if (
output.name ||
output.text ||
output.value ||
output.placeholder ||
hasMeaningfulRelations(node) ||
(output.rect && ["fixed", "sticky"].includes(output.position))
) {
return { kind: "keep", reason: "semantic" };
}
return { kind: "transparent", reason: "wrapper" };
}
function buildChildren(
nodes: ResolvedSemanticNode[],
relation: "axParentId" | "domParentId",
): Map<SemanticNodeId, SemanticNodeId[]> {
const children = new Map<SemanticNodeId, SemanticNodeId[]>();
for (const node of nodes) {
const parentId = node[relation];
if (!parentId) continue;
const siblings = children.get(parentId) ?? [];
siblings.push(node.id);
children.set(parentId, siblings);
}
return children;
}
function explicitDomAncestors(
graph: ResolvedSemanticGraph,
nodes: ResolvedSemanticNode[],
domChildren: ReadonlyMap<SemanticNodeId, SemanticNodeId[]>,
): Set<SemanticNodeId> {
const inside = new Set<SemanticNodeId>();
const roots = nodes.filter((node) => !node.domParentId);
const queue = roots.map((node) => node.id);
for (let index = 0; index < queue.length; index += 1) {
const parentId = queue[index];
const parent = graph.nodes.get(parentId);
const parentIsBoundary =
parent !== undefined &&
explicitStructural(parent) &&
!ROOT_ROLES.has(parent.vom.role?.toLowerCase() ?? "");
for (const childId of domChildren.get(parentId) ?? []) {
if (inside.has(parentId) || parentIsBoundary) inside.add(childId);
queue.push(childId);
}
}
return inside;
}
function reliableGroupCandidates(
graph: ResolvedSemanticGraph,
nodes: ResolvedSemanticNode[],
recovered: ReadonlyMap<SemanticNodeId, VomNode>,
decisions: ReadonlyMap<SemanticNodeId, StructureDecision>,
domChildren: ReadonlyMap<SemanticNodeId, SemanticNodeId[]>,
): Set<SemanticNodeId> {
const insideExplicitStructure = explicitDomAncestors(graph, nodes, domChildren);
const remainingChildren = new Map(
nodes.map((node) => [node.id, domChildren.get(node.id)?.length ?? 0]),
);
const summaries = new Map<SemanticNodeId, { blocked: boolean; interactions: number }>();
const queue = nodes
.filter((node) => (remainingChildren.get(node.id) ?? 0) === 0)
.map((node) => node.id);
for (let index = 0; index < queue.length; index += 1) {
const nodeId = queue[index];
const node = graph.nodes.get(nodeId);
const output = recovered.get(nodeId);
if (!node || !output) continue;
const role = output.role?.toLowerCase() ?? "";
const excluded = decisions.get(nodeId)?.kind === "excluded";
const blocked = excluded || explicitStructural(node) || CONTENT_BOUNDARY_ROLES.has(role);
const summary = excluded
? { blocked: true, interactions: 0 }
: isVomReferenceNode(output)
? { blocked: false, interactions: 1 }
: blocked
? { blocked: true, interactions: 0 }
: (domChildren.get(nodeId) ?? []).reduce(
(combined, childId) => {
const child = summaries.get(childId);
return {
blocked: combined.blocked || !child || child.blocked,
interactions: Math.min(2, combined.interactions + (child?.interactions ?? 0)),
};
},
{ blocked: false, interactions: 0 },
);
summaries.set(nodeId, summary);
const parentId = node.domParentId;
if (!parentId) continue;
const remaining = (remainingChildren.get(parentId) ?? 1) - 1;
remainingChildren.set(parentId, remaining);
if (remaining === 0) queue.push(parentId);
}
const candidates = new Set<SemanticNodeId>();
for (const node of nodes) {
const output = recovered.get(node.id) as VomNode;
const role = output.role?.toLowerCase() ?? "";
if (
!node.dom ||
node.dom.tag.toLowerCase() !== "div" ||
!TRANSPARENT_ROLES.has(role) ||
insideExplicitStructure.has(node.id) ||
decisions.get(node.id)?.kind === "excluded" ||
node.dom.attrs.hidden !== undefined ||
(node.dom.attrs["aria-hidden"] ?? "").toLowerCase() === "true" ||
node.dom.attrs.inert !== undefined
) {
continue;
}
const branches = (domChildren.get(node.id) ?? []).map((id) => summaries.get(id));
const blocked = branches.some((summary) => !summary || summary.blocked);
const contributingBranches = branches.filter(
(summary) => (summary?.interactions ?? 0) > 0,
).length;
const interactions = branches.reduce(
(count, summary) => Math.min(2, count + (summary?.interactions ?? 0)),
0,
);
if (!blocked && contributingBranches >= 2 && interactions >= 2) {
candidates.add(node.id);
}
}
return candidates;
}
function leafCandidates(
nodes: ResolvedSemanticNode[],
candidates: ReadonlySet<SemanticNodeId>,
domChildren: ReadonlyMap<SemanticNodeId, SemanticNodeId[]>,
): Set<SemanticNodeId> {
const parentById = new Map(
nodes.flatMap((node) => (node.domParentId ? ([[node.id, node.domParentId]] as const) : [])),
);
const childCount = new Map(nodes.map((node) => [node.id, domChildren.get(node.id)?.length ?? 0]));
const containsCandidate = new Set<SemanticNodeId>();
const queue = nodes.filter((node) => (childCount.get(node.id) ?? 0) === 0).map((node) => node.id);
const selected = new Set<SemanticNodeId>();
for (let index = 0; index < queue.length; index += 1) {
const nodeId = queue[index];
if (candidates.has(nodeId) && !containsCandidate.has(nodeId)) selected.add(nodeId);
const parentId = parentById.get(nodeId);
if (!parentId) continue;
if (candidates.has(nodeId) || containsCandidate.has(nodeId)) containsCandidate.add(parentId);
const remaining = (childCount.get(parentId) ?? 1) - 1;
childCount.set(parentId, remaining);
if (remaining === 0) queue.push(parentId);
}
return selected;
}
function nearestKept(
graph: ResolvedSemanticGraph,
startId: SemanticNodeId | undefined,
relation: "axParentId" | "domParentId",
kept: ReadonlySet<SemanticNodeId>,
memo: Map<SemanticNodeId, SemanticNodeId | undefined>,
): SemanticNodeId | undefined {
let currentId = startId;
const path: SemanticNodeId[] = [];
const seen = new Set<SemanticNodeId>();
let result: SemanticNodeId | undefined;
while (currentId && !seen.has(currentId)) {
if (kept.has(currentId)) {
result = currentId;
break;
}
if (memo.has(currentId)) {
result = memo.get(currentId);
break;
}
seen.add(currentId);
path.push(currentId);
currentId = graph.nodes.get(currentId)?.[relation];
}
for (const id of path) memo.set(id, result);
return result;
}
function finalParentId(
graph: ResolvedSemanticGraph,
node: ResolvedSemanticNode,
kept: ReadonlySet<SemanticNodeId>,
decisions: ReadonlyMap<SemanticNodeId, StructureDecision>,
axParentMemo: Map<SemanticNodeId, SemanticNodeId | undefined>,
domParentMemo: Map<SemanticNodeId, SemanticNodeId | undefined>,
): SemanticNodeId | undefined {
const axParent = nearestKept(graph, node.axParentId, "axParentId", kept, axParentMemo);
const domParent = nearestKept(graph, node.domParentId, "domParentId", kept, domParentMemo);
const axNode = axParent ? graph.nodes.get(axParent) : undefined;
const domNode = domParent ? graph.nodes.get(domParent) : undefined;
const axIsRoot = ROOT_ROLES.has(axNode?.vom.role?.toLowerCase() ?? "");
const axIsExplicit = axNode ? explicitStructural(axNode) && !axIsRoot : false;
const domIsStructural =
domNode !== undefined &&
(explicitStructural(domNode) || decisions.get(domNode.id)?.kind === "inferred-group");
if (axIsExplicit) return axParent;
if (domIsStructural) return domParent;
if (axParent && !axIsRoot) return axParent;
if (domParent) return domParent;
if (axParent) return axParent;
if (node.frameId === graph.rootFrameId) return undefined;
return graph.frames.get(node.frameId)?.ownerNodeId;
}
function recoveredVom(node: VomNode, inferredGroup: boolean): ResolvedSemanticNode["vom"] {
const {
id: _id,
parentId: _parentId,
backendNodeId: _backendNodeId,
frameId: _frameId,
contextScopeId: _contextScopeId,
referenceable: _referenceable,
...vom
} = node;
if (!inferredGroup) return vom;
const { name: _name, text: _text, nearbyText: _nearbyText, ...groupVom } = vom;
return { ...groupVom, role: "group" };
}
export function normalizeSemanticStructure(graph: ResolvedSemanticGraph): StructuredSemanticGraph {
const nodes = orderedNodes(graph);
const recovered = recoveredNodes(graph, nodes);
const redundant = redundantSourceNodes(graph, nodes, recovered);
const frameOwnerIds = new Set(
[...graph.frames.values()].flatMap((frame) => frame.ownerNodeId ?? []),
);
const decisions = new Map<SemanticNodeId, StructureDecision>();
for (const node of nodes) {
decisions.set(
node.id,
baseDecision(graph, node, recovered.get(node.id) as VomNode, frameOwnerIds, redundant),
);
}
const domChildren = buildChildren(nodes, "domParentId");
const candidates = reliableGroupCandidates(graph, nodes, recovered, decisions, domChildren);
for (const nodeId of leafCandidates(nodes, candidates, domChildren)) {
decisions.set(nodeId, {
kind: "inferred-group",
reason: "independent-interaction-branches",
});
}
const kept = new Set(
nodes
.filter((node) => {
const kind = decisions.get(node.id)?.kind;
return kind === "keep" || kind === "inferred-group";
})
.map((node) => node.id),
);
const axParentMemo = new Map<SemanticNodeId, SemanticNodeId | undefined>();
const domParentMemo = new Map<SemanticNodeId, SemanticNodeId | undefined>();
const structuredNodes = new Map<SemanticNodeId, StructuredSemanticNode>();
for (const node of nodes) {
const structure = decisions.get(node.id) as StructureDecision;
const inferredGroup = structure.kind === "inferred-group";
structuredNodes.set(node.id, {
...node,
vom: recoveredVom(recovered.get(node.id) as VomNode, inferredGroup),
structure,
...(kept.has(node.id)
? {
semanticParentId: finalParentId(
graph,
node,
kept,
decisions,
axParentMemo,
domParentMemo,
),
}
: {}),
});
}
return { ...graph, nodes: structuredNodes };
}
@@ -0,0 +1,88 @@
import type { Viewport, VomNode } from "@browser-skill/vom";
import type { CdpTarget } from "@/browser-driver/frame-graph";
import type { CapturedNode } from "../capture";
export interface SemanticAxNode {
nodeId: string;
frameId?: string;
parentId?: string;
backendDOMNodeId?: number;
ignored?: boolean;
role?: { type: string; value?: string };
name?: { type: string; value?: string };
description?: { value?: string };
value?: { value?: string | number | boolean };
properties?: Array<{ name?: string; value?: { value?: string | number | boolean } }>;
childIds?: string[];
}
export type SemanticNodeId = string;
export interface SemanticFrame {
frameId: string;
parentFrameId?: string;
ownerBackendNodeId?: number;
ownerNodeId?: SemanticNodeId;
contextScopeId: string;
target: CdpTarget;
url?: string;
order: number;
}
export interface SemanticGraphNode {
id: SemanticNodeId;
frameId: string;
backendNodeId?: number;
axNodeId?: string;
ax?: SemanticAxNode;
dom?: CapturedNode;
axParentId?: SemanticNodeId;
domParentId?: SemanticNodeId;
sourceOrder: number;
excluded: boolean;
}
export interface SemanticGraph {
viewport: Viewport;
rootFrameId: string;
frames: Map<string, SemanticFrame>;
nodes: Map<SemanticNodeId, SemanticGraphNode>;
nodeIdsByFrame: Map<string, SemanticNodeId[]>;
nodeByFrameBackend: Map<string, SemanticNodeId>;
}
export interface ResolvedSemanticNode extends SemanticGraphNode {
vom: Omit<VomNode, "id" | "parentId" | "backendNodeId" | "frameId" | "contextScopeId">;
referenceable: boolean;
roleSource: "ax" | "ax-ignored" | "dom-explicit" | "dom-native" | "none";
}
export interface ResolvedSemanticGraph extends Omit<SemanticGraph, "nodes"> {
nodes: Map<SemanticNodeId, ResolvedSemanticNode>;
}
export type StructureDecision =
| { kind: "keep"; reason: "semantic" | "frame-owner" }
| { kind: "inferred-group"; reason: "independent-interaction-branches" }
| { kind: "transparent"; reason: "wrapper" | "child-document-root" | "ax-ignored" }
| {
kind: "excluded";
reason: "excluded" | "redundant-source" | "non-rendered-dom-fallback";
};
export interface StructuredSemanticNode extends ResolvedSemanticNode {
structure: StructureDecision;
semanticParentId?: SemanticNodeId;
}
export interface StructuredSemanticGraph extends Omit<ResolvedSemanticGraph, "nodes"> {
nodes: Map<SemanticNodeId, StructuredSemanticNode>;
}
export function frameBackendKey(frameId: string, backendNodeId: number): string {
return `${frameId}\u0000${backendNodeId}`;
}
export function frameAxKey(frameId: string, axNodeId: string): string {
return `${frameId}\u0000${axNodeId}`;
}
+65
View File
@@ -23,6 +23,7 @@ export type ErrorCode =
export type RpcErrorReason =
| "agent_window_scope"
| "element_not_visible"
| "ref_capability_denied"
| "ref_not_found"
| "selector_not_found"
| "target_not_fillable"
@@ -33,10 +34,22 @@ export type RpcErrorReason =
| "restricted_tab_url"
| "borrow_conflict"
| "screenshot_capture_failed"
| "file_input_probe_failed"
| "file_input_not_activated"
| "set_file_input_failed"
| "download_capture_failed"
| "transfer_outcome_unknown"
| "transfer_timeout"
| "cleanup_failed";
export type TransferEffectState = "none" | "committed" | "unknown";
export type TransferCleanupState = "complete" | "failed";
export interface RpcErrorData {
reason?: RpcErrorReason;
effect_state?: TransferEffectState;
phase?: string;
cleanup_state?: TransferCleanupState;
[key: string]: unknown;
}
@@ -318,9 +331,14 @@ export interface SnapshotResult {
export interface ObserveParams extends SnapshotParams {
debug_surfaces?: boolean;
probe_hover?: boolean;
}
export interface ObserveResult extends SnapshotResult {
hover_probe?: {
performed: boolean;
revealed_content: boolean;
};
debug?: {
surface_probes?: Array<{
trigger_backend_node_id: number;
@@ -500,6 +518,50 @@ export interface SelectResult {
dialogs?: JavaScriptDialogInfo[];
}
export interface UploadFile {
transfer_id: string;
name: string;
staged_path?: string;
}
export interface UploadParams {
session_id: string;
ref?: string;
selector?: string;
tab_id?: number;
files: UploadFile[];
timeout_ms?: number;
}
export interface UploadResult {
tab_id: number;
used_ref?: string;
used_selector?: string;
file_names: string[];
}
export interface DownloadParams {
session_id: string;
ref?: string;
selector?: string;
tab_id?: number;
timeout_ms?: number;
browser_relative_dir?: string;
max_byte_size?: number;
}
export interface DownloadResult {
tab_id: number;
used_ref?: string;
used_selector?: string;
suggested_filename: string;
byte_size: number;
mime?: string;
danger?: string;
browser_path?: string;
transfer_id?: string;
}
// --------------------------------------------------------------------------
// M9 tool payloads — evaluate / wait_for_navigation / wait_ms
// --------------------------------------------------------------------------
@@ -780,6 +842,7 @@ export interface SelectedOptionV3 {
export type StepV3 =
| ({ op: "navigate" } & StepCommonV3 & { to: string; cause: NavigationCause })
| ({ op: "switch_tab" } & StepCommonV3)
| ({ op: "click" } & StepCommonV3 & { target: TargetDescriptorV3 })
| ({ op: "hover" } & StepCommonV3 & { target: TargetDescriptorV3 })
| ({ op: "fill" } & StepCommonV3 & {
@@ -823,6 +886,8 @@ export interface RecordStartParams {
redact_values?: boolean;
/** Omitted means v2; `3` requests a state-linked v3 trace. */
trace_version?: number;
/** Client can decode the v3 `switch_tab` step variant. */
supports_tab_switch_steps?: boolean;
}
export interface RecordStartResult {
@@ -0,0 +1,105 @@
# Rendered Surface browser fixtures
These pages exercise Canvas discovery through a real browser DOMSnapshot and AX tree. They are deliberately framework- and site-independent.
## Serve
From the repository root:
```bash
python3 -m http.server 4173 --bind 127.0.0.1 \
--directory apps/extension/test-fixtures/rendered-surfaces
```
Use an Agent Window large enough to keep the primary matrix visible:
```bash
bsk session start --width 1440 --height 1100
bsk navigate http://127.0.0.1:4173/index.html --session <id>
bsk observe --session <id>
```
Always stop the session after the run.
## Pages
### `index.html`
Expected individual Surface labels:
- `fixture-basic-large`
- `fixture-tiny-8px`
- `fixture-pointer-events-none`
- `fixture-partial-ax`
- `fixture-contained-outer`
- `fixture-contained-inner`
- `fixture-different-parent-a`
- `fixture-different-parent-b`
Expected one grouped Surface:
- `fixture-layered-group`, with `layers=2`
Expected absent:
- `fixture-display-none`
- `fixture-visibility-hidden`
- `fixture-opacity-zero`
- `fixture-parent-opacity-zero`
- `fixture-hidden-attribute`
- `fixture-inert-parent`
- `fixture-aria-hidden-parent`
- `fixture-zero-size`
- `fixture-offscreen-right`
- `fixture-fully-overflow-clipped`
`fixture-partially-overflow-clipped` must remain present because part of it is visible. Its screenshot must contain only the browser's final composited view.
### `unlabeled.html`
This page contains eight Canvas elements but must render seven generic Surface markers because
the two coincident layers form one group.
- Every marker uses the stable fallback label `canvas visual surface`.
- Exactly one marker has `layers=2`.
- The two contained canvases remain separate markers.
- `id`, `class`, `data-testid`, nearby prose, and fallback AX content never become a Surface label.
- Blank `aria-label` and `title` values behave as no label.
- Fallback AX content does not suppress the corresponding Surface.
### `frames.html`
Expected Surface labels:
- `fixture-same-origin-frame`
- `fixture-cross-origin-frame`
- `fixture-nested-frame`
The cross-origin frame uses `http://localhost:4173` while the parent uses `http://127.0.0.1:4173`. With Chromium site isolation this exercises the OOPIF path without an external service.
### `viewport.html`
Run with a `900x700` Agent Window.
- `fixture-partial-viewport` is partly visible and must be represented.
- `fixture-below-fold` is absent initially.
- After scrolling it into view and observing again, `fixture-below-fold` must be represented with a fresh ref.
### `dynamic.html`
Use the normal DOM buttons and re-observe after each action:
- Initial state: `fixture-dynamic` absent.
- After **Insert canvas**: present.
- After **Resize to zero**: absent.
- After **Restore size**: present with a fresh ref.
- After **Remove canvas**: absent.
## Invariants
- `snapshot` must not emit any of these Surface markers.
- `observe` must emit only currently visible Canvas surfaces.
- Existing DOM/AX controls remain readable alongside Surface markers.
- Missing labels never prevent discovery; naming and discovery are independent.
- A Surface ref allows `screenshot --ref` and rejects ordinary click/fill/hover/select.
- No fixture URL, id, label, or layout constant may appear in production discovery code.
@@ -0,0 +1,46 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Rendered Surface dynamic matrix</title>
<style>
html, body { margin: 0; font: 14px system-ui, sans-serif; }
body { padding: 16px; }
.controls { display: flex; gap: 8px; margin-bottom: 16px; }
button { padding: 8px 12px; }
canvas { display: block; width: 260px; height: 120px; background: #e8daef; border: 2px solid #7d3c98; }
</style>
</head>
<body>
<h1>Dynamic Canvas lifecycle</h1>
<div class="controls">
<button id="insert" type="button">Insert canvas</button>
<button id="zero" type="button">Resize to zero</button>
<button id="restore" type="button">Restore size</button>
<button id="remove" type="button">Remove canvas</button>
</div>
<div id="mount"></div>
<script>
const mount = document.querySelector("#mount");
const getCanvas = () => mount.querySelector("canvas");
document.querySelector("#insert").addEventListener("click", () => {
if (getCanvas()) return;
const canvas = document.createElement("canvas");
canvas.width = 260;
canvas.height = 120;
canvas.setAttribute("aria-label", "fixture-dynamic");
mount.append(canvas);
});
document.querySelector("#zero").addEventListener("click", () => {
const canvas = getCanvas();
if (canvas) canvas.style.cssText = "width:0;height:0;border:0";
});
document.querySelector("#restore").addEventListener("click", () => {
const canvas = getCanvas();
if (canvas) canvas.removeAttribute("style");
});
document.querySelector("#remove").addEventListener("click", () => getCanvas()?.remove());
</script>
</body>
</html>
@@ -0,0 +1,28 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<style>
html, body { margin: 0; font: 14px system-ui, sans-serif; }
body { padding: 12px; background: #fdfefe; }
canvas { display: block; width: 220px; height: 90px; background: #d5f5e3; border: 2px solid #1e8449; }
iframe { width: 280px; height: 150px; margin-top: 10px; border: 2px solid #85929e; }
</style>
</head>
<body>
<canvas id="frame-canvas" width="220" height="90"></canvas>
<script>
const params = new URLSearchParams(location.search);
const canvas = document.querySelector("#frame-canvas");
if (params.get("canvas") === "0") canvas.remove();
else canvas.setAttribute("aria-label", params.get("label") || "fixture-frame-default");
if (params.get("nested") === "1") {
const iframe = document.createElement("iframe");
iframe.title = "nested fixture frame";
iframe.src = "frame.html?label=fixture-nested-frame";
document.body.append(iframe);
}
</script>
</body>
</html>
@@ -0,0 +1,22 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Rendered Surface frame matrix</title>
<style>
html, body { margin: 0; font: 14px system-ui, sans-serif; }
body { padding: 16px; }
.frames { display: grid; grid-template-columns: repeat(3, 320px); gap: 16px; }
iframe { width: 300px; height: 300px; border: 2px solid #5d6d7e; }
</style>
</head>
<body>
<h1>Frame matrix</h1>
<div class="frames">
<iframe title="same-origin fixture" src="frame.html?label=fixture-same-origin-frame"></iframe>
<iframe title="cross-origin fixture" src="http://localhost:4173/frame.html?label=fixture-cross-origin-frame"></iframe>
<iframe title="nested fixture" src="frame.html?canvas=0&nested=1"></iframe>
</div>
</body>
</html>
@@ -0,0 +1,118 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Rendered Surface discovery matrix</title>
<style>
* { box-sizing: border-box; }
html, body { margin: 0; min-height: 100%; font: 14px/1.4 system-ui, sans-serif; }
body { padding: 16px; background: #f4f6f8; color: #17202a; }
h1 { margin: 0 0 12px; font-size: 20px; }
.matrix { display: grid; grid-template-columns: repeat(4, minmax(220px, 1fr)); gap: 12px; }
.case { min-height: 150px; padding: 10px; overflow: hidden; border: 1px solid #aeb6bf; border-radius: 6px; background: white; }
.case h2 { margin: 0 0 8px; font-size: 13px; }
canvas { display: block; background: #d6eaf8; border: 2px solid #2874a6; }
.stack { position: relative; width: 180px; height: 90px; }
.stack canvas { position: absolute; inset: 0; width: 180px; height: 90px; }
.stack canvas + canvas { background: rgb(244 208 63 / 45%); border-color: #b7950b; }
.contained { position: relative; width: 190px; height: 100px; }
.contained > canvas:first-child { width: 190px; height: 100px; }
.contained > canvas:last-child { position: absolute; left: 55px; top: 30px; width: 70px; height: 40px; background: #fadbd8; border-color: #c0392b; }
.same-place { position: relative; width: 180px; height: 90px; }
.same-place > div { position: absolute; inset: 0; }
.same-place canvas { width: 180px; height: 90px; }
.same-place > div:last-child canvas { border-color: #7d3c98; background: rgb(210 180 222 / 35%); }
.clip-box { width: 100px; height: 70px; overflow: hidden; border: 2px dashed #566573; }
.partially-clipped { width: 140px; height: 60px; transform: translateX(45px); }
.fully-clipped { width: 80px; height: 50px; transform: translateX(130px); }
.tiny { width: 8px; height: 8px; border-width: 1px; }
</style>
</head>
<body>
<h1>Rendered Surface discovery matrix</h1>
<main class="matrix">
<section class="case">
<h2>Visible named canvas</h2>
<canvas aria-label="fixture-basic-large" width="200" height="90"></canvas>
</section>
<section class="case">
<h2>Tiny canvas remains addressable</h2>
<canvas class="tiny" aria-label="fixture-tiny-8px" width="8" height="8"></canvas>
</section>
<section class="case">
<h2>Visible but pointer-events none</h2>
<canvas aria-label="fixture-pointer-events-none" width="180" height="80" style="pointer-events: none"></canvas>
</section>
<section class="case">
<h2>Partial AX/fallback content</h2>
<canvas aria-label="fixture-partial-ax" width="180" height="80">
<button type="button">Fallback button must not suppress the surface</button>
</canvas>
</section>
<section class="case">
<h2>Two real rendering layers</h2>
<div class="stack">
<canvas aria-label="fixture-layered-group" width="180" height="90"></canvas>
<canvas width="180" height="90"></canvas>
</div>
</section>
<section class="case">
<h2>Contained canvases are independent</h2>
<div class="contained">
<canvas aria-label="fixture-contained-outer" width="190" height="100"></canvas>
<canvas aria-label="fixture-contained-inner" width="70" height="40"></canvas>
</div>
</section>
<section class="case">
<h2>Same geometry, different parents</h2>
<div class="same-place">
<div><canvas aria-label="fixture-different-parent-a" width="180" height="90"></canvas></div>
<div><canvas aria-label="fixture-different-parent-b" width="180" height="90"></canvas></div>
</div>
</section>
<section class="case">
<h2>Partially overflow-clipped</h2>
<div class="clip-box">
<canvas class="partially-clipped" aria-label="fixture-partially-overflow-clipped" width="140" height="60"></canvas>
</div>
</section>
<section class="case">
<h2>Fully overflow-clipped</h2>
<div class="clip-box">
<canvas class="fully-clipped" aria-label="fixture-fully-overflow-clipped" width="80" height="50"></canvas>
</div>
</section>
<section class="case">
<h2>CSS-hidden variants</h2>
<canvas aria-label="fixture-display-none" width="80" height="40" style="display: none"></canvas>
<canvas aria-label="fixture-visibility-hidden" width="80" height="40" style="visibility: hidden"></canvas>
<canvas aria-label="fixture-opacity-zero" width="80" height="40" style="opacity: 0"></canvas>
</section>
<section class="case">
<h2>Hidden ancestor variants</h2>
<div style="opacity: 0"><canvas aria-label="fixture-parent-opacity-zero" width="80" height="40"></canvas></div>
<div hidden><canvas aria-label="fixture-hidden-attribute" width="80" height="40"></canvas></div>
<div inert><canvas aria-label="fixture-inert-parent" width="80" height="40"></canvas></div>
<div aria-hidden="true"><canvas aria-label="fixture-aria-hidden-parent" width="80" height="40"></canvas></div>
</section>
<section class="case">
<h2>Zero-size and offscreen</h2>
<canvas aria-label="fixture-zero-size" width="80" height="40" style="width: 0; height: 0; border: 0"></canvas>
<canvas aria-label="fixture-offscreen-right" width="80" height="40" style="position: fixed; left: 5000px; top: 0"></canvas>
</section>
</main>
</body>
</html>
@@ -0,0 +1,68 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Unlabeled Rendered Surface matrix</title>
<style>
* { box-sizing: border-box; }
html, body { margin: 0; min-height: 100%; font: 14px/1.4 system-ui, sans-serif; }
body { padding: 16px; background: #f4f6f8; color: #17202a; }
h1 { margin: 0 0 12px; font-size: 20px; }
.matrix { display: grid; grid-template-columns: repeat(3, minmax(240px, 1fr)); gap: 12px; }
.case { min-height: 150px; padding: 10px; overflow: hidden; border: 1px solid #aeb6bf; border-radius: 6px; background: white; }
.case h2 { margin: 0 0 8px; font-size: 13px; }
.case p { margin: 0 0 8px; }
canvas { display: block; background: #d6eaf8; border: 2px solid #2874a6; }
.stack { position: relative; width: 180px; height: 90px; }
.stack canvas { position: absolute; inset: 0; width: 180px; height: 90px; }
.stack canvas + canvas { background: rgb(244 208 63 / 45%); border-color: #b7950b; }
.contained { position: relative; width: 190px; height: 100px; }
.contained > canvas:first-child { width: 190px; height: 100px; }
.contained > canvas:last-child { position: absolute; left: 55px; top: 30px; width: 70px; height: 40px; background: #fadbd8; border-color: #c0392b; }
</style>
</head>
<body>
<h1>Unlabeled Rendered Surface matrix</h1>
<main class="matrix">
<section class="case">
<h2>Only engineering attributes</h2>
<canvas id="must-not-be-a-label" class="must-not-be-a-label" data-testid="must-not-be-a-label" width="200" height="90"></canvas>
</section>
<section class="case">
<h2>Blank explicit names</h2>
<canvas aria-label=" " title=" " width="180" height="80"></canvas>
</section>
<section class="case">
<h2>Nearby prose is not a Canvas name</h2>
<p>must-not-be-inferred-as-a-surface-label</p>
<canvas width="180" height="80"></canvas>
</section>
<section class="case">
<h2>Fallback AX content does not name or suppress the Canvas</h2>
<canvas width="180" height="80">
<button type="button">Fallback control remains separate</button>
</canvas>
</section>
<section class="case">
<h2>Unnamed rendering layers still cluster</h2>
<div class="stack">
<canvas width="180" height="90"></canvas>
<canvas width="180" height="90"></canvas>
</div>
</section>
<section class="case">
<h2>Unnamed contained canvases stay independent</h2>
<div class="contained">
<canvas width="190" height="100"></canvas>
<canvas width="70" height="40"></canvas>
</div>
</section>
</main>
</body>
</html>
@@ -0,0 +1,22 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Rendered Surface viewport matrix</title>
<style>
html, body { margin: 0; font: 14px system-ui, sans-serif; }
body { min-height: 1800px; padding: 16px; }
canvas { display: block; background: #fdebd0; border: 2px solid #ca6f1e; }
#partial { position: fixed; left: -70px; top: 120px; width: 180px; height: 100px; }
#below { position: absolute; left: 40px; top: 1350px; width: 240px; height: 120px; }
</style>
</head>
<body>
<h1>Viewport matrix</h1>
<p>The orange canvas on the left is intentionally clipped by the viewport.</p>
<canvas id="partial" aria-label="fixture-partial-viewport" width="180" height="100"></canvas>
<canvas id="below" aria-label="fixture-below-fold" width="240" height="120"></canvas>
</body>
</html>
+1
View File
@@ -30,6 +30,7 @@ export default defineConfig({
permissions: [
"alarms",
"debugger",
"downloads",
"idle",
"notifications",
"tabs",
+1
View File
@@ -44,6 +44,7 @@
"!packages/dsh-plugin-browserskill/src/client/bsk-ui.nomodule.css",
"!packages/dsh-plugin-browserskill/.bsk-ui.raw.css",
"!packages/dsh-plugin-browserskill/.bsk-ui.input.css",
"!evals/browser/results/**",
"!**/target/**",
"!**/.wxt/**",
"!**/.output/**",
+2
View File
@@ -53,6 +53,7 @@ flate2 = { workspace = true }
tar = { workspace = true }
zip = { workspace = true }
sha2 = { workspace = true }
base64 = { workspace = true }
[target.'cfg(unix)'.dependencies]
nix = { workspace = true }
@@ -63,6 +64,7 @@ windows-sys = { version = "0.59", features = [
"Win32_Foundation",
"Win32_System_Threading",
"Win32_Security",
"Win32_Storage_FileSystem",
] }
[dev-dependencies]
+136 -160
View File
@@ -1,160 +1,135 @@
---
name: browser-skill
description: |
Use when the user asks to perform browser automation tasks against their
logged-in browser: visit and read pages, fill forms, scrape data, click
through a flow, regression-test a PR's UI, validate a deployed page.
Requires the bsk CLI installed and the browser-skill extension loaded.
Use when the user asks to automate their logged-in Chromium browser: visit
and read pages, fill forms, scrape data, click through flows, regression-test
a PR's UI, validate a deployed page, or operate a tab they identify. Requires
the bsk CLI and browser extension.
---
# browser-skill
Drive the user's **real Chromium browser** (with their logins and cookies) through the `bsk` CLI. The extension opens an isolated **Agent Window** for automation; the user's normal windows stay protected unless you explicitly borrow a tab.
Drive the user's real Chromium browser through `bsk`. Automation runs in an isolated **Agent
Window** with the user's existing logins and cookies. User-window tabs remain protected unless they
are explicitly borrowed.
## When to use
Do not use this skill for tasks with no browser, for extension installation, or when the user only
wants instructions. Never extract credentials, cookies, tokens, or other secrets from pages.
- Open pages, read titles/text, scrape structured data from sites the user can already access
- Fill forms, click through multi-step flows, smoke-test a UI change
- Understand pages with `bsk snapshot` first; use `bsk get-html` or `bsk screenshot` only when the snapshot is insufficient
- Operate on a specific user tab they point you at (after `bsk tab borrow`)
## Required lifecycle
## When NOT to use
Every browser task owns a bounded session:
- Tasks with **no browser** involved (files, APIs, databases only)
- Installing or configuring the extension (point the user to setup docs instead)
- **Credential harvesting** — never run `bsk evaluate` on banking, SSO, or password-manager pages to extract tokens, cookies, or secrets
- Long-lived control of a user's personal login window — borrow only for the immediate step, then `bsk tab return` or end the session
- Replacing the user's manual browsing when they only wanted an explanation
## Prerequisites
1. `bsk` on `PATH` (Rust CLI from browser-skill)
2. browser-skill **extension** loaded in Chromium and connected (popup shows green)
3. Any `bsk` command auto-starts background services as needed; use `bsk doctor` if anything fails
## Mandatory workflow
Every automation task **must** follow this lifecycle. Do **not** rely on idle timeouts (default session idle is 5 minutes).
```
1. bsk session start → capture the 4-letter session id printed on stdout
2. … every tool command … → always pass --session <id>
3. bsk session stop <id> → REQUIRED when done (even on error paths)
```text
1. bsk session start # retain the printed 4-letter session id
2. bsk ... --session <id> # pass it to every session-scoped command
3. bsk session stop <id> # always run on success and error paths
```
Optional: `bsk session start --browser <instance-id-or-label>` when multiple browsers are connected (`bsk browsers` / error output lists them). Add `--no-focus` to open the Agent Window in the background without stealing focus from the user's current window.
Do not rely on the idle timeout for cleanup. Stop the session as soon as the goal is met unless the
user explicitly asks to keep it open. Stopping also returns borrowed tabs.
Emergency cleanup: `bsk session stop --all` or the Agent Window overlay **Stop all**.
Any `bsk` command auto-starts the background services it needs; never manage the daemon by hand.
When multiple browsers are connected, use `bsk browsers` and start with
`bsk session start --browser <id-or-label>`. Add `--no-focus` to that same start command when the
Agent Window does not need to interrupt the user's current work; it is not a flag on other commands.
Run `bsk doctor` when startup or transport problems persist after one retry.
## Stop when the goal is met
## Work toward one observable goal
Every task is a **bounded goal**, not open-ended browsing. The goal may come from the user's request, a recorded `trace.json`, or both.
- Derive a concrete success condition from the user's request or a supplied trace.
- Take the shortest purposeful path: observe, act, then make at most one observation to confirm an
ambiguous result.
- Once success is visible, do not click, refresh, navigate, switch tabs, or perform extra checks.
- If a human-only step appears or two attempts make no progress, request help instead of
brute-forcing.
1. **Define success first** — one concrete, observable condition derived from the user's words, `purpose`, or the last meaningful step in a trace (e.g. "form submitted", "item added to cart", "playback started").
2. **Take the shortest path** — snapshot → act → at most one check. Do not wander, re-try unrelated actions, or stack exploratory steps.
3. **Stop as soon as success is reached** — run `bsk session stop <id>` immediately unless the user explicitly asked to keep the session open (e.g. "don't close yet", "keep browsing").
4. **No post-success work** — once the goal is met, do not click, refresh, navigate, re-search, switch tabs, or "double-check" that it worked. Further verification is a new task.
5. **When blocked, pause — do not brute-force** — if the page requires human input (login, captcha, OTP, payment confirmation) or an action fails twice with no progress, call `bsk request-help` instead of retrying blindly. See **Ask the human for help** below.
6. **When unsure** — at most one extra `bsk snapshot`. If success looks met, stop. If not, ask the user; do not keep clicking.
With a trace, follow its semantic target information and values in order, but treat its refs as
record-local hints. Stop when its purpose or last meaningful effect is satisfied. A trace guides the
task; it does not expand the user's goal or authorize additional actions.
**With a trace:** replay steps in order using `target` role/name/tag and raw `value`/`selection` fields. After the last step (or when its `effect.navigated_to` / success hint is satisfied), apply rules 3–4 immediately. The trace guides execution; it does not extend control beyond the goal.
## Observe, act, observe
**Without a trace:** the user's request *is* the success condition. Satisfying it ends the task — same stop rules apply.
Use this default loop:
## Core interaction loop
Write operations only affect tabs in the **Agent Window** (or tabs you **borrowed** into it).
```
```text
bsk navigate <url> --session <id>
bsk observe --session <id> → primary semantic VOM view; reveals hover/focus surfaces
bsk snapshot --session <id> → static aria tree fallback when VOM is insufficient
bsk hover @e3 --session <id> → reveal hover-triggered menus before re-observing/clicking
bsk click @e4 --session <id> → or bsk fill, bsk select, bsk press
bsk observe --session <id> → again after navigation / DOM change
bsk observe --session <id>
bsk click|hover|fill|select|press ... --session <id>
bsk observe --session <id> # after navigation or a meaningful DOM change
```
**Refs invalidate after navigation** — always re-snapshot before clicking, filling, or selecting on a new page.
Prefer fresh `@eN` refs over CSS selectors. Navigation invalidates refs; large DOM changes may also
make them stale. Observe again before the next interaction.
Prefer `@eN` refs from the latest snapshot over raw CSS selectors. Use `--ref` / `--selector` when ambiguous (`bsk click --help`).
An observation marks a hover-only surface as `@e1 button "Products" [hover first: Shoes | Bags]`.
The listed items are labels, not usable refs: hover the trigger, observe again, then act on the
revealed item's own ref. Do not click the trigger itself unless the user wants the trigger's action.
`[has-submenu]` and `[expanded]` mark the same kind of trigger without listing what it hides.
When VOM renders `[hover first: …]` on an element, the listed items are not currently clickable refs. Run `bsk hover <that-ref> --session <id>`, then immediately run `bsk snapshot` or `bsk observe` again and click the newly visible menu item ref. Do not click the trigger itself unless the user explicitly wants the trigger action.
`bsk observe` does not hover the page on its own. Reach for `--probe-hover` when a control you have
good reason to expect is absent **and** no marker points at a trigger — that combination is what a
CSS-only hover menu looks like from here. It hovers a bounded set of likely triggers, so it costs a
few seconds and touches the live page; once you know which element hides the menu, `bsk hover <ref>`
is cheaper and more precise.
## Observation priority
When `bsk observe` renders
`@eN surface ... [visual-only; requires=image-understanding; ...]`, the ref identifies rendered
canvas content that is not represented by the text observation, not an interactable DOM control.
Keep using any reliable DOM/AX text and controls that appear alongside it. If you can actually
inspect image output, use `bsk screenshot --ref @eN --session <id>` to obtain the visible crop. If
you lack multimodal or image-reading capability, do not take a screenshot and pretend to know its
contents, and never guess coordinates; tell the user that they need to switch to a model with
image-understanding capability. Do not pass a visual surface ref to click, fill, hover, or select;
those commands intentionally reject screenshot-only refs.
Start with `bsk observe` to understand page structure, text, controls, element refs, and conditional hover/focus surfaces. Use `bsk snapshot` only when you need the stricter static accessibility tree or VOM is insufficient. Only escalate to raw HTML or screenshots when the latest observation cannot answer the question:
Escalate page reading only as needed:
1. `bsk observe` — primary semantic VOM observation; may run bounded perception probes such as hover-surface discovery
2. `bsk snapshot` — strict static accessibility tree fallback
3. `bsk get-html` — when hidden DOM, metadata, or markup details are required
4. `bsk screenshot` — when visual layout, canvas/image content, or styling cannot be inferred from the observation. Use `--ref @eN` (from the latest snapshot/observe) to crop to one element; omit `--ref` for the full visible tab.
1. `bsk observe` for normal semantic understanding, text, controls, and refs.
2. `bsk observe --probe-hover` once when an expected control is missing and no marker points at a
trigger.
3. `bsk snapshot` when a stricter static accessibility tree is more useful.
4. `bsk get-html` for exact markup or hidden metadata that semantic views cannot provide.
5. `bsk screenshot` for layout, styling, canvas, images, or requested visual evidence.
Do **not** call `bsk get-html` or `bsk screenshot` first just to inspect a page.
Do not start with raw HTML or screenshots merely to discover ordinary controls. When interaction is
needed, obtain a fresh observation before acting on screenshot or HTML findings.
## Sandbox rules
## Respect the Agent Window boundary
| Rule | Detail |
|------|--------|
| Agent Window | `bsk tab create`, `bsk navigate`, `bsk click`, etc. work on agent tabs by default |
| User tabs | Read-only until borrowed: `bsk tab list --session <id> --scope user` then `bsk tab borrow <tab-id> --session <id>` |
| Return borrowed tabs | Call `bsk tab return <tab-id> --session <id>` when finished; unreturned tabs are **auto-returned** on `bsk session stop` |
| Writes off-agent | Commands that mutate the page fail if the tab is not in the Agent Window — borrow or create a tab first |
Normal page writes affect only Agent Window tabs. To operate a user tab, first list it with
`bsk tab list --scope user --session <id>`, then `bsk tab borrow <tab-id>`. Return it immediately
after the relevant step with `bsk tab return <tab-id>`; never invent a tab id or keep a personal tab
borrowed across unrelated work.
## Global flags
## Ask the human when needed
| Flag | Purpose |
|------|---------|
| `--json` | Machine-readable JSON on stdout (errors too) |
| `--quiet` | Suppress informational stderr |
| `-v` / `-vv` | More verbose logging |
Use `bsk request-help` for login, captcha, OTP, payment confirmation, consent, or another step the
user must complete. Give a precise prompt and pass fresh `--target` refs/selectors when concrete
controls can be highlighted. Use completion criteria only when the page has a clear stable success
signal.
Auto-update is **on by default** — the background daemon upgrades `bsk` itself when a new release is available (postponed while a session is active). Set `BSK_AUTO_UPDATE=off` to disable it and upgrade manually with `bsk update`.
The result `outcome` is one of `continued`, `completed`, `cancelled`, `timed_out`, or `disabled`
(`navigated` is deprecated — never treat navigation as a completion signal). Resume only after
`continued` or `completed`. Treat `cancelled` as rejection, and `timed_out` or `disabled` as a
blocker rather than a reason to retry. After control returns, run a fresh `bsk observe` before
reasoning about the page or using refs.
Command-specific flags (timeouts, `--tab-id`, `--wait-until`, …): **`bsk <cmd> --help`**
## Command inventory
## CLI command reference (one line each)
This list of names is complete. Never invent a command outside it; read
`bsk <command...> --help` for flags instead of guessing them.
Details and flags: **`bsk <cmd> --help`**
### Diagnostics
| Command | Summary |
|---------|---------|
| `bsk status` | Connection health, connected browsers, active sessions |
| `bsk doctor` | Deep diagnostics and repair hints |
| `bsk browsers` | List connected browser instances (ids, labels, versions) |
### Session
| Command | Summary |
|---------|---------|
| `bsk session start` | Open Agent Window (`--width`/`--height` for initial size); prints **4-letter session id** |
| `bsk session start --no-focus` | Open Agent Window in the background without stealing focus |
| `bsk session stop <id>` | End session, close Agent Window, auto-return borrowed tabs |
| `bsk session stop --all` | Stop every active session |
| `bsk session list` | List active sessions |
### Window (require `--session <id>`)
| Command | Summary |
|---------|---------|
| `bsk window resize` | Resize the Agent Window (`--width`, `--height`; 100..=7680 CSS px) |
### Device emulation — `bsk emulate` (requires `--session <id>`)
Emulate a mobile device environment on the agent tab via CDP — viewport, User-Agent, and touch — to debug a page's mobile layout and behaviour:
```bash
bsk emulate --session <id> --device iphone-14
bsk emulate --session <id> --width 390 --height 844 --dpr 3 --mobile --ua "Mozilla/5.0 (iPhone…" --touch
bsk emulate --session <id> --off
```text
session start|stop|list browsers status doctor update logs
navigate navigate-back navigate-forward reload wait-for-navigation wait-ms
observe snapshot get-html screenshot console network
click hover fill select press evaluate
tab list|create|close|select|borrow|return window resize emulate
request-help record start|stop
```
- Presets (`--device`): `iphone-14`, `iphone-14-pro-max`, `iphone-se`, `pixel-7`, `galaxy-s23`, `ipad-mini`, `galaxy-tab-s8`. Manual flags (`--width`/`--height`/`--dpr`/`--mobile`/`--ua`/`--accept-language`/`--touch`/`--max-touch-points`) also work without a preset, or override individual preset fields; `--no-mobile`/`--no-touch` turn a preset's mobile viewport / touch emulation back off.
- Repeated runs merge field by field onto the tab's current emulation state — only the flags you pass change. E.g. after `--device iphone-14`, `bsk emulate --session <id> --width 390 --height 844` keeps the preset's dpr (`3`) and mobile viewport. (The extension remembers the state per tab; after an extension reload the next run applies only the fields it carries.)
- `--off` clears every override (viewport, UA, touch) and the remembered state, restoring the tab's real environment.
- Scope: overrides apply to **one tab only** (CDP per-target) and are **not inherited by new tabs** — re-run `bsk emulate` after opening or switching to another tab (default target is the session's active tab; `--tab-id` overrides).
- Emulation covers viewport/UA/touch only; it does not throttle the network, fake geolocation, or synthesise real touch-event streams.
Required flags that are easy to get wrong:
### Tabs (require `--session <id>`)
@@ -206,11 +181,34 @@ Both capture from the moment the tab is attached and read a bounded per-tab buff
| `bsk select <ref-or-selector> --value <v>` | Set `<select>` option(s) by `value` (repeat `--value` for multi-select) |
| `bsk press <key>` | Key/combo (`Enter`, `Ctrl+A`, …; optional `--ref` to focus first) |
### File transfer (require `--session`)
| Command | Summary |
|---------|---------|
| `bsk upload <ref-or-selector> --file <path>` | Click one upload trigger and attach an agent-readable local file (`--file` is repeatable) |
| `bsk download <ref-or-selector> --out <path>` | Click one download trigger and copy the single completed file to an exact local path (`--overwrite` is opt-in) |
The agent/harness decides whether a file transfer is appropriate and which local path belongs to the task. Treat upload as disclosure of that file to the current website, and download as accepting website-controlled bytes onto the local filesystem. Use only paths that are necessary for the user's bounded goal.
BrowserSkill enforces the mechanical boundary: files are staged under a session-scoped opaque transfer, only daemon-minted capabilities reach the extension, upload/download still obey Agent Window tab checks, and transfers are chunk/size bounded. Upload intercepts the native chooser for one transaction, locates the input activated in the resolved target's document, and assigns only the staged file paths. Download uniquely correlates one exact-target browser intent with one Chrome download in either event order, routes it through a daemon-minted relative directory, and lets the daemon validate and import it. Upload staging remains available for a later form submission and is removed when the session ends. Downloads cannot overwrite an existing destination unless `--overwrite` is explicit. BrowserSkill does not inspect file content or decide whether its meaning is sensitive.
Do not use `request-help` merely because a native file chooser or browser download is involved; try these commands first. For transfer failures, use the structured error instead of retrying blindly:
- `reason=file_input_not_activated` means the requested click did not activate exactly one `<input type="file">`; the page may use a non-input picker such as `window.showOpenFilePicker()`. Do not retry blindly. If human help is available, call `request-help` and tell the user the exact original local path to choose. The staged daemon path is internal and must not be shown to the user.
- `reason=file_input_probe_failed` means BrowserSkill could not safely establish the browser-side upload transaction. Do not repeat the same action; use `request-help` when available.
- `reason=set_file_input_failed` means BrowserSkill found the activated file input but Chrome rejected the staged path or assignment. Check the extension's file-URL access permission; otherwise use `request-help`.
- `reason=download_capture_failed` means BrowserSkill could not attribute exactly one completed download to the requested target. Do not retry blindly or accept an unrelated browser download; use `request-help` when available.
- `effect_state=none` means BrowserSkill confirmed that no file-transfer effect was committed. Follow the accompanying reason; a corrected target or explicit human fallback may be attempted.
- `effect_state=unknown` means the browser may already have attached or created the file. Do not repeat the transfer. Observe the page if that can establish the result; otherwise stop and report the uncertainty.
- `effect_state=committed` means the browser-side effect occurred even if later completion or cleanup failed. Do not repeat it; continue only after verifying the resulting page/download state.
If `request-help` returns `outcome="disabled"`, do not retry it. Stop gracefully and report the transfer mechanism that requires human intervention.
### Scripting & timing
| Command | Summary |
|---------|---------|
| `bsk evaluate <expression>` | Run JS in agent tab (see red lines); JS throw → stderr, **exit 0** |
| `bsk evaluate <expression>` | Run JS in agent tab (see red lines). JS throw → stderr, **exit 0** (RPC success); use `--json` and check `.ok` to detect JS errors |
| `bsk wait-for-navigation` | Block until load/DOM idle/etc. (`--wait-until`, `--timeout`) |
| `bsk wait-ms <duration>` | Sleep (`500ms`, `2s`, `1m`; **no** `--session`) |
@@ -280,48 +278,26 @@ bsk record start --browser <instance-id-or-label> \
bsk record stop [--output trace] # terminal fallback if the browser panel is unavailable
```
- **v3 bundle (preferred):** `--output` is a directory (default `./trace`) containing `trace.json` and `states/`. `trace.json` lists `states[]` (page observation ids) and `steps[]` (each step binds `state` = observe snapshot *before* the action and `result.state` = snapshot *after* settle). Page bodies live in `states/sN.txt` using the same VOM format as `bsk observe`.
- **v2 fallback:** when the connected extension is older, export may contain only legacy `trace.json` with `pages[]` action context and no `states/` observation files. Update the extension for full v3 bundles.
- Each `states[]` entry is one **settled page observation** (captured after recording start, navigation landing, or action settle — not on a timer).
- `target.ref` values like `@e12` exist **only inside** the bundle for disambiguation; do **not** copy `@eN` refs into SKILL.md or agent runbooks — use visible names from the observation text instead.
- `--purpose` is optional context metadata; it does **not** change what gets captured.
- `--redact-values` masks all form values in page files as `[filled]` / `[empty]`.
- Do **not** record on banking/SSO/password-manager pages; passwords are redacted but traces may still contain sensitive text.
`select` matches an option's `value` attribute, not its visible label. Device preset ids are
lowercase and hyphenated, such as `iphone-14`.
## Error handling
- `console` and `network` provide bounded, read-only debugging evidence.
- `emulate` applies viewport, user-agent, and touch overrides to one tab; new tabs do not inherit
them. Use `--off` to restore the real environment.
- `evaluate` is a last resort when observe plus normal interactions cannot complete the task. With
`--json`, inspect `.ok`: a JavaScript exception may still have CLI exit code 0 because the RPC
succeeded. Never evaluate credential surfaces to read storage, cookies, or auth data.
- `record` captures a user's actions for later replay. Read `bsk record start --help` before use,
and never record banking, SSO, password-manager, or other sensitive pages.
### Exit codes (`echo $?` after `bsk …`)
## Recover without wandering
| Code | Meaning | What to do |
|------|---------|------------|
| `0` | Success (including `evaluate` where JS threw but RPC succeeded) | Continue |
| `1` | User error — bad args, unknown session, tab not in Agent Window, stale ref | Fix args; `bsk session list`; re-snapshot |
| `2` | Protocol / transport — service unreachable, IPC failure | `bsk doctor`; check extension connected; retry the command |
| `3` | Browser / CDP execution failed | Retry; simplify selector; check tab still open |
| `4` | Timeout | Increase `--timeout`; try `--wait-until domcontentloaded` |
| `5` | Version skew (CLI vs extension) | Upgrade/reinstall matching versions |
- Stale ref: observe again and retry the intended action once.
- Unknown tab or session: list current tabs/sessions; never guess identifiers.
- Timeout: inspect current page state before deciding whether one longer purposeful wait is useful.
- Unsupported command: continue with available capabilities; suggest updating only when the missing
command is necessary.
- Unrecoverable failure: report the blocker and stop the session in a finally-style path.
Human errors print `error:` + `hint:` on stderr; `--json` includes `code`, `message`, `hint`, `exit_code`.
### When to run diagnostics
| Situation | Command |
|-----------|---------|
| Before first task in a session | `bsk status` — extension connected? |
| Any failure you cannot fix in one retry | `bsk doctor` |
| Multiple browsers / wrong target | `bsk browsers` then `bsk session start --browser <id>` |
Always **`bsk session stop <id>`** in a `finally`-style path so the Agent Window closes and borrowed tabs return.
## Red lines
1. **No token theft** — do not `bsk evaluate` on sensitive sites to read `localStorage`, cookies, or auth headers for exfiltration.
2. **No long borrow** — do not leave a user's personal tab in the Agent Window across unrelated tasks.
3. **No skip stop** — always `bsk session stop <id>`; never assume idle timeout will clean up.
4. **No post-success control** — once the user’s goal (or last trace step) is met, do not keep operating the page; stop the session unless they asked to keep it open.
5. **No raw observe escalation before snapshot/observe** — use `bsk snapshot` first; use `bsk observe` when VOM semantics or conditional surfaces help. Only use `bsk get-html` or `bsk screenshot` when snapshot/observe is insufficient. Element screenshots (`--ref @eN`) still require a fresh snapshot/observe ref — never skip observation just to grab a visual.
6. **`evaluate` is powerful and risky** — use only when snapshot + click/fill/select cannot suffice; never on credential surfaces.
---
**More detail for any command:** `bsk <cmd> --help`
The CLI's current help and error hints are authoritative for flags, parameters, and recovery
details.
+75
View File
@@ -0,0 +1,75 @@
//! Atomic visibility boundary for files received into a same-directory temp.
use std::path::Path;
pub fn commit(temp: &Path, out: &Path, overwrite: bool) -> std::io::Result<()> {
if !overwrite {
std::fs::hard_link(temp, out)?;
std::fs::remove_file(temp)?;
return Ok(());
}
replace(temp, out)
}
#[cfg(unix)]
fn replace(temp: &Path, out: &Path) -> std::io::Result<()> {
// POSIX rename replaces an existing non-directory destination atomically.
std::fs::rename(temp, out)
}
#[cfg(windows)]
fn replace(temp: &Path, out: &Path) -> std::io::Result<()> {
use std::os::windows::ffi::OsStrExt;
use windows_sys::Win32::Storage::FileSystem::{
MOVEFILE_REPLACE_EXISTING, MOVEFILE_WRITE_THROUGH, MoveFileExW,
};
let from: Vec<u16> = temp.as_os_str().encode_wide().chain(Some(0)).collect();
let to: Vec<u16> = out.as_os_str().encode_wide().chain(Some(0)).collect();
// SAFETY: both buffers are NUL-terminated and remain alive for the call.
let ok = unsafe {
MoveFileExW(
from.as_ptr(),
to.as_ptr(),
MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH,
)
};
if ok == 0 {
Err(std::io::Error::last_os_error())
} else {
Ok(())
}
}
#[cfg(not(any(unix, windows)))]
fn replace(temp: &Path, out: &Path) -> std::io::Result<()> {
std::fs::rename(temp, out)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn no_replace_never_overwrites_existing_output() {
let dir = tempfile::tempdir().unwrap();
let temp = dir.path().join("new.part");
let out = dir.path().join("out.bin");
std::fs::write(&temp, b"new").unwrap();
std::fs::write(&out, b"old").unwrap();
assert!(commit(&temp, &out, false).is_err());
assert_eq!(std::fs::read(&out).unwrap(), b"old");
}
#[test]
fn overwrite_replaces_existing_output_without_predelete() {
let dir = tempfile::tempdir().unwrap();
let temp = dir.path().join("new.part");
let out = dir.path().join("out.bin");
std::fs::write(&temp, b"new").unwrap();
std::fs::write(&out, b"old").unwrap();
commit(&temp, &out, true).unwrap();
assert_eq!(std::fs::read(&out).unwrap(), b"new");
assert!(!temp.exists());
}
}
+18 -11
View File
@@ -351,10 +351,10 @@ fn check_version_compatible(status: Option<&StatusResult>) -> CheckResult {
}
}
/// `bsk doctor` check: every connected browser should use the same
/// protocol version as the daemon. Minor protocol drift is accepted by
/// the daemon but flagged here so the user can see who needs updating
/// (M10.4).
/// `bsk doctor` check: connected browsers should speak a protocol the
/// daemon accepts. A different protocol string is still a live
/// connection — report Ok so agents keep working. The detail is an
/// upgrade reminder, not a blocker.
///
/// Review M2 (round-1 minor): when no browsers are connected, the
/// check has nothing to compare against, so it now reports
@@ -396,13 +396,12 @@ fn check_browsers_protocol_compatible(status: Option<&StatusResult>) -> CheckRes
})
.collect::<Vec<_>>()
.join(", ");
CheckResult::fail(
CheckResult::ok(
name,
format!(
"{} browser(s) have protocol minor drift from the daemon: {stale}",
"{} browser(s) report a different protocol version (still usable — continue, and upgrade soon): {stale}",
status.version_skew_browsers.len()
),
"upgrade the browser-skill extension or bsk CLI so both sides use the same protocol version",
)
}
@@ -581,7 +580,7 @@ mod m2_tests {
}
#[test]
fn browsers_check_reports_fail_when_skew_present() {
fn browsers_check_reports_ok_when_skew_present() {
let status = fake_status(
vec![BrowserStatusEntry {
instance_id: "alpha".into(),
@@ -605,9 +604,17 @@ mod m2_tests {
}],
);
let check = check_browsers_protocol_compatible(Some(&status));
assert_eq!(check.status, CheckStatus::Fail);
assert_eq!(check.status, CheckStatus::Ok);
assert!(check.detail.contains("alpha"));
assert!(check.hint.is_some());
assert!(
check
.detail
.contains("still usable — continue, and upgrade soon")
);
assert!(
check.hint.is_none(),
"a protocol-version note must not hint to stop"
);
}
#[test]
@@ -635,7 +642,7 @@ mod m2_tests {
}],
);
let check = check_browsers_protocol_compatible(Some(&status));
assert_eq!(check.status, CheckStatus::Fail);
assert_eq!(check.status, CheckStatus::Ok);
assert!(
check
.detail
+147
View File
@@ -0,0 +1,147 @@
//! `bsk download` — capture one browser download and atomically commit it.
use std::fs::OpenOptions;
use std::io::Write;
use std::path::{Path, PathBuf};
use std::time::Duration;
use anyhow::Context;
use base64::Engine;
use bsk_protocol::Method;
use bsk_protocol::tools::{
DownloadParams, DownloadResult, TransferChunkParams, TransferChunkResult, TransferIdParams,
TransferReleaseResult,
};
use clap::Args;
use uuid::Uuid;
use crate::cli::atomic_output;
use crate::cli::ensure_daemon::ensure_daemon;
use crate::cli::error::{CliError, Format};
use crate::cli::interaction::split_target;
use crate::cli::navigate::parse_timeout_ms;
#[derive(Debug, Clone, Args)]
pub struct DownloadArgs {
/// Snapshot ref (`@e3`) or CSS selector for the download trigger.
pub target: Option<String>,
#[arg(long = "ref")]
pub ref_: Option<String>,
#[arg(long = "selector")]
pub selector: Option<String>,
#[arg(long)]
pub out: PathBuf,
#[arg(long)]
pub session: String,
#[arg(long = "tab-id")]
pub tab_id: Option<i64>,
#[arg(long, default_value = "2m", value_parser = parse_timeout_ms)]
pub timeout: u32,
#[arg(long)]
pub overwrite: bool,
}
pub fn dispatch(args: DownloadArgs, format: Format) -> Result<(), CliError> {
if args.out.exists() && !args.overwrite {
return Err(CliError::Local(anyhow::anyhow!(
"output already exists (pass --overwrite to replace it): {}",
args.out.display()
)));
}
let info = ensure_daemon().context("ensure daemon is running")?;
let (ref_, selector) = split_target(args.target, args.ref_, args.selector)?;
let params = DownloadParams {
session_id: args.session,
ref_,
selector,
tab_id: args.tab_id,
timeout_ms: Some(args.timeout),
browser_relative_dir: None,
max_byte_size: None,
};
let reply: DownloadResult = crate::cli::business_rpc::call(
info.sock_path.clone(),
"download",
Method::ToolDownload,
Some(params),
ipc_timeout(args.timeout),
)?;
let transfer_id = reply.transfer_id.clone().ok_or_else(|| {
CliError::Local(anyhow::anyhow!("daemon returned no download transfer id"))
})?;
let write_result = write_transfer(&info.sock_path, &transfer_id, &args.out, args.overwrite);
let _: Result<TransferReleaseResult, CliError> = crate::cli::business_rpc::call(
info.sock_path,
"transfer-release",
Method::TransferRelease,
Some(TransferIdParams { transfer_id }),
Duration::from_secs(5),
);
write_result?;
match format {
Format::Json => {
let mut value = serde_json::to_value(&reply).unwrap();
value["path"] = serde_json::json!(args.out.to_string_lossy());
if let Some(obj) = value.as_object_mut() {
obj.remove("transfer_id");
obj.remove("browser_path");
}
println!("{}", serde_json::to_string_pretty(&value).unwrap());
}
Format::Human => println!("{}", args.out.display()),
}
Ok(())
}
fn write_transfer(sock: &Path, id: &str, out: &Path, overwrite: bool) -> Result<(), CliError> {
let parent = out
.parent()
.filter(|p| !p.as_os_str().is_empty())
.unwrap_or(Path::new("."));
let temp = parent.join(format!(".bsk-download-{}.part", Uuid::new_v4().simple()));
let result = (|| {
let mut file = OpenOptions::new()
.write(true)
.create_new(true)
.open(&temp)
.with_context(|| format!("create temporary download {}", temp.display()))
.map_err(CliError::Local)?;
let mut offset = 0u64;
loop {
let chunk: TransferChunkResult = crate::cli::business_rpc::call(
sock.to_path_buf(),
"transfer-read",
Method::TransferRead,
Some(TransferChunkParams {
transfer_id: id.to_string(),
offset,
data_base64: String::new(),
}),
Duration::from_secs(30),
)?;
let encoded = chunk.data_base64.as_deref().unwrap_or("");
let bytes = base64::engine::general_purpose::STANDARD
.decode(encoded)
.map_err(|e| CliError::Local(anyhow::anyhow!("decode download chunk: {e}")))?;
file.write_all(&bytes)
.map_err(|e| CliError::Local(e.into()))?;
offset = chunk.next_offset;
if chunk.eof {
break;
}
}
file.sync_all().map_err(|e| CliError::Local(e.into()))?;
drop(file);
atomic_output::commit(&temp, out, overwrite)
.with_context(|| format!("atomically commit download to {}", out.display()))
.map_err(CliError::Local)
})();
if result.is_err() {
let _ = std::fs::remove_file(&temp);
}
result
}
fn ipc_timeout(timeout_ms: u32) -> Duration {
Duration::from_millis(u64::from(timeout_ms) + 5_000)
}

Some files were not shown because too many files have changed in this diff Show More