Compare commits

...
Author SHA1 Message Date
drakezhang e831d65db3 fix(extension): prevent closed borrowed tabs from blocking session stop 2026-09-07 11:12:44 +08:00
Zhang GH 47ac947f18 Merge pull request #174 from iAstro/fix/extension-locale-normalization
fix(i18n): match browser locale by language subtag instead of exact code
2026-09-04 17:29:31 +08:00
Zhang GH 69d64c4eb1 Merge pull request #176 from Tencent/fix/windows-update-detach
fix(bsk): fix Windows self-update failing to replace binary
2026-09-04 17:11:31 +08:00
Zhang GH 99c997d6d5 Merge pull request #66 from iAstro/fix/issue-57-user-tab-freedom
fix(extension): keep user-opened tabs free & fix agent-tab leak on session stop (issue #57)
2026-09-04 17:10:15 +08:00
iAstro 93853ebc65 refactor(i18n): drive locale normalisation from shipped resource keys
Per review feedback on #174: the hardcoded if/else in
normalizeLanguageCode required touching the function for every new
language, so a bare `ko` would miss the `ko-KR` resource #161 is about
to add and Korean users would land on English despite the fix.

Replace it with createLanguageNormalizer(resourceKeys), a factory that
derives its mapping from the resource keys i18next is configured with:

  1. exact resource key              -> unchanged
  2. one resource for the language   -> that key (ko -> ko-KR)
  3. several (zh-CN / zh-TW)         -> script subtag, then region
  4. nothing shipped                 -> unchanged, fallbackLng applies

getLanguageDetectionOptions now takes the resource keys and i18n.ts
passes Object.keys(resources). Chinese disambiguation stays as the only
special case - one language, two scripts; every other mapping derives
from the index automatically.

Test expectations for the zh-TW family change from "zh-TW" to "zh-CN":
only zh-CN ships today and the factory maps to shipped keys only, so
Traditional resolves directly to Simplified instead of leaking into the
fallback chain. The rendered UI is identical either way, and the new
extensibility cases lock in the automatic disambiguation once a zh-TW
bundle registers - plus the auto-resolution of a newly registered
language like ko-KR.

28 -> 31 tests; extension suite 832/832; tsc, biome and ext:build clean.
2026-09-04 15:42:40 +08:00
drakezhang 248bb90a2d test(extension): return home tab ids from mocks 2026-09-04 14:37:04 +08:00
drakezhang 6e6c178434 Merge remote-tracking branch 'origin/main' into fix/issue-57-explicit-tab-claims 2026-09-04 14:31:48 +08:00
drakezhang 386ef91c24 fix(extension): preserve existing tab scope semantics 2026-09-04 13:14:26 +08:00
drakezhang 0dc11b30ea fix(extension): claim agent tabs explicitly 2026-09-04 11:17:05 +08:00
iAstro 759eebd4b6 fix(i18n): match browser locale by language subtag instead of exact code
The extension ships en-US and zh-CN resources, but only en-US was ever
selected: every other browser language fell through to fallbackLng
("zh-CN"), so users on en, en-GB, en-CN, en-AU, ja-JP, fr-FR saw Chinese.

Match on the BCP 47 language subtag and normalise it to a resource key:

  en*                                -> en-US
  zh-Hans* / zh-CN / zh-SG / zh      -> zh-CN
  zh-Hant* / zh-TW / zh-HK / zh-MO   -> zh-TW
  anything else                      -> unchanged (fallbackLng default: en-US)

Simplified vs Traditional is decided by the script subtag (Hans/Hant) when
present, then by the region subtag, since Chrome reports zh-CN / zh-TW
rather than script subtags. The zh-TW branch is written now so adding a
Traditional translation later needs no code change.

chrome.i18n.getUILanguage() is now preferred over navigator.language, which
is unreliable on the chrome-extension:// origin the popup runs on.

Adds a locale-resolution regression matrix (28 cases).

Closes #168
2026-09-04 10:03:43 +08:00
drakezhangandCursor 618bd6a0df fix(bsk): fix Windows self-update failing to replace binary
Two issues prevented the detached update helper from completing:

1. Only CREATE_NO_WINDOW was set — the child cmd.exe process was still
   attached to the parent console session and got killed when the
   terminal closed. Add CREATE_NEW_PROCESS_GROUP to fully detach it.

2. `timeout /t 1 /nobreak` requires a console and fails silently in a
   no-window context, causing the retry loop to spin with no delay.
   Replace with `ping -n 2 127.0.0.1` which works in any environment.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-03 12:24:53 +08:00
Zhang GH c455a09fc3 Merge pull request #163 from dayi/fix/recording-http-random-uuid
fix(extension): support UUID generation on insecure origins
2026-09-03 10:46:12 +08:00
Dean Wang b6be356e40 test(extension): cover UUID fallback during recording startup 2026-09-03 07:40:57 +08:00
Zhang GH 945bf1523d Merge pull request #173 from Tencent/release/0.2.0
Release/0.2.0
2026-09-02 19:00:21 +08:00
drakezhangandCursor a928a9fb76 test: update popup version display assertions for CLI/Ext labels
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 18:32:33 +08:00
drakezhangandCursor 0b2d7e95cc style: format release script for biome
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 18:26:35 +08:00
drakezhangandCursor 556bea1645 chore(release): 0.2.0
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 18:17:56 +08:00
drakezhangandCursor 0302d5592b chore: add release script, changelog, and version display labels
- scripts/release.mjs: unified version bump + tag + push automation
- CHANGELOG.md: document all changes from 0.1.x through 0.2.0
- popup: show "CLI {ver} / Ext {ver}" instead of bare version numbers

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 18:05:32 +08:00
Hannah Liu eb34ec0dbf Merge pull request #124 from Tencent/feat/expand-upload-download
feat(bsk): support drop to upload
2026-09-02 17:56:55 +08:00
drakezhangandCursor 43067b6bfe style: remove extra blank lines in file-transfer test
Fix biome formatter error on CI.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 17:52:57 +08:00
drakezhangandCursor 6cc0d38489 docs(skill): compress CLI SKILL.md with file transfer guidance
Restore the lean style from PR #167 while integrating essential file
transfer documentation for upload/download. Reduces the skill from
362 lines / 26k chars to 180 lines / 9.2k chars.

A/B eval (24 browser runs, Composer 2.5 Fast, core suite) confirms
no pass-rate regression: candidate 12/12 vs baseline 10/12 (baseline
failures were a network blip and a behavioral variance, not skill
quality).

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 17:43:58 +08:00
drakezhangandCursor 0b71963344 merge: resolve conflicts with updated feat/upload-download base
Keep feat/expand-upload-download (drop mode superset) over the base
branch version for all 12 conflicting files. The drop-to-upload branch
already includes all upload/download functionality plus the new --mode
drop mechanism, overlay suppression, OOPIF coordinate handling, and
expanded error reasons.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 17:03:35 +08:00
Hannah Liu 59e2c3224e Merge pull request #123 from Tencent/feat/upload-download
feat(bsk): add secure, session-scoped file upload and download | 支持安全、会话级的文件上传与下载
2026-09-02 16:47:19 +08:00
drakezhangandCursor d70df753f6 merge: resolve SKILL.md conflicts with main
Keep the expanded command-reference tables from feat/upload-download
(including the new File Transfer section) over main's compact two-line
format.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 16:42:15 +08:00
Zhang GH 66ae555e40 Merge pull request #172 from Tencent/fix/pip-close-and-obs-mediatype
fix(dsh-plugin): PiP window close button & observation thumbnail media type detection
2026-09-02 16:25:33 +08:00
drakezhangandCursor fee7c0983c fix: sort import order to satisfy biome organizeImports
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 16:21:26 +08:00
drakezhangandCursor 51473c01ff fix(dsh-plugin): add PiP close button and sniff observation thumbnail media type
The Document PiP window had no close/return entry once popped out —
users could not dismiss it from inside the window. Add an onClosePip
callback and render a close button in the PiP header; clicking it
calls pipWindow.close() and the content returns to its carrier.

The observation thumbnail publisher hardcoded mediaType as "image/png"
regardless of the actual capture bytes. Reuse the existing
sniffImageMediaType helper (exported from image.ts) so the HTTP
thumbnail endpoint declares the correct Content-Type when Chromium
returns JPEG captures.

Closes #153 (Bug 2 fully; Bug 1 was fixed in #135, this covers the
remaining observation-thumbnail path).

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 16:14:24 +08:00
Zhang GH 49744233f7 Merge pull request #171 from Tencent/fix/borrow-confirmation-ux
fix(borrow): improve confirmation UX — proactive focus and longer tim…
2026-09-02 15:23:12 +08:00
drakezhangandCursor 95571f8b70 fix(borrow): improve confirmation UX — proactive focus and longer timeout
The borrow confirmation overlay was easily missed because the Agent
Window stole focus and the 5-second auto-deny was too aggressive for
users who had context-switched away.

- Focus the user window proactively when a borrow request is issued so
  the overlay is immediately visible even when the Agent Window was
  last focused.
- Raise CONFIRMATION_TIMEOUT_MS from 5 s to 60 s, giving users enough
  time to notice and respond — the OS notification with
  requireInteraction: true now has a matching window of usefulness.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 15:11:40 +08:00
Zhang GH 8344e311fa Merge pull request #169 from Tencent/perf/vom-hover-probe-opt-in
Perf/vom hover probe opt in
2026-09-02 14:27:29 +08:00
drakezhangandCursor 63338cd98e Merge branch 'main' into perf/vom-hover-probe-opt-in
main rewrote SKILL.md into a leaner core (#167) while this branch documented
the opt-in --probe-hover flag against the old structure. Keep main's rewritten
prose and re-apply the probe-hover guidance on top of it: the [has-submenu] /
[expanded] markers, when reaching for --probe-hover is warranted, and its slot
in the page-reading escalation ladder ahead of get-html/screenshot.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-02 12:18:06 +08:00
Zhang GH bd193ceb5d Merge pull request #167 from Tencent/refactor/lean-browser-skill
Refactor/lean browser skill
2026-09-02 12:09:42 +08:00
Zhang GH 5ed828736f Merge pull request #165 from Tencent/perf/vom-algorithmic-optimizations
perf(vom): optimize rendering algorithms
2026-09-02 12:09:22 +08:00
drakezhang ee620acf64 docs(skill): give probe-hover a decidable trigger condition
The previous wording asked the agent to notice when a page "looks like"
it hides content behind hover. That is not something an agent reading a
text tree can judge: hover-only content is invisible by definition, and
the marker that used to point at it came from the probing that is now off
by default.

Replaces it with a condition the agent can actually evaluate — an
expected control is missing and no [has-submenu] marker points at a
trigger — and places it on the escalation ladder ahead of get-html and
screenshot, which is where that dead end is currently reached.

Also documents [has-submenu]/[expanded] as the primary hover signal. It
comes from aria-haspopup in the accessibility tree, costs nothing, and is
unaffected by probing being off, so it covers menus that are marked up
correctly.
2026-09-02 10:59:18 +08:00
drakezhang 05a36075b8 perf(vom): make active hover probing opt-in on observe
Every tool.observe moved the real cursor across the page to look for
hover-only menus and tooltips. Measured against real Chrome, that cost a
median of 2.5s and up to 4.7s per observation — 73% to 99% of total
observe wall clock — and returned 3 hits across 12 pages. A positive
control confirms the probes work; the yield on real pages simply does not
justify putting them on the default path.

observe now probes only when asked, via `probe_hover` (`--probe-hover`).
Agents that know which element hides a menu should keep using
`bsk hover <ref>`, which is both cheaper and precise. A held hover latch
still suppresses probing, since probing would move the cursor off the
element the caller is deliberately holding.

Correctness fixes that apply whether or not probing is requested:

- Hover probing ran between the DOM snapshot and the accessibility tree
  fetch, so a hover that opened a menu left one half of an observation
  describing the page before the change and the other half after it. Both
  chains now run after all capture is complete.
- The overlay bypass toggled once per chain and swallowed restore
  failures, which could leave the agent overlay hidden for the rest of
  the tab's life. It is now reference counted across both chains, and a
  failed restore is reported instead of discarded.
- Observations report whether they hovered the page and whether that
  revealed content the static tree lacks, so callers can judge how far
  the live page may have drifted from the returned snapshot.

Probe budgets were checked at the top of the loop, so a candidate could
start with 1ms remaining and still run a full settle window, overshooting
by up to 634ms. Candidates are now admitted only if they fit. Raising
MAX_HOVER_PROBE_MS to 2600 is part of that fix rather than a relaxation:
2000 was never the real ceiling, and enforcing it literally would have
silently dropped a candidate the positive control depends on.

Tooltip misses are cached too, with a shorter TTL than hits, so repeat
observations of the same page no longer re-probe every unlabelled control
from scratch.
2026-09-02 10:59:18 +08:00
Ljy-0827 1bd13d5c3c fix(bsk): fix upload download racing 2026-09-02 10:56:02 +08:00
drakezhangandCursor 59d6533eff docs(skill): trim agent instructions to a leaner core
The CLI skill had grown to 20.8 KB of command tables and flag catalogs that
`bsk --help` already answers at runtime, so every agent paid for them on every
task. Rewrite both skills around the durable rules an agent cannot rediscover
-- session lifecycle, the observe/act/observe loop, the Agent Window boundary,
and the stop-on-success contract -- and delegate flag details to the CLI's own
help output. The CLI skill drops to 6.9 KB (-67%) and the DSH skill to 5.1 KB.

Keep the parts that help discovery has no answer for: the literal
`[hover first: ...]` observation marker, the `request-help` outcome values, the
complete command-name inventory, and the required flags agents guess wrong
(`fill --value`, `select --value`, `screenshot --out`, `emulate --device`).
Also bind `--no-focus` explicitly to `bsk session start`, since a bare mention
led agents to pass it to `bsk navigate`.

Validated with 36 agent runs of the `core` eval suite against a real browser,
comparing the old and new instructions with everything else held constant.
Pass rate did not regress (the lean version was better on `form-controls`) and
token cost per task came out even. The inventory and flag signatures cut
`--help` round-trips from 1.7 to 0.7 per task and removed the invented
`bsk back` / `bsk forward` calls the first draft produced.

Tighten the DSH size assertion to a range so the budget is bounded from both
sides and accidental truncation fails the test.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-01 18:39:19 +08:00
drakezhangandCursor 536d3a9844 chore: ignore local pnpm store and git worktrees
Both directories are local scratch space that kept showing up as untracked
noise in `git status`.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-01 18:39:06 +08:00
iAstro a77353bce8 fix(extension): classify the home tab by id, not a racy boot flag (issue #57)
SessionManager.classifyNewTab's `initializing` branch relied on a
windowInitializing flag that start() cleared *before* the home tab's
chrome.tabs.onCreated event was dispatched (the event is asynchronous), so
the branch never ran in production and the home tab's classification was
subject to a timing race. Match the home tab by its tab id instead — the id
is already returned by ensureActiveTab and is reliable regardless of event
ordering.

Also refresh the overlay-bridge docstring: overlay.ready now makes the
background *push* the per-tab overlay state (and the Agent Window also
pushes on tab create / control-mode change), rather than a request/reply.
2026-09-01 15:45:36 +08:00
iAstro 91621b2545 fix(extension): release the pending agent-tab slot when tab creation fails
tool.tab_create increments pendingAgentTabCount *before* chrome.tabs.create
so the onCreated listener can tell an agent tab from a user-opened tab. When
the create failed, no onCreated event arrived and the counter leaked, so the
next tab the user opened was misclassified as agent-created — the exact
confusion the counter exists to prevent, a variant of issue #57.

handleTabCreate now releases the slot on failure via
SessionManager.releaseAgentTabPending. An abort (cancelled) is excluded: the
tab did open (and was cleaned up), so its onCreated still consumes the slot.

Added a regression test asserting that after a failed tab_create, a
user-opened tab is still classified as 'user' (not 'agent').
2026-09-01 15:36:44 +08:00
drakezhang 641d934337 perf(vom): optimize rendering algorithms 2026-09-01 14:55:36 +08:00
Ljy-0827 a4fed136e3 fix(extension): keep user-opened tabs free & fix agent-tab leak on session stop (issue #57)
- session_stop: exclude tabs still tracked in agentCreatedTabs from the
  release decision so a tab that failed to close is no longer mistaken for
  a user tab (which previously kept the window open and leaked the tab).
- Distinguish user-created tabs from agent-created tabs in the Agent Window
  via a pending-tab counter consumed by chrome.tabs.onCreated, so tabs the
  user opens themselves (new-tab button / Cmd+T / open in new tab) are never
  taken over by the BrowserSkill control mask.
- background overlay: decide per-tab state BEFORE showing controls
  (overlayStateForTab + pushOverlayStateForTab) so a user tab receives hidden
  on its first overlay.ready ping instead of flashing control then hiding.
- AgentWindowApi.ensureActiveTab now resolves with the activated/created tab
  id instead of void, so SessionManager can record homeTabId. This fixes the
  tsc failure flagged in review (the interface previously returned
  Promise<void> while callers treated the result as number | null).
- Wire the production tabManagement.tabs / tabsQuery deps into
  tool.session_stop, both in the dispatcher and in the disconnect cleanup.
  They were never injected, which left the agent-tab cleanup and the
  window-release path as dead code outside tests (behaviour identical to
  main). Added a dispatcher-level integration test that fails without the
  wiring: a surviving user tab must release the window, not close it.
2026-09-01 11:24:56 +08:00
Dean Wang 3c77ef7b01 fix(extension): support UUID generation on insecure origins 2026-09-01 10:30:40 +08:00
Ljy-0827 3b21f2f0a7 fix(bsk): download bug fix 2026-08-31 17:58:33 +08:00
Ljy-0827 405433ab65 fix(bsk): fix ci 2026-08-31 17:58:33 +08:00
Ljy-0827 58c7f38a05 feat(bsk): harden browser file transfer transactions 2026-08-31 17:58:33 +08:00
Ljy-0827 e0c88295c7 fix(bsk): fix ci 2026-08-31 17:58:33 +08:00
Ljy-0827 dc36e38cb9 fix(bsk): upload function safety 2026-08-31 17:58:33 +08:00
Ljy-0827 d5de9b0154 feat(bsk): fix upload function that cannot bypass layout 2026-08-31 17:58:33 +08:00
Ljy-0827 995061514c feat(bsk): fix different type of upload 2026-08-31 17:58:33 +08:00
Ljy-0827 406bb21b7e feat(bsk): support upload and download 2026-08-31 17:58:33 +08:00
Zhang GH f06a514bd2 Merge pull request #149 from Tencent/fix/protocol-upgrade-reminder
fix:将协议版本差异改为升级提醒
2026-08-31 12:25:29 +08:00
Zhang GH 1e53ee63ae Merge pull request #155 from Tencent/ci/edge-addons-publishing
ci(extension): publish updates to Microsoft Edge Add-ons
2026-08-31 11:24:22 +08:00
Zhang GH 120dc441bb Merge pull request #158 from Tencent/codex/fix-dsh-screenshot-test-name
test(dsh-plugin): use renamed screenshot tool
2026-08-29 16:08:01 +08:00
drakezhang 710b1219f4 test(dsh-plugin): use renamed screenshot tool 2026-08-29 16:05:06 +08:00
Zhang GH 177ecd3ebf Merge pull request #157 from Tencent/refactor/simplify-dsh-commands
Refactor/simplify dsh commands
2026-08-29 15:55:16 +08:00
Zhang GH 0e6b9c4542 Merge pull request #135 from cyhano/fix/screenshot-media-type
fix: use detected media type for screenshots instead of hard-coded image/png
2026-08-29 15:52:43 +08:00
drakezhang 506adbe708 Merge remote-tracking branch 'origin/main' into refactor/simplify-dsh-commands 2026-08-29 15:16:31 +08:00
Zhang GH 984d017b0f Merge pull request #156 from Tencent/fix/unify-dsh-plugin-package-id
fix(dsh-plugin): use the published package name as the Cordis id
2026-08-29 15:12:02 +08:00
drakezhang 5df93143c5 refactor(dsh-plugin): simplify browser commands 2026-08-29 14:38:48 +08:00
Zhang GH 7163f2f9b0 Merge pull request #151 from xiyanjun/docs/evaluate-json-ok
docs(skill): clarify bsk evaluate exit-code vs --json .ok semantics
close #133
2026-08-28 22:06:04 +08:00
drakezhang e9f0584af3 ci(extension): publish updates to Microsoft Edge Add-ons 2026-08-28 21:39:53 +08:00
drakezhangandCursor 2997f495db fix(dsh-plugin): use the published package name as the Cordis id
dsh 0.1.0-rc.8 imports and registers the client bundle by export const name, so a leftover dsh-plugin-browserskill id fails to load the 0.1.1 web UI.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-28 21:28:06 +08:00
xiyanjun f96bd72c3c docs(skill): clarify bsk evaluate exit-code vs --json .ok semantics
The exit code reflects RPC success only, not JS success. Document that programmatic callers must use --json and check the .ok field to detect JS errors. Addresses the discoverability gap raised in issue #133.
2026-08-28 18:02:38 +08:00
Zhang GH 329c68054b Merge pull request #150 from Tencent/codex/browser-eval-harness
test: 增加可扩展的浏览器能力回归测试框架
2026-08-28 16:41:00 +08:00
drakezhang 5cd43553ed docs(eval): clarify agent-neutral scope 2026-08-28 15:36:38 +08:00
drakezhang 9ad33801c7 test: add extensible browser evaluation harness 2026-08-28 12:16:39 +08:00
Zhang GH 7aaa5a2d01 Merge pull request #147 from Tencent/fix/dsh-plugin-session-lifecycle
fix(dsh-plugin): stabilize skill loading and browser sessions
2026-08-28 12:11:08 +08:00
drakezhang 210b21b637 style(dsh-plugin): format observation 2026-08-28 12:06:47 +08:00
Hannah Liu 48f64ec0df Merge pull request #122 from Tencent/feat/vom-functional-refactor
feat(vom): refactor semantic structure generation for grouped observations | 重构语义结构生成,支持分组式 VOM 观察
2026-08-28 11:24:34 +08:00
“haonan”andCursor c48a53269e style: rustfmt doctor protocol-version test asserts
cargo fmt --check failed on two long assert lines in PR CI.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-27 22:50:01 +08:00
“haonan”andCursor 3bbd6ab19f fix: treat protocol version differences as upgrade reminders
Agents were stopping on popup and status copy that looked like a hard incompatibility. Keep the connection usable, reframe exit 5 as an unknown method, and ask users to upgrade instead of aborting.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-27 22:22:30 +08:00
drakezhang 1a70d2200e fix(dsh-plugin): stabilize skill loading and browser sessions
Register bundled browser instructions at agent scope, prioritize foreground commands over thumbnail capture, and unify cancellation and session cleanup.
2026-08-27 19:59:20 +08:00
Zhang GH 554861d6b1 Merge pull request #131 from Tencent/feat/dsh-browser-tools-parity
feat(dsh-plugin): add Browser tool parity and dedicated Harness skill
2026-08-27 16:29:30 +08:00
Ljy-0827 b1b1ae1d48 feat(bsk): update skill to support drop 2026-08-24 16:55:16 +08:00
Ljy-0827 8822bba9a5 feat(bsk): support drop to upload 2026-08-24 16:55:16 +08:00
Ljy-0827 c74c119a82 fix(bsk): download bug fix 2026-08-24 16:26:04 +08:00
cyhanoandDeepSeek Harness 60fbf30e24 fix: use detected media type for screenshots instead of hard-coded image/png
browser_screenshot declared every capture as image/png regardless of the
actual bytes. On Chromium/Edge builds where captureVisibleTab returns JPEG
bytes, the stored attachment reference recorded mediaType "image/png" against
JPEG bytes, so dsh-attachment-local's read-time metadata verification fails
with ATTACHMENT_CORRUPT ("Stored attachment metadata does not match its
reference"), surfacing as code UNKNOWN and triggering endless LLM retries.

- image.ts: sniff the capture's magic bytes (PNG/JPEG/WebP/GIF) and declare
  that real media type to attachments.saveImage, instead of always "image/png".
- tools.ts: assemble the tool result from ref.mediaType (the store's verified
  type) rather than re-declaring "image/png" as const, and relax the output
  schema's const to the accepted media-type enum.
- tests: add a regression case asserting JPEG capture bytes surface as
  image/jpeg.

Co-Authored-By: DeepSeek Harness <noreply@deepseek.com>
2026-08-24 15:31:34 +08:00
Ljy-0827 b801bc1f62 fix(bsk): fix ci 2026-08-24 15:02:52 +08:00
Ljy-0827 bf59753aa3 feat(bsk): harden browser file transfer transactions 2026-08-24 15:02:52 +08:00
Ljy-0827 a86cbf724f fix(bsk): fix ci 2026-08-24 15:00:22 +08:00
Ljy-0827 76d447caa0 fix(bsk): upload function safety 2026-08-24 15:00:22 +08:00
Ljy-0827 14e5bbe4fc feat(bsk): fix upload function that cannot bypass layout 2026-08-24 15:00:22 +08:00
Ljy-0827 bef722107f feat(bsk): fix different type of upload 2026-08-24 14:59:53 +08:00
Ljy-0827 6800f8fdfe feat(bsk): support upload and download 2026-08-24 14:59:11 +08:00
drakezhang a4b6a61d29 docs(dsh-plugin): clarify package description 2026-08-22 21:29:33 +08:00
drakezhang cad0b5cbc9 feat(dsh-plugin): add Browser tool parity 2026-08-22 21:15:53 +08:00
196 changed files with 15431 additions and 1602 deletions
+177
View File
@@ -325,3 +325,180 @@ jobs:
echo "Chrome Web Store submission created successfully:"
jq '{itemId, state, warningInfo}' "$PUBLISH_RESPONSE"
publish-edge-addons:
needs: [resolve, build-extension]
runs-on: ubuntu-latest
environment: edge-addons
permissions:
contents: read
steps:
- name: Download build artifact
uses: actions/download-artifact@v8
with:
name: extension-zip
path: dist
- name: Validate Microsoft Edge Add-ons configuration
shell: bash
env:
API_KEY: ${{ secrets.EDGE_ADDONS_API_KEY }}
CLIENT_ID: ${{ vars.EDGE_ADDONS_CLIENT_ID }}
PRODUCT_ID: ${{ vars.EDGE_ADDONS_PRODUCT_ID }}
run: |
set -euo pipefail
if [ -z "$API_KEY" ]; then
echo "::error::Environment secret EDGE_ADDONS_API_KEY is not configured"
exit 1
fi
if [ -z "$CLIENT_ID" ]; then
echo "::error::Environment variable EDGE_ADDONS_CLIENT_ID is not configured"
exit 1
fi
if [ -z "$PRODUCT_ID" ]; then
echo "::error::Environment variable EDGE_ADDONS_PRODUCT_ID is not configured"
exit 1
fi
if ! [[ "$PRODUCT_ID" =~ ^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$ ]]; then
echo "::error::EDGE_ADDONS_PRODUCT_ID must be the Partner Center product GUID, not the public Edge Add-ons extension ID"
exit 1
fi
ARCHIVE="$(find dist -maxdepth 1 -name '*.zip' -type f -print -quit)"
if [ -z "$ARCHIVE" ]; then
echo "::error::No extension zip found in the downloaded artifact"
exit 1
fi
echo "ARCHIVE=$ARCHIVE" >> "$GITHUB_ENV"
- name: Upload and publish Microsoft Edge Add-ons extension
shell: bash
env:
API_KEY: ${{ secrets.EDGE_ADDONS_API_KEY }}
CLIENT_ID: ${{ vars.EDGE_ADDONS_CLIENT_ID }}
PRODUCT_ID: ${{ vars.EDGE_ADDONS_PRODUCT_ID }}
RELEASE_VERSION: ${{ needs.resolve.outputs.version }}
run: |
set -euo pipefail
API_ROOT="https://api.addons.microsoftedge.microsoft.com/v1"
UPLOAD_HEADERS="$RUNNER_TEMP/edge-upload-headers.txt"
UPLOAD_RESPONSE="$RUNNER_TEMP/edge-upload-response.json"
STATUS_RESPONSE="$RUNNER_TEMP/edge-status-response.json"
PUBLISH_HEADERS="$RUNNER_TEMP/edge-publish-headers.txt"
PUBLISH_RESPONSE="$RUNNER_TEMP/edge-publish-response.json"
operation_id_from_headers() {
local headers="$1"
local location
location="$(awk '
tolower($1) == "location:" {
sub(/\r$/, "")
sub(/^[^:]*:[[:space:]]*/, "")
print
exit
}
' "$headers")"
location="${location%%\?*}"
printf '%s' "${location##*/}"
}
poll_operation() {
local operation_name="$1"
local status_url="$2"
local status="InProgress"
local http_status
for attempt in $(seq 1 30); do
sleep 5
http_status="$(curl --silent --show-error \
--output "$STATUS_RESPONSE" \
--write-out '%{http_code}' \
--header "Authorization: ApiKey $API_KEY" \
--header "X-ClientID: $CLIENT_ID" \
"$status_url")"
if [ "$http_status" -lt 200 ] || [ "$http_status" -ge 300 ]; then
echo "::error::Microsoft Edge Add-ons $operation_name status check failed with HTTP $http_status"
jq . "$STATUS_RESPONSE" || cat "$STATUS_RESPONSE"
exit 1
fi
status="$(jq -r '.status // empty' "$STATUS_RESPONSE")"
echo "$operation_name status check $attempt/30: $status"
case "$status" in
Succeeded)
return 0
;;
InProgress)
;;
Failed)
echo "::error::Microsoft Edge Add-ons $operation_name failed"
jq . "$STATUS_RESPONSE"
exit 1
;;
*)
echo "::error::Microsoft Edge Add-ons returned an unknown $operation_name status: $status"
jq . "$STATUS_RESPONSE"
exit 1
;;
esac
done
echo "::error::Microsoft Edge Add-ons $operation_name did not complete after 30 checks"
jq . "$STATUS_RESPONSE"
exit 1
}
HTTP_STATUS="$(curl --silent --show-error \
--output "$UPLOAD_RESPONSE" \
--dump-header "$UPLOAD_HEADERS" \
--write-out '%{http_code}' \
--request POST \
--header "Authorization: ApiKey $API_KEY" \
--header "X-ClientID: $CLIENT_ID" \
--header "Content-Type: application/zip" \
--data-binary "@$ARCHIVE" \
"$API_ROOT/products/$PRODUCT_ID/submissions/draft/package")"
if [ "$HTTP_STATUS" != "202" ]; then
echo "::error::Microsoft Edge Add-ons upload failed with HTTP $HTTP_STATUS"
jq . "$UPLOAD_RESPONSE" || cat "$UPLOAD_RESPONSE"
exit 1
fi
UPLOAD_OPERATION_ID="$(operation_id_from_headers "$UPLOAD_HEADERS")"
if [ -z "$UPLOAD_OPERATION_ID" ]; then
echo "::error::Microsoft Edge Add-ons upload response has no Location operation ID"
cat "$UPLOAD_HEADERS"
exit 1
fi
poll_operation \
"package upload" \
"$API_ROOT/products/$PRODUCT_ID/submissions/draft/package/operations/$UPLOAD_OPERATION_ID"
HTTP_STATUS="$(curl --silent --show-error \
--output "$PUBLISH_RESPONSE" \
--dump-header "$PUBLISH_HEADERS" \
--write-out '%{http_code}' \
--request POST \
--header "Authorization: ApiKey $API_KEY" \
--header "X-ClientID: $CLIENT_ID" \
--header "Content-Type: application/json" \
--data "{\"notes\":\"Automated BrowserSkill extension v${RELEASE_VERSION} release from GitHub Actions.\"}" \
"$API_ROOT/products/$PRODUCT_ID/submissions")"
if [ "$HTTP_STATUS" != "202" ]; then
echo "::error::Microsoft Edge Add-ons publish failed with HTTP $HTTP_STATUS"
jq . "$PUBLISH_RESPONSE" || cat "$PUBLISH_RESPONSE"
exit 1
fi
PUBLISH_OPERATION_ID="$(operation_id_from_headers "$PUBLISH_HEADERS")"
if [ -z "$PUBLISH_OPERATION_ID" ]; then
echo "::error::Microsoft Edge Add-ons publish response has no Location operation ID"
cat "$PUBLISH_HEADERS"
exit 1
fi
poll_operation \
"submission" \
"$API_ROOT/products/$PRODUCT_ID/submissions/operations/$PUBLISH_OPERATION_ID"
echo "Microsoft Edge Add-ons submission created successfully:"
jq '{status, message, errorCode, errors}' "$STATUS_RESPONSE"
+8
View File
@@ -7,10 +7,18 @@ dist/
.output/
.wxt/
.web-extension-id
.pnpm-store/
# Git worktrees
.worktrees/
# Logs
*.log
# Local browser evaluation artifacts
evals/browser/results/
evals/browser/agents.local.json
# macOS
.DS_Store
+92
View File
@@ -0,0 +1,92 @@
# Changelog
All notable changes to BrowserSkill will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/).
Starting from 0.2.0, CLI / Extension / DSH Plugin share the same version number.
## [Unreleased]
## [0.2.0] - 2026-09-02
### Added
- File transfer: upload and download support across CLI, Extension, and DSH Plugin
- File transfer: drag-and-drop upload (`drop-to-upload`)
- VOM semantic graph, name enrichment, and hover perception modules
- VOM hover probing (opt-in via `observe` parameter)
- DSH Plugin: browser tool parity with CLI commands
- Edge Add-ons automated publishing in CI
- Protocol upgrade reminder when CLI / Extension protocol versions differ
- Browser evaluation harness (`evals/browser/`)
- Unified release script (`scripts/release.mjs`)
### Changed
- **Version scheme**: all three components now share the same semver
- VOM rendering algorithm optimizations
- Leaner SKILL.md agent instructions
- DSH Plugin: simplified browser commands
- Borrow confirmation UX — proactive focus and longer timeout
- PiP window now has a close button
### Fixed
- Screenshot media type detection (was hard-coded to `image/png`)
- DSH Plugin session lifecycle stability
- DSH Plugin Cordis package ID mismatch
- Observation thumbnail media type sniffing
- Upload/download race conditions and layout bypass issues
- VOM repeated name and safety policy issues
---
*Previous releases used independent version numbers per component.*
## CLI 0.1.11 / Extension 0.1.7 / DSH Plugin 0.1.2 — 2026-08-26 ~ 2026-08-29
### Added
- DSH Plugin sidebar integration, session lifecycle, and archive cleanup
- Recorder iframe and OOPIF support
### Changed
- VOM functional refactor
### Fixed
- Recorder safety policy and bug fixes
## CLI 0.1.10 / Extension 0.1.6 — 2026-08-08
### Added
- VOM observation recording and settled-state detection
- Record overlay timer
### Fixed
- Browser keepalive disconnect handling
## CLI 0.1.9 / Extension 0.1.5 — 2026-07-29
### Added
- CLI auto-update mechanism
- Trace v3 protocol and recorder
### Fixed
- MV3 keepalive disconnect
## CLI 0.1.8 / Extension 0.1.4 — 2026-07-22
### Added
- More browser interaction actions
### Fixed
- Windows named-pipe hash-only path issue
## CLI 0.1.7 / Extension 0.1.3 — 2026-07-07
Initial public release pair.
## CLI 0.1.6 — 2026-06-30
### Fixed
- Minor CLI fixes
## CLI 0.1.5 / Extension 0.1.2 — 2026-06-22
First tagged releases.
Generated
+3 -2
View File
@@ -134,9 +134,10 @@ dependencies = [
[[package]]
name = "bsk"
version = "0.1.11"
version = "0.2.0"
dependencies = [
"anyhow",
"base64",
"bsk-protocol",
"clap",
"console",
@@ -168,7 +169,7 @@ dependencies = [
[[package]]
name = "bsk-protocol"
version = "0.1.11"
version = "0.2.0"
dependencies = [
"schemars",
"semver",
+3 -2
View File
@@ -7,7 +7,7 @@ members = [
default-members = ["crates/bsk-cli"]
[workspace.package]
version = "0.1.11"
version = "0.2.0"
edition = "2024"
rust-version = "1.85"
license = "MIT"
@@ -36,13 +36,14 @@ flate2 = "1.1.9"
tar = "0.4.46"
zip = { version = "7.2.0", default-features = false, features = ["deflate"] }
sha2 = "0.11.0"
base64 = "0.22"
tempfile = "3"
nix = { version = "0.29", features = ["signal", "process"] }
rand = "0.8"
uuid = { version = "1", features = ["v4", "serde"] }
# Workspace crates (version required for `cargo publish` of dependents)
bsk-protocol = { version = "0.1.9", path = "crates/bsk-protocol" }
bsk-protocol = { version = "0.2.0", path = "crates/bsk-protocol" }
[profile.release]
strip = "symbols"
+1
View File
@@ -208,6 +208,7 @@ The repository is a Cargo + pnpm workspace:
- `apps/extension` — browser extension
- `packages/ui` and `packages/i18n` — shared extension UI support
- `packages/dsh-plugin-browserskill` — DeepSeek Harness plugin (`@wxg-prc-cpg/browser-skill-dsh-plugin`)
- [`evals/browser`](evals/browser/README.md) — deterministic local pages and agent-neutral browser capability evaluation
## License
+1
View File
@@ -178,6 +178,7 @@ Agent 不直接与浏览器通信。它通过 `bsk` CLI 下发浏览器任务;
- `apps/extension` — 浏览器扩展
- `packages/ui` 和 `packages/i18n` — 扩展 UI 共享支持
- `packages/dsh-plugin-browserskill` — DeepSeek Harness 插件(`@wxg-prc-cpg/browser-skill-dsh-plugin`)
- [`evals/browser`](evals/browser/README.zh-CN.md) — 确定性本地页面与 Agent 无关的浏览器能力测试台
## 许可证
+6 -3
View File
@@ -1,6 +1,6 @@
# BrowserSkill — Privacy Policy
**Last updated:** May 25, 2026
**Last updated:** August 20, 2026
This Privacy Policy describes how the **BrowserSkill** browser extension (the "Extension") handles information when you install and use it. BrowserSkill is published as part of the open-source [BrowserSkill](https://github.com/Tencent/BrowserSkill) project. The source code is publicly auditable.
@@ -14,7 +14,7 @@ BrowserSkill is a local automation bridge that lets AI coding agents (such as Cu
## 2. Single Purpose
The Extension's single purpose is to expose browser automation primitives (navigation, DOM observation, screenshots, clicks, form filling, tab management) to a locally running BrowserSkill daemon over a WebSocket connection on `127.0.0.1`, so that an AI agent invoked by the user can interact with web pages on the user's behalf.
The Extension's single purpose is to expose browser automation primitives (navigation, DOM observation, screenshots, clicks, form filling, task-scoped file transfer, and tab management) to a locally running BrowserSkill daemon over a WebSocket connection on `127.0.0.1`, so that an AI agent invoked by the user can interact with web pages on the user's behalf.
## 3. Data the Extension Accesses
@@ -26,6 +26,7 @@ Depending on the commands the user (via their AI agent) sends to the local daemo
| **User input simulated by the agent** | Mouse clicks, keystrokes, and form values that the AI agent dispatches through the Chrome DevTools Protocol (CDP). | Required to perform automation actions the user has asked the agent to do. |
| **Tab and window metadata** | Tab IDs, URLs, titles, window IDs of the Agent Window and any tabs the user explicitly authorizes. | Required to target automation commands at the correct tab/window. |
| **Local extension storage** | A randomly generated 8-character instance ID and an optional user-supplied label. | Used so the local daemon can recognize this browser instance across reconnects. No personal data is stored. |
| **File transfers requested by the agent** | Local files explicitly supplied to `bsk upload`, and the file created by a single `bsk download` action. | Required to attach a task file to a web page or return a browser-generated download to the invoking local agent. |
| **OS notifications** | Permission to display a system notification when the agent requests to "borrow" one of the user's existing tabs. | Required to obtain explicit, per-tab user consent before the agent touches any pre-existing tab. |
## 4. Data the Extension Does **Not** Collect
@@ -34,7 +35,7 @@ BrowserSkill does **not**:
- Send any data to remote servers, the Extension's authors, or any third party.
- Call any LLM, AI, or cloud API. The Extension contains no API keys, model identifiers, or remote endpoints.
- Read or transmit cookies, browsing history, bookmarks, downloads, saved passwords, or autofill data.
- Read or transmit cookies, browsing history, bookmarks, saved passwords, or autofill data. It observes only the download initiated by an active `bsk download` call, not download history generally.
- Use webcam, microphone, geolocation, or any device sensor.
- Include analytics, telemetry, crash reporting, advertising SDKs, or fingerprinting code.
- Track users across websites or across sessions.
@@ -50,6 +51,7 @@ The Extension requests the following Chrome permissions. Each is used solely for
- **`alarms`** — Periodically wake the service worker to keep the local WebSocket connection alive.
- **`idle`** — Detect when the device returns from idle/locked so the Extension can promptly re-establish the local WebSocket connection after the machine wakes. No idle data is stored or transmitted.
- **`notifications`** — Show a system notification to obtain user approval before the agent borrows a user-owned tab.
- **`downloads`** — Correlate and route the one browser download initiated by an active `bsk download` command. If that claimed transaction fails, BrowserSkill cancels an in-progress file or removes its completed temporary browser file. It is not used to enumerate download history or alter unclaimed downloads.
- **`storage`** — Persist a random instance ID and optional label in `chrome.storage.local`.
- **Host permission `<all_urls>`** — Inject a small status overlay (showing "Agent Active") on pages controlled by the agent, and enable automation across whatever sites the user directs the agent to. The Extension does **not** read or transmit page content from sites the agent is not actively driving.
@@ -61,6 +63,7 @@ All Extension activity stays on the user's local device. The only network traffi
- The instance ID and optional label persist in `chrome.storage.local` until the user uninstalls the Extension or clears extension storage.
- Page content, screenshots, DOM snapshots, and other observed data are returned to the local daemon in response to commands and are **not retained by the Extension**. They live only as long as the agent's tool call.
- Upload and download bytes are staged by the local daemon in a private, session-scoped directory. Download staging is removed after it is copied to the requested destination. Upload staging is retained until the session ends so a later form submission can still read the attached file. Remaining staging is removed when the session ends or disconnects, or when the daemon next starts after a crash.
## 8. User Control
@@ -0,0 +1,81 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>BrowserSkill file-transfer fixture</title>
<style>
body { font: 16px system-ui; max-width: 760px; margin: 48px auto; }
.actions { display: flex; gap: 12px; align-items: center; margin: 24px 0; }
button, a { padding: 10px 16px; border: 1px solid #777; border-radius: 8px; }
a[aria-disabled="true"] { opacity: .45; pointer-events: none; }
#drop { padding: 36px; border: 3px dashed #777; border-radius: 12px; text-align: center; }
#drop.dragging { border-color: #2563eb; background: #eff6ff; }
img { display: block; max-width: 480px; max-height: 360px; margin-top: 20px; }
</style>
</head>
<body>
<h1>Upload then download the same image</h1>
<p id="status">No image selected</p>
<div class="actions">
<button id="upload" type="button">Upload through file input</button>
<input id="file" type="file" accept="image/png" hidden />
<a id="download" aria-disabled="true">Download uploaded image</a>
</div>
<div id="drop" role="button" tabindex="0">Drop an image here</div>
<img id="preview" alt="Uploaded image preview" hidden />
<script>
const input = document.querySelector('#file');
const upload = document.querySelector('#upload');
const drop = document.querySelector('#drop');
const download = document.querySelector('#download');
const preview = document.querySelector('#preview');
const status = document.querySelector('#status');
let objectUrl;
window.__bskTransferFixture = { events: [], result: null };
function accept(file, mechanism) {
if (!file) return;
if (objectUrl) URL.revokeObjectURL(objectUrl);
objectUrl = URL.createObjectURL(file);
preview.src = objectUrl;
preview.hidden = false;
download.href = objectUrl;
download.download = file.name;
download.removeAttribute('aria-disabled');
status.textContent = `Ready via ${mechanism}: ${file.name} (${file.size} bytes)`;
window.__bskTransferFixture.result = {
mechanism,
name: file.name,
size: file.size,
type: file.type,
hasPathProperty: 'path' in file,
};
}
upload.addEventListener('click', () => input.click());
input.addEventListener('change', () => accept(input.files[0], 'input'));
for (const type of ['dragenter', 'dragover']) {
drop.addEventListener(type, event => {
const types = [...event.dataTransfer.types];
window.__bskTransferFixture.events.push({ type, types });
if (!types.includes('Files')) return;
event.preventDefault();
drop.classList.add('dragging');
});
}
drop.addEventListener('dragleave', () => drop.classList.remove('dragging'));
drop.addEventListener('drop', event => {
window.__bskTransferFixture.events.push({
type: 'drop',
types: [...event.dataTransfer.types],
fileCount: event.dataTransfer.files.length,
});
event.preventDefault();
drop.classList.remove('dragging');
accept(event.dataTransfer.files[0], 'drop');
});
</script>
</body>
</html>
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@browser-skill/extension",
"version": "0.1.7",
"version": "0.2.0",
"private": true,
"type": "module",
"scripts": {
@@ -23,7 +23,7 @@ describe("BorrowConfirmationOverlay", () => {
id: "req-1",
isActiveTab: true,
tabTitle: "Example",
timeoutMs: 5000,
timeoutMs: 60_000,
onAllow,
onDeny,
},
@@ -32,7 +32,7 @@ describe("BorrowConfirmationOverlay", () => {
);
expect(screen.getByText("允许借用标签页?")).toBeTruthy();
for (let i = 0; i < 5; i++) {
for (let i = 0; i < 60; i++) {
await act(async () => {
await vi.advanceTimersByTimeAsync(1000);
});
@@ -66,7 +66,7 @@ describe("BorrowConfirmationOverlay", () => {
id: "req-fallback",
isActiveTab: true,
tabTitle: "Example",
timeoutMs: 5000,
timeoutMs: 60_000,
onAllow,
onDeny,
},
@@ -74,7 +74,7 @@ describe("BorrowConfirmationOverlay", () => {
/>,
);
for (let i = 0; i < 5; i++) {
for (let i = 0; i < 60; i++) {
await act(async () => {
await vi.advanceTimersByTimeAsync(1000);
});
@@ -45,10 +45,32 @@ function portHarness() {
describe("RecordFrameAgent", () => {
afterEach(() => {
vi.unstubAllGlobals();
document.documentElement.removeAttribute(RECORD_DOCUMENT_ATTRIBUTE);
document.body.replaceChildren();
});
it("starts when crypto.randomUUID is unavailable", async () => {
const harness = portHarness();
const getRandomValues = vi.fn((bytes: Uint8Array) => {
bytes.fill(0x11);
return bytes;
});
vi.stubGlobal("crypto", { getRandomValues });
vi.stubGlobal("chrome", {
runtime: {
connect: vi.fn(() => harness.port),
sendMessage: vi.fn(),
},
});
const agent = new RecordFrameAgent();
await expect(
agent.start({ type: RECORD_FRAME_START, requestId: "rec-http", startedAtMs: 10 }),
).resolves.toEqual({ ok: true });
expect(getRandomValues).toHaveBeenCalled();
});
it("keeps a failed stop retryable and flushes the final dirty fill", async () => {
const harness = portHarness();
const sendMessage = vi
@@ -1,3 +1,4 @@
import { createRandomUuid } from "@/lib/random-uuid";
import {
isAcceptedRecordStepAck,
RECORD_STEP,
@@ -18,7 +19,7 @@ export class RecordStepDelivery {
constructor(
requestId: string,
send: SendRecordStepMessage = (message) => chrome.runtime.sendMessage(message),
producerId: string = crypto.randomUUID(),
producerId: string = createRandomUuid(),
) {
this.#requestId = requestId;
this.#producerId = producerId;
@@ -1,3 +1,4 @@
import { createRandomUuid } from "@/lib/random-uuid";
import {
markRecordingDocument,
type RecordingDocumentMarker,
@@ -67,7 +68,7 @@ export class RecordFrameAgent {
async #start(
message: RecordFrameStartMessage,
): Promise<{ ok: true } | { ok: false; error: string }> {
const producerId = crypto.randomUUID();
const producerId = createRandomUuid();
const root = await waitForDocumentElement();
if (this.#disposed) return { ok: false, error: "recording document is disposed" };
const marker = markRecordingDocument(producerId, root);
+56 -7
View File
@@ -25,7 +25,7 @@ import { recordFrameCoordinator } from "@/lib/recording/frame-coordinator";
import { attachSessionsLiveFlag } from "@/lib/sessions-live-flag";
import { createDisconnectCleanup } from "@/session-manager/disconnect-cleanup";
import { attachSessionEventHandler } from "@/session-manager/event-handler";
import { SessionManager } from "@/session-manager/manager";
import { isAgentControlledTab, SessionManager } from "@/session-manager/manager";
import {
attachBorrowNotificationButtonHandler,
attachBorrowNotificationClickHandler,
@@ -41,6 +41,8 @@ import {
attachRecordStepListener,
type RecordRuntimeDeps,
} from "@/tools/record";
import { chromeTabsApi } from "@/tools/shared";
import { chromeTabMutationApi } from "@/tools/tabs";
import { detectBrowserMeta } from "@/transport/handshake";
import type { Transport } from "@/transport/transport";
import { WSTransport } from "@/transport/ws-transport";
@@ -80,6 +82,24 @@ export default defineBackground(() => {
};
}
/**
* Authoritative overlay state for a specific tab. Agent Window tabs are
* free by default; only tabs explicitly claimed through session startup,
* `tab_create`, or `tab_borrow` receive the control overlay.
*/
function overlayStateForTab(tabId?: number, windowId?: number): OverlayAgentStateMessage {
if (typeof tabId === "number" && typeof windowId === "number") {
const ctx = sessions.findByWindowId(windowId);
if (ctx && isAgentControlledTab(ctx, tabId)) return overlayStateForWindow(windowId);
}
return {
type: OVERLAY_AGENT_STATE,
sessionId: null,
mode: "hidden",
generation: overlayGeneration,
};
}
async function pushOverlayStateToTab(
tabId: number,
state: OverlayAgentStateMessage,
@@ -91,13 +111,18 @@ export default defineBackground(() => {
}
}
async function pushOverlayStateForTab(tabId: number, windowId?: number): Promise<void> {
const state = overlayStateForTab(tabId, windowId);
await pushOverlayStateToTab(tabId, state);
}
async function pushOverlayStateForWindow(windowId: number): Promise<void> {
const state = overlayStateForWindow(windowId);
const tabs = await chrome.tabs.query({ windowId });
await Promise.all(
tabs.map((tab) =>
typeof tab.id === "number" ? pushOverlayStateToTab(tab.id, state) : Promise.resolve(),
),
tabs.map((tab) => {
if (typeof tab.id !== "number") return Promise.resolve();
return pushOverlayStateForTab(tab.id, windowId);
}),
);
}
@@ -136,13 +161,30 @@ export default defineBackground(() => {
if (typeof tab.windowId !== "number") return;
pushOverlayStateForAgentWindow(tab.windowId);
});
// Newly observed tabs are free unless the agent's own creation path has
// already claimed their concrete id. This listener only renders that
// state; it never infers or mutates ownership from event ordering.
chrome.tabs.onCreated.addListener((tab) => {
if (typeof tab.windowId !== "number" || typeof tab.id !== "number") return;
if (!sessions.findByWindowId(tab.windowId)) return;
void pushOverlayStateForTab(tab.id, tab.windowId);
});
chrome.tabs.onRemoved.addListener((tabId, removeInfo) => {
sessions.forgetClosedTab(tabId, { isWindowClosing: removeInfo.isWindowClosing });
});
// Re-sync the storage.session flag on SW startup so a previous SW's
// stale `true` does not keep waking us on every page load until the
// first mutation (review M4/M5 round 3 m-R3-1).
void sessionsLive.refresh();
const cleanupAfterDisconnect = createDisconnectCleanup({
manager: sessions,
sessionStopDeps: { cdp },
// Same contract as the dispatcher's `tool.session_stop`: without these
// deps the agent-tab cleanup and the window-release path are dead code.
sessionStopDeps: {
cdp,
tabManagement: { tabs: chromeTabMutationApi },
tabsQuery: chromeTabsApi,
},
onSessionsChanged: () => {
void sessionsLive.syncFromManager();
},
@@ -176,6 +218,9 @@ export default defineBackground(() => {
recording: recordDeps,
onSessionsChanged: onOverlaySessionStateChanged,
onBrowserControlResumed,
onAgentTabClaimed: (tabId, windowId) => {
void pushOverlayStateForTab(tabId, windowId);
},
approveBorrow: (ctx) =>
requestBorrowConfirmation(ctx.tabId, {
...(ctx.signal !== undefined ? { signal: ctx.signal } : {}),
@@ -309,7 +354,11 @@ export default defineBackground(() => {
}
if (msg.kind === OVERLAY_MSG_READY) {
sendResponse(overlayStateForWindow(sender.tab?.windowId));
// Resolve from explicit tab ownership before sending so free tabs never
// flash the control mask.
if (typeof sender.tab?.id === "number") {
void pushOverlayStateForTab(sender.tab.id, sender.tab.windowId);
}
return false;
}
@@ -122,8 +122,8 @@ describe("App", () => {
expect(screen.queryByText(/^扩展 v/)).toBeNull();
expect(screen.queryByText(/^daemon v/)).toBeNull();
expect(screen.getByTitle("扩展版本").textContent).toBe(EXTENSION_VERSION);
expect(screen.getByTitle("bsk 版本").textContent).toBe(mockDaemonVersion);
expect(screen.getByTitle("扩展版本").textContent).toBe(`Ext ${EXTENSION_VERSION}`);
expect(screen.getByTitle("bsk 版本").textContent).toBe(`CLI ${mockDaemonVersion}`);
expect(screen.getByText("03c3e47f")).toBeTruthy();
const copyButton = screen.getByRole("button", { name: "复制实例 ID" });
@@ -247,6 +247,41 @@ describe("App", () => {
expect(copyButton.getAttribute("disabled")).not.toBeNull();
expect(screen.getByText("连接后可用")).toBeTruthy();
});
it("treats protocol drift as still connected and prompts an upgrade", async () => {
mockUseConnectionState.mockReturnValue({
snapshot: {
...baseSnapshot,
state: "version_skew",
instanceId: "03c3e47f",
handshake: {
server: "bh",
version: mockDaemonVersion,
protocol_version: "1.0",
},
},
statusState: "version_skew",
setLabel,
setConnectionEnabled,
});
render(<App />);
expect(screen.getByText("已连接")).toBeTruthy();
expect(screen.getByText("可升级")).toBeTruthy();
expect(screen.queryByText("协议不一致")).toBeNull();
expect(screen.queryByText("Action needed")).toBeNull();
expect(screen.queryByText("兼容")).toBeNull();
const warning = screen.getByText(/协议版本不同,请及时升级/);
expect(warning.textContent).toContain("CLI 协议");
expect(warning.textContent).toContain("扩展协议");
openRecordView();
const copyButton = screen.getByRole("button", { name: "复制录制指令" });
expect(copyButton.getAttribute("disabled")).toBeNull();
fireEvent.click(copyButton);
expect(navigator.clipboard.writeText).toHaveBeenCalledTimes(1);
});
});
describe("control hints toggle", () => {
+6 -5
View File
@@ -79,6 +79,7 @@ export function App() {
}, [copiedTick]);
const isSkewed = statusState === "version_skew";
const connectionLive = statusState === "connected" || isSkewed;
const daemonVersion = snapshot.handshake?.version ?? "—";
const daemonProtocol = snapshot.handshake?.protocol_version ?? "—";
const extensionVersion = snapshot.extensionVersion || "—";
@@ -90,7 +91,7 @@ export function App() {
setCopiedInstanceId(true);
};
const recordReady = statusState === "connected" && Boolean(snapshot.instanceId);
const recordReady = connectionLive && Boolean(snapshot.instanceId);
const recordPurpose = purposeDraft.trim();
const recordStartUrl = startUrlDraft.trim();
const recordCommand = snapshot.instanceId
@@ -206,12 +207,12 @@ export function App() {
</div>
{isSkewed && (
<p
className="mt-2 text-xs leading-snug text-amber-600 dark:text-amber-400"
className="mt-2 text-xs leading-snug text-muted-foreground"
data-slot="popup-version-skew-warning"
>
{t("popup.versionSkewWarning", {
extensionProtocol: PROTOCOL_VERSION,
daemonProtocol,
cliProtocol: daemonProtocol,
})}
</p>
)}
@@ -266,9 +267,9 @@ export function App() {
data-slot="popup-meta"
>
<div className="flex shrink-0 items-center gap-1">
<span title={t("popup.extensionVersionHint")}>{extensionVersion}</span>
<span title={t("popup.daemonVersionHint")}>CLI {daemonVersion}</span>
<span aria-hidden>/</span>
<span title={t("popup.daemonVersionHint")}>{daemonVersion}</span>
<span title={t("popup.extensionVersionHint")}>Ext {extensionVersion}</span>
</div>
<div className="flex min-w-0 items-center justify-end gap-1">
<span className="shrink-0">{t("popup.instanceTitle")}</span>
@@ -2,7 +2,7 @@ import { describe, expect, it, vi } from "vitest";
import {
CAPTURE_SUPPRESS,
type CaptureSuppressMessage,
withOverlaysHiddenForCapture,
withExtensionOverlayHidden,
} from "../capture-suppress-bridge";
function recordingSendToTab(events: string[]) {
@@ -12,12 +12,12 @@ function recordingSendToTab(events: string[]) {
});
}
describe("withOverlaysHiddenForCapture", () => {
describe("withExtensionOverlayHidden", () => {
it("sends begin → fn → end and returns fn's result", async () => {
const events: string[] = [];
const sendToTab = recordingSendToTab(events);
const result = await withOverlaysHiddenForCapture(
const result = await withExtensionOverlayHidden(
7,
async () => {
events.push("capture");
@@ -37,7 +37,7 @@ describe("withOverlaysHiddenForCapture", () => {
const sendToTab = recordingSendToTab(events);
await expect(
withOverlaysHiddenForCapture(
withExtensionOverlayHidden(
7,
async () => {
events.push("capture");
@@ -55,7 +55,7 @@ describe("withOverlaysHiddenForCapture", () => {
});
const fn = vi.fn(async () => "shot");
const result = await withOverlaysHiddenForCapture(7, fn, sendToTab);
const result = await withExtensionOverlayHidden(7, fn, sendToTab);
expect(result).toBe("shot");
expect(fn).toHaveBeenCalledTimes(1);
@@ -69,7 +69,7 @@ describe("withOverlaysHiddenForCapture", () => {
return { type: CAPTURE_SUPPRESS, ok: true };
});
const result = await withOverlaysHiddenForCapture(7, async () => "shot", sendToTab);
const result = await withExtensionOverlayHidden(7, async () => "shot", sendToTab);
expect(result).toBe("shot");
expect(sendToTab).toHaveBeenCalledTimes(2);
@@ -0,0 +1,30 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { createRandomUuid } from "../random-uuid";
describe("createRandomUuid", () => {
afterEach(() => {
vi.unstubAllGlobals();
});
it("uses crypto.randomUUID when it is available", () => {
const randomUUID = vi.fn(() => "00000000-0000-4000-8000-000000000000");
vi.stubGlobal("crypto", { randomUUID, getRandomValues: vi.fn() });
expect(createRandomUuid()).toBe("00000000-0000-4000-8000-000000000000");
expect(randomUUID).toHaveBeenCalledOnce();
});
it("falls back to getRandomValues and sets UUID v4 bits", () => {
const bytes = new Uint8Array(16).fill(0xff);
const getRandomValues = vi.fn((target: Uint8Array) => {
target.set(bytes);
return target;
});
vi.stubGlobal("crypto", { getRandomValues });
const uuid = createRandomUuid();
expect(uuid).toBe("ffffffff-ffff-4fff-bfff-ffffffffffff");
expect(getRandomValues).toHaveBeenCalledOnce();
});
});
@@ -1,9 +1,9 @@
/**
* Wire protocol and background-side helper for hiding the extension's
* in-page overlay while a screenshot is captured, so captured frames only
* contain the page itself.
* Wire protocol and background-side helper for temporarily excluding the
* extension's in-page overlay. Screenshots use it to avoid captured chrome;
* coordinate-driven native input uses it so Chrome hit-tests the page.
*
* The background wraps every capture in `withOverlaysHiddenForCapture`,
* The background wraps each bounded operation in `withExtensionOverlayHidden`,
* which sends `begin` (the content script hides the overlay host and only
* acks once the compositor has produced an overlay-free frame) and always
* follows up with `end`, even when the capture throws. Tabs without the
@@ -46,7 +46,7 @@ const defaultSendToTab: CaptureSuppressSendToTab = (tabId, message) =>
* (or any other `begin` failure) simply skips suppression — there is no
* overlay to hide in that tab.
*/
export async function withOverlaysHiddenForCapture<T>(
export async function withExtensionOverlayHidden<T>(
tabId: number,
fn: () => Promise<T>,
sendToTab: CaptureSuppressSendToTab = defaultSendToTab,
+4 -2
View File
@@ -3,8 +3,10 @@
* between the content-script control overlay and the background SW.
*
* Content script → background:
* - `{ kind: "overlay.ready" }` → background replies with the authoritative
* overlay state for the sender's window.
* - `{ kind: "overlay.ready" }` → background decides the per-tab state first
* (see overlayStateForTab) and pushes the authoritative overlay state for
* that tab. It also proactively pushes on tab create and control-mode
* change, so a tab usually receives state before it ever sends ready.
* - `{ kind: "overlay.interrupt", sessionId }` → background asks the
* daemon (via a `session.user_interrupt` WS event) to cancel
* every inflight + queued tool call for that session with
+26
View File
@@ -0,0 +1,26 @@
/**
* Generate a UUID v4 in browser contexts where randomUUID() is unavailable.
*
* Crypto.randomUUID() is restricted to secure contexts, while
* crypto.getRandomValues() is available to content scripts on ordinary HTTP
* origins. BrowserSkill records many intranet pages served over HTTP, so the
* fallback is required for recording to work on those pages.
*/
export function createRandomUuid(): string {
const cryptoApi = globalThis.crypto;
if (typeof cryptoApi.randomUUID === "function") return cryptoApi.randomUUID();
const bytes = new Uint8Array(16);
cryptoApi.getRandomValues(bytes);
bytes[6] = (bytes[6]! & 0x0f) | 0x40;
bytes[8] = (bytes[8]! & 0x3f) | 0x80;
const hex = Array.from(bytes, (byte) => byte.toString(16).padStart(2, "0")).join("");
return [
hex.slice(0, 8),
hex.slice(8, 12),
hex.slice(12, 16),
hex.slice(16, 20),
hex.slice(20),
].join("-");
}
@@ -1,3 +1,4 @@
import { createRandomUuid } from "../random-uuid";
import {
isRecordFrameQueryMessage,
RECORD_FRAME_PORT,
@@ -188,7 +189,7 @@ export class RecordFrameCoordinator {
const recording = this.#recordings.get(requestId);
if (!recording) return true;
recording.finishing = true;
const commandId = crypto.randomUUID();
const commandId = createRandomUuid();
const results = await Promise.all(
[...recording.agents.values()].map((agent) => this.#stopAgent(agent, requestId, commandId)),
);
@@ -9,7 +9,7 @@ describe("disconnect session cleanup", () => {
const manager = new SessionManager({
agentWindow: {
create: vi.fn(async () => nextWindowId++),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
remove,
},
});
@@ -41,7 +41,7 @@ describe("disconnect session cleanup", () => {
const manager = new SessionManager({
agentWindow: {
create: vi.fn(async () => 100),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
remove: vi.fn(() => removeGate),
},
});
@@ -39,7 +39,7 @@ describe("attachSessionEventHandler", () => {
agentWindow: {
create: vi.fn(async () => 4242),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
await manager.start("aa11");
@@ -73,7 +73,7 @@ describe("attachSessionEventHandler", () => {
agentWindow: {
create: vi.fn(async () => 4242),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
const ctx = await manager.start("aa11");
@@ -87,6 +87,7 @@ describe("attachSessionEventHandler", () => {
windowEvents: events.api,
});
manager.forgetClosedTab(7, { isWindowClosing: true });
events.emit(4242);
for (let i = 0; i < 4; i += 1) await Promise.resolve();
@@ -113,7 +114,7 @@ describe("attachSessionEventHandler", () => {
agentWindow: {
create: vi.fn(async () => 1),
remove: vi.fn(),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
const transport = fakeTransport();
@@ -129,7 +130,7 @@ describe("attachSessionEventHandler", () => {
agentWindow: {
create: vi.fn(async () => 1),
remove: vi.fn(),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
const transport = fakeTransport();
@@ -1,6 +1,6 @@
import { describe, expect, it, vi } from "vitest";
import type { AgentWindowApi, AgentWindowCreateOptions } from "../agent-window";
import { SessionManager } from "../manager";
import { isAgentControlledTab, SessionManager } from "../manager";
function fakeAgentWindow(): AgentWindowApi & {
createMock: ReturnType<typeof vi.fn>;
@@ -13,7 +13,7 @@ function fakeAgentWindow(): AgentWindowApi & {
return id;
});
const removeMock = vi.fn(async (_id: number) => {});
const ensureActiveTabMock = vi.fn(async (_windowId: number, _url: string) => {});
const ensureActiveTabMock = vi.fn(async (_windowId: number, _url: string) => 0);
return {
create: createMock,
remove: removeMock,
@@ -151,6 +151,76 @@ describe("SessionManager", () => {
expect(sm.list()).toEqual([]);
});
describe("explicit tab claims", () => {
it("claims the exact home tab returned by AgentWindowApi", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow() });
const ctx = await sm.start("aa11");
expect(ctx.agentCreatedTabs).toEqual(new Set([0]));
expect(isAgentControlledTab(ctx, 0)).toBe(true);
});
it("keeps every other tab free without classification state", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow() });
const ctx = await sm.start("aa11");
expect(isAgentControlledTab(ctx, 99)).toBe(false);
expect(ctx).not.toHaveProperty("userTabs");
expect(ctx).not.toHaveProperty("pendingAgentTabCount");
});
it("treats a committed borrow as an explicit claim", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow() });
const ctx = await sm.start("aa11");
const reservation = sm.tryReserveBorrow(42, ctx.sessionId);
if ("borrowedBy" in reservation) throw new Error("unexpected borrow conflict");
reservation.commit({ tabId: 42, originalWindowId: 7, originalIndex: 3 });
expect(isAgentControlledTab(ctx, 42)).toBe(true);
});
it("forgets an agent-created tab after Chrome removes it", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow() });
const ctx = await sm.start("aa11");
ctx.agentCreatedTabs.add(42);
sm.forgetClosedTab(42);
expect(ctx.agentCreatedTabs.has(42)).toBe(false);
expect(isAgentControlledTab(ctx, 0)).toBe(true);
});
it("forgets closed borrows idempotently without affecting other tabs or sessions", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow() });
const a = await sm.start("aa11");
const b = await sm.start("bb22");
a.borrowedTabs.set(42, { tabId: 42, originalWindowId: 7, originalIndex: 0 });
a.borrowedTabs.set(43, { tabId: 43, originalWindowId: 7, originalIndex: 1 });
b.borrowedTabs.set(44, { tabId: 44, originalWindowId: 8, originalIndex: 0 });
sm.forgetClosedTab(42);
sm.forgetClosedTab(42);
sm.forgetClosedTab(999);
expect(isAgentControlledTab(a, 42)).toBe(false);
expect(sm.findBorrowingSession(42, null)).toBeNull();
expect([...a.borrowedTabs.keys()]).toEqual([43]);
expect([...b.borrowedTabs.keys()]).toEqual([44]);
expect(sm.list()).toHaveLength(2);
});
it("prevents an in-flight borrow from reclaiming a closed tab", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow() });
const ctx = await sm.start("aa11");
const reservation = sm.tryReserveBorrow(42, ctx.sessionId);
if ("borrowedBy" in reservation) throw new Error("unexpected borrow conflict");
sm.forgetClosedTab(42);
expect(() =>
reservation.commit({ tabId: 42, originalWindowId: 7, originalIndex: 0 }),
).toThrow(/reservation disappeared/);
reservation.release();
expect(sm.findBorrowingSession(42, null)).toBeNull();
expect(ctx.borrowedTabs.has(42)).toBe(false);
});
});
describe("findBorrowingSession", () => {
it("returns null when no session has borrowed the tab", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow() });
@@ -14,8 +14,11 @@ export interface AgentWindowApi {
* Guarantee the Agent Window has an active, CDP-navigable tab.
* `chrome://` pages (including the New Tab page) reject `Page.navigate`,
* so sessions bootstrap with `about:blank` instead.
*
* Resolves with the id of the activated (or newly created) tab, so callers
* can track the session's home tab without re-querying Chrome.
*/
ensureActiveTab(windowId: number, url: string): Promise<void>;
ensureActiveTab(windowId: number, url: string): Promise<number>;
}
/** Creation hints for a new Agent Window. */
@@ -49,15 +52,19 @@ export const chromeAgentWindowApi: AgentWindowApi = {
// cancellation success while the Agent Window remains open.
await chrome.windows.remove(windowId);
},
async ensureActiveTab(windowId: number, url: string): Promise<void> {
async ensureActiveTab(windowId: number, url: string): Promise<number> {
const tabs = await chrome.tabs.query({ windowId });
const first = tabs.find((t) => typeof t.id === "number");
if (first?.id) {
if (first?.id !== undefined) {
if (!first.active) {
await chrome.tabs.update(first.id, { active: true });
}
return;
return first.id;
}
await chrome.tabs.create({ windowId, url, active: true });
const created = await chrome.tabs.create({ windowId, url, active: true });
if (typeof created?.id !== "number") {
throw new Error("[bh] chrome.tabs.create returned no tab id");
}
return created.id;
},
};
+29 -1
View File
@@ -6,9 +6,20 @@ export interface SessionContext {
agentWindowId: number;
refStore: RefStore;
borrowedTabs: Map<number, BorrowedTab>;
/**
* Tabs explicitly claimed by the agent because it created them. This
* includes the Agent Window's home tab and tabs created by `tool.tab_create`.
* Tabs opened through Chrome UI never enter this set.
*/
agentCreatedTabs: Set<number>;
createdAtMs: number;
}
/** Whether this session has explicitly claimed control of `tabId`. */
export function isAgentControlledTab(ctx: SessionContext, tabId: number): boolean {
return ctx.agentCreatedTabs.has(tabId) || ctx.borrowedTabs.has(tabId);
}
export interface BorrowedTab {
tabId: number;
originalWindowId: number;
@@ -106,6 +117,19 @@ export class SessionManager {
return Array.from(this.sessions.values());
}
/**
* Forget a tab Chrome has removed, including any uncommitted borrow.
* Whole-window closures keep committed borrows until the window-removed
* handler reports which user tabs could not be returned.
*/
forgetClosedTab(tabId: number, { isWindowClosing = false } = {}): void {
this.borrowReservations.delete(tabId);
for (const ctx of this.sessions.values()) {
ctx.agentCreatedTabs.delete(tabId);
if (!isWindowClosing) ctx.borrowedTabs.delete(tabId);
}
}
/**
* Look up whether `tabId` is currently borrowed by some *other*
* session than the one calling. Used by M8 `tab_borrow` to refuse
@@ -179,7 +203,7 @@ export class SessionManager {
const { signal: _signal, ...createOptions } = opts;
windowId = await this.agentWindow.create(AGENT_WINDOW_HOME, createOptions);
throwIfSessionStartAborted(opts.signal);
await this.agentWindow.ensureActiveTab(windowId, AGENT_WINDOW_HOME);
const homeTabId = await this.agentWindow.ensureActiveTab(windowId, AGENT_WINDOW_HOME);
throwIfSessionStartAborted(opts.signal);
const ctx: SessionContext = {
@@ -187,6 +211,10 @@ export class SessionManager {
agentWindowId: windowId,
refStore: new RefStore(),
borrowedTabs: new Map(),
// The home tab is the session's first explicit claim. Every other
// tab remains free until `tab_create` or `tab_borrow` identifies it
// by its concrete Chrome tab id.
agentCreatedTabs: new Set([homeTabId]),
createdAtMs: this.now(),
};
this.sessions.set(sessionId, ctx);
@@ -127,6 +127,38 @@ describe("requestBorrowConfirmation", () => {
expect(message.requestId.length).toBeGreaterThan(0);
});
it("proactively focuses the user window before sending the overlay message", async () => {
tabs.get.mockResolvedValueOnce({ id: 42, title: "Tab" });
windows.getLastFocused.mockResolvedValueOnce(
userWindowWithActiveTab({ windowId: 77, tabId: 7, url: "https://app.example/" }),
);
tabs.sendMessage.mockResolvedValueOnce({ type: "borrow-response", allowed: true });
const pending = requestBorrowConfirmation(42, {
deps: { tabs, windows, notifications },
});
await vi.runAllTimersAsync();
await pending;
expect(windows.update).toHaveBeenCalledWith(77, { focused: true });
});
it("still proceeds when proactive window focus fails", async () => {
tabs.get.mockResolvedValueOnce({ id: 42, title: "Tab" });
windows.getLastFocused.mockResolvedValueOnce(
userWindowWithActiveTab({ windowId: 77, tabId: 7, url: "https://app.example/" }),
);
windows.update.mockRejectedValueOnce(new Error("window gone"));
tabs.sendMessage.mockResolvedValueOnce({ type: "borrow-response", allowed: true });
const pending = requestBorrowConfirmation(42, {
deps: { tabs, windows, notifications },
});
await vi.runAllTimersAsync();
expect(await pending).toBe(true);
});
it("sends confirmation to the borrow target when it is the active tab", async () => {
tabs.get.mockResolvedValueOnce({ id: 42, title: "Active Tab" });
windows.getLastFocused.mockResolvedValueOnce(
@@ -13,7 +13,7 @@ function fakeAgentWindow(ids: number[]) {
return id;
}),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
};
}
@@ -55,7 +55,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(async () => 4242),
remove: vi.fn(),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
await sessions.start("aa11");
@@ -117,7 +117,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(async () => 4242),
remove: vi.fn(),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
const dispatcher = new ToolDispatcher({ transport, sessions });
@@ -140,7 +140,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create,
remove: vi.fn(),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
const dispatcher = new ToolDispatcher({ transport, sessions });
@@ -158,7 +158,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(async () => 4242),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
await sessions.start("aa11");
@@ -172,6 +172,47 @@ describe("ToolDispatcher", () => {
expect(sent[0]).toEqual({ id: "r-1", result: {} });
});
it("wires the production tab APIs into tool.session_stop so a surviving user tab releases the window", async () => {
// Regression guard for the deps that session_stop reads directly: when
// `tabManagement.tabs` / `tabsQuery` are not injected by the dispatcher,
// the agent-tab cleanup and the release decision silently no-op and the
// Agent Window is closed even though a user tab is still open (issue #57).
const removeTab = vi.fn(async () => {});
const closeWindow = vi.fn(async () => {});
// After cleanup the window still holds one tab (id 99) that is neither
// the home tab nor an agent-created tab — i.e. a genuine user tab.
vi.stubGlobal("chrome", {
tabs: {
remove: removeTab,
query: vi.fn(async () => [{ id: 99, windowId: 4242, active: true }]),
},
});
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
agentWindow: {
create: vi.fn(async () => 4242),
remove: closeWindow,
ensureActiveTab: vi.fn(async () => 1),
},
});
await sessions.start("aa11");
// Register the exact tab id returned by the agent's creation path.
sessions.get("aa11")?.agentCreatedTabs.add(7);
const dispatcher = new ToolDispatcher({ transport, sessions });
dispatcher.start();
deliver(makeRequest("tool.session_stop", { session_id: "aa11" }));
await flushMicrotasks();
// The agent tab is closed through the real chrome.tabs surface...
expect(removeTab).toHaveBeenCalledWith(7);
// ...and the window is released rather than closed, so the user's tab survives.
expect(closeWindow).not.toHaveBeenCalled();
expect(sessions.has("aa11")).toBe(false);
expect(sent[0]).toEqual({ id: "r-1", result: { window_released: true } });
});
it("routes tool.console through the CDP console buffer", async () => {
vi.stubGlobal("chrome", {
tabs: {
@@ -183,7 +224,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(async () => 4242),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
await sessions.start("aa11");
@@ -232,13 +273,114 @@ describe("ToolDispatcher", () => {
});
});
it("bypasses and restores the control overlay for an upload trigger click", async () => {
const sendMessage = vi.fn(async () => undefined);
vi.stubGlobal("chrome", {
tabs: {
get: vi.fn(async () => ({ id: 7, windowId: 4242, active: true })),
query: vi.fn(async () => [{ id: 7, windowId: 4242, active: true }]),
sendMessage,
},
});
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
agentWindow: {
create: vi.fn(async () => 4242),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
const ctx = await sessions.start("aa11");
ctx.refStore.set("e1", 123, { tabId: 7 });
const send = vi.fn(async <T>(_tabId: number, method: string, params?: object) => {
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } } as T;
}
if (method === "DOM.getContentQuads") {
return { quads: [[0, 0, 20, 0, 20, 20, 0, 20]] } as T;
}
if (method === "DOM.resolveNode") {
return { object: { objectId: "trigger-object" } } as T;
}
if (method === "DOM.describeNode") return { node: { backendNodeId: 456 } } as T;
if (method === "Runtime.callFunctionOn") {
const declaration = (params as { functionDeclaration?: string }).functionDeclaration ?? "";
if (declaration.includes("count: state.inputs.length")) {
return { result: { value: { count: 1, multiple: false } } } as T;
}
if (declaration.includes("inputs[0]")) {
return { result: { objectId: "input-object" } } as T;
}
return { result: { value: true } } as T;
}
if (method === "Runtime.evaluate") {
const expression = (params as { expression?: string }).expression ?? "";
if (expression.includes("overlayDetails")) {
return { result: { value: { hitIndex: 0 } } } as T;
}
if (expression.includes("overlayHostPresent")) {
return {
result: {
value: { overlayHostPresent: true, overlayHostConnected: true },
},
} as T;
}
if (expression.includes("count:")) {
return { result: { value: { count: 1, multiple: false } } } as T;
}
if (expression.includes("?.inputs[0]")) {
return { result: { objectId: "input-object" } } as T;
}
return { result: { value: true } } as T;
}
return {} as T;
});
const cdp = {
send,
detachSession: vi.fn(async () => {}),
ensureNetworkCapture: vi.fn(async () => {}),
networkEntriesSince: vi.fn(() => ({
tab_id: 7,
entries: [],
next_since: 0,
truncated: false,
})),
setDeviceMetricsOverride: vi.fn(async () => {}),
clearDeviceMetricsOverride: vi.fn(async () => {}),
setUserAgentOverride: vi.fn(async () => {}),
setTouchEmulationEnabled: vi.fn(async () => {}),
};
const dispatcher = new ToolDispatcher({ transport, sessions, cdp: cdp as TestDispatcherCdp });
dispatcher.start();
deliver(
makeRequest("tool.upload", {
session_id: "aa11",
ref: "@e1",
files: [{ transfer_id: "tr_1", name: "test.png", staged_path: "/stage/test.png" }],
}),
);
await flushMicrotasks();
await vi.waitFor(() => expect(sent).toHaveLength(1));
expect(sent[0]).toMatchObject({ result: { tab_id: 7, file_names: ["test.png"] } });
expect(sendMessage).toHaveBeenNthCalledWith(1, 7, {
type: "bh-automation-bypass",
enabled: true,
});
expect(sendMessage).toHaveBeenNthCalledWith(2, 7, {
type: "bh-automation-bypass",
enabled: false,
});
});
it("detaches CDP state before stopping a session", async () => {
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
agentWindow: {
create: vi.fn(async () => 4242),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
await sessions.start("aa11");
@@ -274,7 +416,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(async () => 4242),
remove,
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
await sessions.start("aa11");
@@ -322,7 +464,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(async () => 1),
remove: vi.fn(),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
const dispatcher = new ToolDispatcher({ transport, sessions });
@@ -342,7 +484,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(async () => 1),
remove: vi.fn(),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
const dispatcher = new ToolDispatcher({ transport, sessions });
@@ -366,7 +508,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(async () => 1),
remove: vi.fn(),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
const dispatcher = new ToolDispatcher({ transport, sessions });
@@ -383,7 +525,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(async () => 1),
remove: vi.fn(),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
const dispatcher = new ToolDispatcher({ transport, sessions });
@@ -400,7 +542,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(async () => 1),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
const onSessionsChanged = vi.fn();
@@ -417,16 +559,32 @@ describe("ToolDispatcher", () => {
});
it("invokes onBrowserControlResumed for browser-control tools but not passive reads", async () => {
vi.stubGlobal("chrome", {
tabs: {
create: vi.fn(async (props: chrome.tabs.CreateProperties) => ({
id: 7,
windowId: props.windowId,
url: props.url,
active: props.active,
})),
},
});
const { transport, sent, deliver } = fakeTransport();
const sessions = new SessionManager({
agentWindow: {
create: vi.fn(async () => 1),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
const onBrowserControlResumed = vi.fn();
const dispatcher = new ToolDispatcher({ transport, sessions, onBrowserControlResumed });
const onAgentTabClaimed = vi.fn();
const dispatcher = new ToolDispatcher({
transport,
sessions,
onBrowserControlResumed,
onAgentTabClaimed,
});
dispatcher.start();
deliver(makeRequest("tool.session_start", { session_id: "aa11" }));
@@ -443,6 +601,7 @@ describe("ToolDispatcher", () => {
});
await flushMicrotasks();
expect(onBrowserControlResumed).toHaveBeenCalledWith("aa11");
expect(onAgentTabClaimed).toHaveBeenCalledWith(7, 1);
});
it("reasserts remembered hover before follow-up work and releases only after actions", async () => {
@@ -456,7 +615,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(async () => 1),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
const order: string[] = [];
@@ -538,7 +697,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(async () => 1),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
const cdp = {
@@ -599,7 +758,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(async () => 1),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
const dispatcher = new ToolDispatcher({ transport, sessions });
@@ -636,7 +795,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(async () => 4242),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
await sessions.start("aa11");
@@ -688,7 +847,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(async () => 1),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
const dispatcher = new ToolDispatcher({ transport, sessions });
@@ -722,7 +881,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(async () => 4242),
remove: vi.fn(),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
transport.send = () => {
@@ -744,7 +903,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(async () => 5555),
remove,
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
transport.send = () => {
@@ -772,7 +931,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(() => createPromise),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
const dispatcher = new ToolDispatcher({ transport, sessions });
@@ -823,7 +982,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(async () => 1),
remove: vi.fn(),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
const dispatcher = new ToolDispatcher({ transport, sessions });
@@ -844,7 +1003,7 @@ describe("ToolDispatcher", () => {
agentWindow: {
create: vi.fn(() => createPromise),
remove: vi.fn(),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
const dispatcher = new ToolDispatcher({ transport, sessions });
@@ -20,7 +20,7 @@ function fakeAgentWindow(ids: number[]) {
return id;
}),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
};
}
@@ -12,7 +12,7 @@ function fakeAgentWindow(ids: number[]) {
return id;
}),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
};
}
@@ -0,0 +1,988 @@
import { describe, expect, it, vi } from "vitest";
import { SessionManager } from "@/session-manager/manager";
import { type DownloadsApi, handleDownload } from "../download";
import { captureBrowserDownload } from "../download-capture";
import { uploadThroughFileDrop } from "../file-drop-transaction";
import { uploadThroughActivatedFileInput } from "../file-input-transaction";
import type { ResolvedActionTarget } from "../interaction";
import type { CdpRunner } from "../shared";
import { handleUpload } from "../upload";
function sessions() {
return new SessionManager({
agentWindow: {
create: vi.fn(async () => 100),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
},
});
}
function tabsApi() {
return {
get: vi.fn(
async (tabId: number) => ({ id: tabId, windowId: 100, active: true }) as chrome.tabs.Tab,
),
query: vi.fn(async () => [{ id: 4, windowId: 100, active: true } as chrome.tabs.Tab]),
};
}
function fakeEvent<T extends (...args: never[]) => unknown>() {
const listeners = new Set<T>();
return {
addListener: (listener: T) => listeners.add(listener),
removeListener: (listener: T) => listeners.delete(listener),
emit: (...args: Parameters<T>) => {
for (const listener of listeners) listener(...args);
},
};
}
function actionTarget(frameId?: string, sessionId?: string): ResolvedActionTarget {
return {
tab: { tabId: 4, windowId: 100, active: true },
backendNodeId: 123,
cdpTarget: { tabId: 4, ...(sessionId ? { sessionId } : {}) },
...(frameId ? { frameId } : {}),
usedRef: "e3",
};
}
function withoutOverlay<T>(operation: () => Promise<T>): Promise<T> {
return operation();
}
function uploadCdp(
options: {
inputCount?: number;
multiple?: boolean;
chooser?: { frameId?: string; backendNodeId?: number; mode?: string };
pendingResolve?: boolean;
} = {},
) {
const calls: Array<{ method: string; params?: object }> = [];
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const send = vi.fn(async (_tabId: number, method: string, params?: object) => {
calls.push({ method, params });
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "Page.setInterceptFileChooserDialog") return {};
if (method === "Page.getLayoutMetrics")
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
if (method === "DOM.getContentQuads") return { quads: [[0, 0, 20, 0, 20, 20, 0, 20]] };
if (method === "DOM.resolveNode") {
if (options.pendingResolve) return new Promise<never>(() => {});
return { object: { objectId: "trigger-object" } };
}
if (method === "Runtime.callFunctionOn") {
const declaration = (params as { functionDeclaration?: string }).functionDeclaration ?? "";
if (declaration.includes("Object.defineProperty")) return { result: { value: true } };
if (declaration.includes("count: state.inputs.length")) {
return {
result: {
value: {
count: options.inputCount ?? 1,
multiple: options.multiple ?? true,
},
},
};
}
if (declaration.includes("inputs[0]")) {
return { result: { objectId: "input-object" } };
}
return { result: { value: true } };
}
if (method === "DOM.describeNode") return { node: { backendNodeId: 456 } };
if (
method === "Input.dispatchMouseEvent" &&
(params as { type?: string }).type === "mousePressed" &&
options.chooser
) {
cdpEvent?.({ tabId: 4 }, "Page.fileChooserOpened", options.chooser);
}
return {};
});
const cdp: CdpRunner = {
send: send as unknown as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
return {
calls,
emitChooser: (event: { frameId?: string; backendNodeId?: number; mode?: string }) =>
cdpEvent?.({ tabId: 4 }, "Page.fileChooserOpened", event),
cdp,
};
}
function dropCdp(
options: {
targetContainsPoint?: boolean;
dispatchError?: (type: string) => Error | Promise<never> | undefined;
onCall?: (method: string) => void;
} = {},
) {
const calls: Array<{ method: string; params?: object }> = [];
const detach = vi.fn(async () => {});
const send = vi.fn(async (_tabId: number, method: string, params?: object) => {
calls.push({ method, params });
options.onCall?.(method);
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
}
if (method === "DOM.getContentQuads") {
return { quads: [[10, 20, 210, 20, 210, 120, 10, 120]] };
}
if (method === "DOM.getNodeForLocation") return { backendNodeId: 124 };
if (method === "DOM.resolveNode") {
const id = (params as { backendNodeId?: number }).backendNodeId;
return { object: { objectId: id === 123 ? "drop-target" : "hit-target" } };
}
if (method === "Runtime.callFunctionOn") {
return { result: { value: options.targetContainsPoint ?? true } };
}
if (method === "Input.dispatchDragEvent") {
const type = (params as { type?: string }).type ?? "";
const error = options.dispatchError?.(type);
if (error instanceof Promise) return error;
if (error) throw error;
return {};
}
if (method === "Runtime.releaseObjectGroup") return {};
throw new Error(`unexpected drop CDP command ${method}`);
});
const cdp: CdpRunner = {
send: send as unknown as CdpRunner["send"],
detach,
};
return { calls, cdp, detach };
}
describe("file transfer tools", () => {
it("captures the file input activated by the requested click and injects only staged paths", async () => {
const manager = sessions();
const ctx = await manager.start("s1");
ctx.refStore.set("e3", 123, { tabId: 4 });
const { cdp, calls } = uploadCdp();
const result = await handleUpload(
manager,
{
session_id: "s1",
ref: "@e3",
files: [
{ transfer_id: "tr_1", name: "one.png", staged_path: "/private/stage/one" },
{ transfer_id: "tr_2", name: "two.png", staged_path: "/private/stage/two" },
],
},
{ cdp, tabsApi: tabsApi() },
);
expect(result).toMatchObject({ tab_id: 4, file_names: ["one.png", "two.png"] });
expect(calls[0]).toEqual({
method: "Page.setInterceptFileChooserDialog",
params: { enabled: true },
});
expect(calls).toContainEqual({
method: "Page.setInterceptFileChooserDialog",
params: { enabled: false, cancel: true },
});
expect(calls).toContainEqual({
method: "DOM.setFileInputFiles",
params: { files: ["/private/stage/one", "/private/stage/two"], backendNodeId: 456 },
});
});
it("fails immediately when the trigger does not activate a file input", async () => {
const { cdp } = uploadCdp({ inputCount: 0 });
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget(),
files: ["/private/stage/one"],
timeoutMs: 100,
trigger: async () => ({ tab_id: 4, x: 10, y: 10 }),
});
expect(result).toMatchObject({
code: "unsupported",
data: { reason: "file_input_not_activated", phase: "resolve_input" },
});
});
it("fails before clicking when chooser interception is unavailable", async () => {
const trigger = vi.fn(async () => ({ tab_id: 4, x: 10, y: 10 }));
const cdp: CdpRunner = {
send: vi.fn(async (_tabId: number, method: string) => {
if (method === "Page.setInterceptFileChooserDialog") {
throw new Error("method unavailable");
}
return {};
}) as CdpRunner["send"],
};
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget(),
files: ["/private/stage/one"],
timeoutMs: 100,
trigger,
});
expect(result).toMatchObject({
code: "cdp_failed",
data: { effect_state: "none", phase: "arm_interception" },
});
expect(trigger).not.toHaveBeenCalled();
});
it("uses an exact chooser event as an independent input-location signal", async () => {
const { cdp, calls, emitChooser } = uploadCdp({ inputCount: 0 });
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget("f1"),
files: ["/private/stage/one"],
timeoutMs: 100,
trigger: async () => {
emitChooser({ frameId: "f1", backendNodeId: 789, mode: "selectSingle" });
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(result).toMatchObject({ multiple: false });
expect(calls).toContainEqual({
method: "DOM.setFileInputFiles",
params: { files: ["/private/stage/one"], backendNodeId: 789 },
});
});
it("reports a File System Access picker without waiting for a timeout", async () => {
const { cdp, emitChooser } = uploadCdp({
inputCount: 0,
});
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget("f1"),
files: ["/private/stage/one"],
timeoutMs: 100,
trigger: async () => {
emitChooser({ frameId: "f1", mode: "selectSingle" });
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(result).toMatchObject({
code: "unsupported",
message: "upload trigger invoked a non-input file picker",
data: { reason: "file_input_not_activated", phase: "resolve_input" },
});
});
it("bounds a stuck file-input probe", async () => {
const { cdp } = uploadCdp({ pendingResolve: true });
const result = await uploadThroughActivatedFileInput({
cdp,
actionTarget: actionTarget(),
files: ["/private/stage/one"],
timeoutMs: 5,
trigger: vi.fn(),
});
expect(result).toMatchObject({
code: "timeout",
data: { reason: "file_input_probe_failed", phase: "arm_input_probe" },
});
});
it("marks a timed-out file assignment unknown and detaches browser state", async () => {
const fixture = uploadCdp();
const originalSend = fixture.cdp.send;
const detach = vi.fn(async () => {});
fixture.cdp.detach = detach;
fixture.cdp.send = vi.fn((tabId: number, method: string, params?: object) => {
if (method === "DOM.setFileInputFiles") return new Promise<never>(() => {});
return originalSend(tabId, method, params);
}) as CdpRunner["send"];
const result = await uploadThroughActivatedFileInput({
cdp: fixture.cdp,
actionTarget: actionTarget(),
files: ["/private/stage/one"],
timeoutMs: 10,
trigger: async () => ({ tab_id: 4, x: 10, y: 10 }),
});
expect(result).toMatchObject({
code: "timeout",
data: { effect_state: "unknown", phase: "set_files" },
});
expect(detach).toHaveBeenCalledWith(4);
});
it("delivers staged files through one explicit native drop without clicking", async () => {
const manager = sessions();
const ctx = await manager.start("s1");
ctx.refStore.set("e3", 123, { tabId: 4 });
const lifecycle: string[] = [];
const { cdp, calls } = dropCdp({ onCall: (method) => lifecycle.push(method) });
const overlayPhases: string[] = [];
const result = await handleUpload(
manager,
{
session_id: "s1",
ref: "@e3",
mode: "drop",
files: [{ transfer_id: "tr_1", name: "one.png", staged_path: "/stage/one.png" }],
},
{
cdp,
tabsApi: tabsApi(),
sendToTab: vi.fn(async (_tabId, message) => {
overlayPhases.push(message.phase);
lifecycle.push(`overlay:${message.phase}`);
}),
},
);
expect(result).toMatchObject({ tab_id: 4, used_ref: "e3", file_names: ["one.png"] });
expect(calls.filter((call) => call.method === "Input.dispatchDragEvent")).toEqual([
{
method: "Input.dispatchDragEvent",
params: {
type: "dragEnter",
x: 110,
y: 70,
data: { items: [], files: ["/stage/one.png"], dragOperationsMask: 1 },
},
},
{
method: "Input.dispatchDragEvent",
params: {
type: "dragOver",
x: 110,
y: 70,
data: { items: [], files: ["/stage/one.png"], dragOperationsMask: 1 },
},
},
{
method: "Input.dispatchDragEvent",
params: {
type: "drop",
x: 110,
y: 70,
data: { items: [], files: ["/stage/one.png"], dragOperationsMask: 1 },
},
},
]);
expect(calls.some((call) => call.method === "Input.dispatchMouseEvent")).toBe(false);
expect(overlayPhases).toEqual(["begin", "end"]);
expect(lifecycle.indexOf("DOM.getContentQuads")).toBeLessThan(
lifecycle.indexOf("overlay:begin"),
);
expect(lifecycle.indexOf("overlay:begin")).toBeLessThan(
lifecycle.indexOf("DOM.getNodeForLocation"),
);
expect(lifecycle.lastIndexOf("Input.dispatchDragEvent")).toBeLessThan(
lifecycle.indexOf("overlay:end"),
);
expect(lifecycle.indexOf("overlay:end")).toBeLessThan(
lifecycle.indexOf("Runtime.releaseObjectGroup"),
);
});
it("fails closed before dragging when another element owns the action point", async () => {
const { cdp, calls } = dropCdp({ targetContainsPoint: false });
const result = await uploadThroughFileDrop({
cdp,
actionTarget: actionTarget(),
files: ["/stage/one.png"],
timeoutMs: 100,
withOverlayHidden: withoutOverlay,
});
expect(result).toMatchObject({
code: "permission_denied",
data: {
reason: "file_drop_target_unavailable",
mechanism: "drop",
effect_state: "none",
phase: "resolve_target",
},
});
expect(calls.some((call) => call.method === "Input.dispatchDragEvent")).toBe(false);
});
it("dispatches an OOPIF file drop in the child target's local coordinates", async () => {
const rootCalls: Array<{ method: string; params?: object }> = [];
const childCalls: Array<{ method: string; params?: object }> = [];
const cdp: CdpRunner = {
send: vi.fn(async (_tabId, method, params) => {
rootCalls.push({ method, params });
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "DOM.getBoxModel") {
return { model: { content: [100, 100, 300, 100, 300, 200, 100, 200] } };
}
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 800, clientHeight: 600 } };
}
throw new Error(`unexpected root command ${method}`);
}) as CdpRunner["send"],
sendToTarget: vi.fn(async (_target, method, params) => {
childCalls.push({ method, params });
if (method === "DOM.scrollIntoViewIfNeeded") return {};
if (method === "DOM.getContentQuads") {
return { quads: [[10, 20, 210, 20, 210, 120, 10, 120]] };
}
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 400, clientHeight: 300 } };
}
if (method === "DOM.getNodeForLocation") {
return { backendNodeId: 124, frameId: "child" };
}
if (method === "DOM.resolveNode") {
const id = (params as { backendNodeId?: number }).backendNodeId;
return { object: { objectId: id === 123 ? "drop-target" : "hit-target" } };
}
if (method === "Runtime.callFunctionOn") return { result: { value: true } };
if (method === "Input.dispatchDragEvent" || method === "Runtime.releaseObjectGroup") {
return {};
}
throw new Error(`unexpected child command ${method}`);
}) as CdpRunner["sendToTarget"],
getFrameGraph: vi.fn(async () => ({
rootFrameId: "main",
frames: [
{ frameId: "main", target: { tabId: 4 } },
{
frameId: "child",
parentFrameId: "main",
ownerBackendNodeId: 10,
target: { tabId: 4, sessionId: "child-session" },
},
],
})),
};
const result = await uploadThroughFileDrop({
cdp,
actionTarget: actionTarget("child", "child-session"),
files: ["/stage/one.png"],
timeoutMs: 100,
withOverlayHidden: withoutOverlay,
});
expect(result).toMatchObject({ x: 110, y: 70 });
expect(
childCalls
.filter((call) => call.method === "Input.dispatchDragEvent")
.map((call) => call.params),
).toEqual([
expect.objectContaining({ type: "dragEnter", x: 110, y: 70 }),
expect.objectContaining({ type: "dragOver", x: 110, y: 70 }),
expect.objectContaining({ type: "drop", x: 110, y: 70 }),
]);
expect(rootCalls.some((call) => call.method === "Input.dispatchDragEvent")).toBe(false);
});
it("reports a missing native drag command as unsupported before commit", async () => {
const { cdp } = dropCdp({
dispatchError: (type) =>
type === "dragEnter" ? new Error("'Input.dispatchDragEvent' wasn't found") : undefined,
});
const result = await uploadThroughFileDrop({
cdp,
actionTarget: actionTarget(),
files: ["/stage/one.png"],
timeoutMs: 100,
withOverlayHidden: withoutOverlay,
});
expect(result).toMatchObject({
code: "unsupported",
data: {
reason: "upload_mechanism_unsupported",
effect_state: "none",
phase: "drag_enter",
},
});
});
it("cancels an unresolved drag before returning a pre-commit timeout", async () => {
const { cdp, calls } = dropCdp({
dispatchError: (type) => (type === "dragEnter" ? new Promise<never>(() => {}) : undefined),
});
const result = await uploadThroughFileDrop({
cdp,
actionTarget: actionTarget(),
files: ["/stage/one.png"],
timeoutMs: 10,
withOverlayHidden: withoutOverlay,
});
expect(result).toMatchObject({
code: "timeout",
data: { reason: "file_drop_failed", effect_state: "none", phase: "drag_enter" },
});
expect(calls).toContainEqual({
method: "Input.dispatchDragEvent",
params: expect.objectContaining({ type: "dragCancel" }),
});
});
it("marks an unresolved drop unknown and detaches browser state", async () => {
const { cdp, detach } = dropCdp({
dispatchError: (type) => (type === "drop" ? new Promise<never>(() => {}) : undefined),
});
const result = await uploadThroughFileDrop({
cdp,
actionTarget: actionTarget(),
files: ["/stage/one.png"],
timeoutMs: 10,
withOverlayHidden: withoutOverlay,
});
expect(result).toMatchObject({
code: "timeout",
data: { reason: "file_drop_failed", effect_state: "unknown", phase: "drop" },
});
expect(detach).toHaveBeenCalledWith(4);
});
it("routes one exact-target download through a browser-relative capability", async () => {
const manager = sessions();
const ctx = await manager.start("s1");
ctx.refStore.set("e3", 123, { tabId: 4 });
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const initial = {
id: 9,
url: "https://example.test/result.zip",
finalUrl: "https://example.test/result.zip",
filename: "result.zip",
state: "in_progress",
fileSize: -1,
totalBytes: 12,
mime: "application/zip",
danger: "safe",
} as chrome.downloads.DownloadItem;
const completed = {
...initial,
filename: "/profile/Downloads/BrowserSkill/tr_1/result.zip",
state: "complete",
fileSize: 12,
} as chrome.downloads.DownloadItem;
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn(async () => [completed]),
cancel: vi.fn(async () => {}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
let suggested: chrome.downloads.DownloadFilenameSuggestion | undefined;
const send = vi.fn(async (_tabId: number, method: string, params?: object) => {
if (method === "Page.getLayoutMetrics")
return { cssLayoutViewport: { clientWidth: 1280, clientHeight: 720 } };
if (method === "DOM.getContentQuads") return { quads: [[0, 0, 20, 0, 20, 20, 0, 20]] };
if (
method === "Input.dispatchMouseEvent" &&
(params as { type?: string }).type === "mousePressed"
) {
cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", {
url: initial.url,
suggestedFilename: "result.zip",
});
await new Promise<void>((resolve) => {
onDeterminingFilename.emit(initial, (value) => {
suggested = value;
resolve();
});
});
onCreated.emit(completed);
}
return {};
});
const cdp: CdpRunner = {
send: send as unknown as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
const result = await handleDownload(
manager,
{ session_id: "s1", ref: "@e3", browser_relative_dir: "BrowserSkill/tr_1" },
{ cdp, tabsApi: tabsApi(), downloads },
);
expect(suggested).toEqual({
filename: "BrowserSkill/tr_1/result.zip",
conflictAction: "overwrite",
});
expect(result).toMatchObject({
tab_id: 4,
suggested_filename: "result.zip",
byte_size: 12,
browser_path: completed.filename,
});
});
it("does not claim a download without an intent from the exact target", async () => {
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn(async () => []),
cancel: vi.fn(async () => {}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as unknown as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
const unrelated = {
id: 17,
url: "https://example.test/unrelated.zip",
finalUrl: "https://example.test/unrelated.zip",
filename: "unrelated.zip",
state: "in_progress",
} as chrome.downloads.DownloadItem;
let defaultSuggestionCalled = false;
const result = await captureBrowserDownload({
cdp,
target: { tabId: 4, sessionId: "expected-child" },
downloads,
browserRelativeDir: "BrowserSkill/tr_1",
timeoutMs: 5,
trigger: async () => {
cdpEvent?.({ tabId: 4, sessionId: "other-child" }, "Page.downloadWillBegin", {
url: unrelated.url,
suggestedFilename: unrelated.filename,
});
onDeterminingFilename.emit(unrelated, (suggestion) => {
defaultSuggestionCalled = suggestion === undefined;
});
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(defaultSuggestionCalled).toBe(true);
expect(downloads.cancel).not.toHaveBeenCalled();
expect(result).toMatchObject({
code: "cdp_failed",
data: { reason: "download_capture_failed" },
});
});
it("correlates a filename candidate that arrives before the CDP intent", async () => {
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const initial = {
id: 21,
url: "https://example.test/candidate-first.bin",
finalUrl: "https://example.test/candidate-first.bin",
filename: "candidate-first.bin",
state: "in_progress",
fileSize: -1,
totalBytes: 4,
bytesReceived: 0,
} as chrome.downloads.DownloadItem;
const complete = {
...initial,
state: "complete",
fileSize: 4,
} as chrome.downloads.DownloadItem;
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn(async () => [complete]),
cancel: vi.fn(async () => {}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
let suggestion: chrome.downloads.DownloadFilenameSuggestion | undefined;
const result = await captureBrowserDownload({
cdp,
target: { tabId: 4 },
downloads,
browserRelativeDir: "BrowserSkill/tr_21",
timeoutMs: 1_000,
trigger: async () => {
const suggested = new Promise<void>((resolve) => {
onDeterminingFilename.emit(initial, (value) => {
suggestion = value;
resolve();
});
});
cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", {
url: initial.url,
suggestedFilename: initial.filename,
});
await suggested;
onCreated.emit(complete);
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(suggestion).toEqual({
filename: "BrowserSkill/tr_21/candidate-first.bin",
conflictAction: "overwrite",
});
expect(result).toMatchObject({ item: { id: 21, state: "complete" } });
});
it("removes a completed download when the final size exceeds the limit", async () => {
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const initial = {
id: 22,
url: "https://example.test/oversized.bin",
finalUrl: "https://example.test/oversized.bin",
filename: "oversized.bin",
state: "in_progress",
fileSize: -1,
totalBytes: -1,
bytesReceived: 0,
} as chrome.downloads.DownloadItem;
const complete = {
...initial,
filename: "/profile/Downloads/BrowserSkill/tr_22/oversized.bin",
state: "complete",
fileSize: 8,
totalBytes: 8,
bytesReceived: 8,
} as chrome.downloads.DownloadItem;
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn(async () => [complete]),
cancel: vi.fn(async () => {}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
const result = await captureBrowserDownload({
cdp,
target: { tabId: 4 },
downloads,
browserRelativeDir: "BrowserSkill/tr_22",
maxByteSize: 4,
timeoutMs: 1_000,
trigger: async () => {
cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", {
url: initial.url,
suggestedFilename: initial.filename,
});
await new Promise<void>((resolve) => {
onDeterminingFilename.emit(initial, () => resolve());
});
onCreated.emit(initial);
onChanged.emit({ id: initial.id, state: { current: "complete" } });
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(result).toMatchObject({
code: "cdp_failed",
data: { reason: "download_capture_failed", effect_state: "committed" },
});
expect(downloads.removeFile).toHaveBeenCalledWith(initial.id);
expect(downloads.cancel).not.toHaveBeenCalled();
});
it("reconciles a download that completes while cancellation is being requested", async () => {
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const initial = {
id: 23,
url: "https://example.test/racing.bin",
finalUrl: "https://example.test/racing.bin",
filename: "racing.bin",
state: "in_progress",
fileSize: -1,
totalBytes: -1,
bytesReceived: 0,
} as chrome.downloads.DownloadItem;
const complete = {
...initial,
filename: "/profile/Downloads/BrowserSkill/tr_23/racing.bin",
state: "complete",
fileSize: 4,
totalBytes: 4,
bytesReceived: 4,
} as chrome.downloads.DownloadItem;
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn().mockResolvedValueOnce([initial]).mockResolvedValueOnce([complete]),
cancel: vi.fn(async () => {
throw new Error("download already complete");
}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
const result = await captureBrowserDownload({
cdp,
target: { tabId: 4 },
downloads,
browserRelativeDir: "BrowserSkill/tr_23",
timeoutMs: 80,
trigger: async () => {
cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", {
url: initial.url,
suggestedFilename: initial.filename,
});
await new Promise<void>((resolve) => {
onDeterminingFilename.emit(initial, () => resolve());
});
onCreated.emit(initial);
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(result).toMatchObject({
code: "cdp_failed",
data: { reason: "download_capture_failed", effect_state: "committed" },
});
expect(downloads.cancel).toHaveBeenCalledWith(initial.id);
expect(downloads.removeFile).toHaveBeenCalledWith(initial.id);
expect(result).not.toMatchObject({ data: { cleanup_state: "failed" } });
});
it("rejects ambiguous attribution without cancelling either unclaimed download", async () => {
const onCreated = fakeEvent<(item: chrome.downloads.DownloadItem) => void>();
const onChanged = fakeEvent<(delta: chrome.downloads.DownloadDelta) => void>();
const onDeterminingFilename =
fakeEvent<
(
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true
>();
const downloads: DownloadsApi = {
onCreated,
onChanged,
onDeterminingFilename,
search: vi.fn(async () => []),
cancel: vi.fn(async () => {}),
removeFile: vi.fn(async () => {}),
};
let cdpEvent: Parameters<NonNullable<CdpRunner["onEvent"]>>[0] | undefined;
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
onEvent: (handler) => {
cdpEvent = handler;
return { dispose: vi.fn() };
},
};
const defaults: number[] = [];
const candidate = (id: number) =>
({
id,
url: "https://example.test/same.bin",
finalUrl: "https://example.test/same.bin",
filename: "same.bin",
state: "in_progress",
}) as chrome.downloads.DownloadItem;
const result = await captureBrowserDownload({
cdp,
target: { tabId: 4 },
downloads,
browserRelativeDir: "BrowserSkill/tr_ambiguous",
timeoutMs: 100,
trigger: async () => {
cdpEvent?.({ tabId: 4 }, "Page.downloadWillBegin", {
url: "https://example.test/same.bin",
suggestedFilename: "same.bin",
});
for (const id of [31, 32]) {
onDeterminingFilename.emit(candidate(id), (value) => {
if (value === undefined) defaults.push(id);
});
}
return { tab_id: 4, x: 10, y: 10 };
},
});
expect(defaults.sort()).toEqual([31, 32]);
expect(downloads.cancel).not.toHaveBeenCalled();
expect(result).toMatchObject({
code: "cdp_failed",
data: { effect_state: "unknown", phase: "attribution" },
});
});
});
@@ -169,6 +169,7 @@ describe("frame geometry projection", () => {
expect(geometry).toMatchObject({
topBounds: { x: 100, y: 100, width: 200, height: 100 },
actionPoint: { x: 200, y: 150 },
targetActionPoint: { x: 100, y: 50 },
});
});
@@ -261,6 +262,7 @@ describe("frame geometry projection", () => {
expect(geometry).toMatchObject({
topBounds: { x: 100, y: 100, width: 200, height: 100 },
actionPoint: { x: 200, y: 150 },
targetActionPoint: { x: 200, y: 150 },
});
});
@@ -21,7 +21,7 @@ function fakeAgentWindow(ids: number[]) {
return id;
}),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
};
}
@@ -21,7 +21,7 @@ function fakeAgentWindow(ids: number[]) {
return id;
}),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
};
}
@@ -12,7 +12,7 @@ function fakeAgentWindow(ids: number[]) {
return id;
}),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
};
}
@@ -28,7 +28,7 @@ function fakeAgentWindow(ids: number[]) {
return id;
}),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
};
}
@@ -1904,36 +1904,41 @@ describe("buildVomScene", () => {
backendDOMNodeId: 20,
},
];
const scene = buildVomScene(axNodes, {
viewport: { width: 1000, height: 800 },
iframeNodes: new Map(),
excludedBackendNodeIds: new Set(),
surfaceProbes: [
{ triggerBackendNodeId: 20, triggerAction: "hover", subItems: ["Shoes", "Bags"] },
],
nodes: [
{
backendNodeId: 10,
parentBackendNodeId: null,
tag: "body",
attrs: {},
rect: { x: 0, y: 0, w: 1000, h: 800 },
paintOrder: 0,
position: "static",
pointerEvents: "auto",
},
{
backendNodeId: 20,
parentBackendNodeId: 10,
tag: "button",
attrs: {},
rect: { x: 20, y: 20, w: 120, h: 40 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
},
],
});
const scene = buildVomScene(
axNodes,
{
viewport: { width: 1000, height: 800 },
iframeNodes: new Map(),
excludedBackendNodeIds: new Set(),
nodes: [
{
backendNodeId: 10,
parentBackendNodeId: null,
tag: "body",
attrs: {},
rect: { x: 0, y: 0, w: 1000, h: 800 },
paintOrder: 0,
position: "static",
pointerEvents: "auto",
},
{
backendNodeId: 20,
parentBackendNodeId: 10,
tag: "button",
attrs: {},
rect: { x: 20, y: 20, w: 120, h: 40 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
},
],
},
{
surfaceProbes: [
{ triggerBackendNodeId: 20, triggerAction: "hover", subItems: ["Shoes", "Bags"] },
],
},
);
expect(scene.surfaces).toEqual([
{ triggerId: 20, triggerAction: "hover", subItems: ["Shoes", "Bags"] },
@@ -1957,46 +1962,51 @@ describe("buildVomScene", () => {
backendDOMNodeId: 21,
},
];
const scene = buildVomScene(axNodes, {
viewport: { width: 1000, height: 800 },
iframeNodes: new Map(),
excludedBackendNodeIds: new Set(),
surfaceProbes: [
{ triggerBackendNodeId: 20, triggerAction: "hover", subItems: ["My profile"] },
],
nodes: [
{
backendNodeId: 10,
parentBackendNodeId: null,
tag: "body",
attrs: {},
rect: { x: 0, y: 0, w: 1000, h: 800 },
paintOrder: 0,
position: "static",
pointerEvents: "auto",
},
{
backendNodeId: 20,
parentBackendNodeId: 10,
tag: "div",
attrs: { class: "tg-avatar" },
rect: { x: 900, y: 10, w: 30, h: 30 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
},
{
backendNodeId: 21,
parentBackendNodeId: 20,
tag: "div",
attrs: { class: "tg-avatar__inner" },
rect: { x: 902, y: 12, w: 26, h: 26 },
paintOrder: 2,
position: "static",
pointerEvents: "auto",
},
],
});
const scene = buildVomScene(
axNodes,
{
viewport: { width: 1000, height: 800 },
iframeNodes: new Map(),
excludedBackendNodeIds: new Set(),
nodes: [
{
backendNodeId: 10,
parentBackendNodeId: null,
tag: "body",
attrs: {},
rect: { x: 0, y: 0, w: 1000, h: 800 },
paintOrder: 0,
position: "static",
pointerEvents: "auto",
},
{
backendNodeId: 20,
parentBackendNodeId: 10,
tag: "div",
attrs: { class: "tg-avatar" },
rect: { x: 900, y: 10, w: 30, h: 30 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
},
{
backendNodeId: 21,
parentBackendNodeId: 20,
tag: "div",
attrs: { class: "tg-avatar__inner" },
rect: { x: 902, y: 12, w: 26, h: 26 },
paintOrder: 2,
position: "static",
pointerEvents: "auto",
},
],
},
{
surfaceProbes: [
{ triggerBackendNodeId: 20, triggerAction: "hover", subItems: ["My profile"] },
],
},
);
expect(renderVom(scene).text).toContain('@e1 button "image" [hover first: My profile]');
});
@@ -2017,48 +2027,53 @@ describe("buildVomScene", () => {
backendDOMNodeId: 21,
},
];
const scene = buildVomScene(axNodes, {
viewport: { width: 1000, height: 800 },
iframeNodes: new Map(),
excludedBackendNodeIds: new Set(),
surfaceProbes: [
{ triggerBackendNodeId: 20, triggerAction: "hover", subItems: ["My profile"] },
{ triggerBackendNodeId: 20, triggerAction: "hover", subItems: ["Sign out"] },
{ triggerBackendNodeId: 21, triggerAction: "hover", subItems: ["Settings"] },
],
nodes: [
{
backendNodeId: 10,
parentBackendNodeId: null,
tag: "body",
attrs: {},
rect: { x: 0, y: 0, w: 1000, h: 800 },
paintOrder: 0,
position: "static",
pointerEvents: "auto",
},
{
backendNodeId: 20,
parentBackendNodeId: 10,
tag: "div",
attrs: { class: "tg-avatar" },
rect: { x: 900, y: 10, w: 30, h: 30 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
},
{
backendNodeId: 21,
parentBackendNodeId: 20,
tag: "div",
attrs: { class: "tg-avatar__inner" },
rect: { x: 902, y: 12, w: 26, h: 26 },
paintOrder: 2,
position: "static",
pointerEvents: "auto",
},
],
});
const scene = buildVomScene(
axNodes,
{
viewport: { width: 1000, height: 800 },
iframeNodes: new Map(),
excludedBackendNodeIds: new Set(),
nodes: [
{
backendNodeId: 10,
parentBackendNodeId: null,
tag: "body",
attrs: {},
rect: { x: 0, y: 0, w: 1000, h: 800 },
paintOrder: 0,
position: "static",
pointerEvents: "auto",
},
{
backendNodeId: 20,
parentBackendNodeId: 10,
tag: "div",
attrs: { class: "tg-avatar" },
rect: { x: 900, y: 10, w: 30, h: 30 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
},
{
backendNodeId: 21,
parentBackendNodeId: 20,
tag: "div",
attrs: { class: "tg-avatar__inner" },
rect: { x: 902, y: 12, w: 26, h: 26 },
paintOrder: 2,
position: "static",
pointerEvents: "auto",
},
],
},
{
surfaceProbes: [
{ triggerBackendNodeId: 20, triggerAction: "hover", subItems: ["My profile"] },
{ triggerBackendNodeId: 20, triggerAction: "hover", subItems: ["Sign out"] },
{ triggerBackendNodeId: 21, triggerAction: "hover", subItems: ["Settings"] },
],
},
);
expect(scene.surfaces).toEqual([
{ triggerId: 21, triggerAction: "hover", subItems: ["My profile"] },
@@ -2081,54 +2096,59 @@ describe("buildVomScene", () => {
backendDOMNodeId: 21,
},
];
const scene = buildVomScene(axNodes, {
viewport: { width: 1000, height: 800 },
iframeNodes: new Map(),
excludedBackendNodeIds: new Set(),
surfaceProbes: [
{
triggerBackendNodeId: 20,
triggerPoint: { x: 915, y: 25 },
triggerAction: "hover",
subItems: ["My profile", "Sign out"],
},
],
nodes: [
{
backendNodeId: 10,
parentBackendNodeId: null,
tag: "body",
attrs: {},
rect: { x: 0, y: 0, w: 1000, h: 800 },
paintOrder: 0,
position: "static",
pointerEvents: "auto",
cursor: "auto",
},
{
backendNodeId: 20,
parentBackendNodeId: 10,
tag: "div",
attrs: { class: "tg-avatar" },
rect: { x: 900, y: 10, w: 30, h: 30 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
cursor: "pointer",
},
{
backendNodeId: 21,
parentBackendNodeId: 20,
tag: "div",
attrs: { class: "tg-avatar__inner" },
rect: { x: 902, y: 12, w: 26, h: 26 },
paintOrder: 2,
position: "static",
pointerEvents: "auto",
cursor: "pointer",
},
],
});
const scene = buildVomScene(
axNodes,
{
viewport: { width: 1000, height: 800 },
iframeNodes: new Map(),
excludedBackendNodeIds: new Set(),
nodes: [
{
backendNodeId: 10,
parentBackendNodeId: null,
tag: "body",
attrs: {},
rect: { x: 0, y: 0, w: 1000, h: 800 },
paintOrder: 0,
position: "static",
pointerEvents: "auto",
cursor: "auto",
},
{
backendNodeId: 20,
parentBackendNodeId: 10,
tag: "div",
attrs: { class: "tg-avatar" },
rect: { x: 900, y: 10, w: 30, h: 30 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
cursor: "pointer",
},
{
backendNodeId: 21,
parentBackendNodeId: 20,
tag: "div",
attrs: { class: "tg-avatar__inner" },
rect: { x: 902, y: 12, w: 26, h: 26 },
paintOrder: 2,
position: "static",
pointerEvents: "auto",
cursor: "pointer",
},
],
},
{
surfaceProbes: [
{
triggerBackendNodeId: 20,
triggerPoint: { x: 915, y: 25 },
triggerAction: "hover",
subItems: ["My profile", "Sign out"],
},
],
},
);
expect(scene.surfaces).toEqual([
{ triggerId: 20, triggerAction: "hover", subItems: ["My profile", "Sign out"] },
@@ -2154,41 +2174,46 @@ describe("buildVomScene", () => {
backendDOMNodeId: 42,
},
];
const scene = buildVomScene(axNodes, {
viewport: { width: 1000, height: 800 },
iframeNodes: new Map(),
excludedBackendNodeIds: new Set(),
surfaceProbes: [
{
triggerBackendNodeId: 999,
triggerPoint: { x: 900, y: 20 },
triggerAction: "hover",
subItems: ["My profile"],
},
],
nodes: [
{
backendNodeId: 10,
parentBackendNodeId: null,
tag: "body",
attrs: {},
rect: { x: 0, y: 0, w: 1000, h: 800 },
paintOrder: 0,
position: "static",
pointerEvents: "auto",
},
{
backendNodeId: 42,
parentBackendNodeId: 10,
tag: "a",
attrs: {},
rect: { x: 860, y: 120, w: 40, h: 30 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
},
],
});
const scene = buildVomScene(
axNodes,
{
viewport: { width: 1000, height: 800 },
iframeNodes: new Map(),
excludedBackendNodeIds: new Set(),
nodes: [
{
backendNodeId: 10,
parentBackendNodeId: null,
tag: "body",
attrs: {},
rect: { x: 0, y: 0, w: 1000, h: 800 },
paintOrder: 0,
position: "static",
pointerEvents: "auto",
},
{
backendNodeId: 42,
parentBackendNodeId: 10,
tag: "a",
attrs: {},
rect: { x: 860, y: 120, w: 40, h: 30 },
paintOrder: 1,
position: "static",
pointerEvents: "auto",
},
],
},
{
surfaceProbes: [
{
triggerBackendNodeId: 999,
triggerPoint: { x: 900, y: 20 },
triggerAction: "hover",
subItems: ["My profile"],
},
],
},
);
expect(scene.surfaces).toBeUndefined();
expect(renderVom(scene).text).not.toContain("[hover first:");
@@ -2606,7 +2631,106 @@ describe("handleSnapshot", () => {
);
});
it("enables conditional surface probing for semantic observe", async () => {
it("hovers the page only after the accessibility tree has been captured", async () => {
// Hovering can open menus and reflow the page. Probing between the DOM and
// AX captures would leave the two halves of one observation describing the
// page on either side of that change.
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
await sm.start("aa11");
const root: CdpAxNode = {
nodeId: "1",
role: { type: "role", value: "RootWebArea" },
name: { type: "computedString", value: "Example" },
backendDOMNodeId: 100,
childIds: ["2"],
};
const button: CdpAxNode = {
nodeId: "2",
parentId: "1",
role: { type: "role", value: "button" },
name: { type: "computedString", value: "Products" },
backendDOMNodeId: 200,
};
const strings = ["body", "button", "position", "static", "pointer-events", "auto", "cursor"];
const i = (s: string) => strings.indexOf(s);
const methodOrder: string[] = [];
const send = vi.fn(async (_tabId: number, method: string, params?: object) => {
methodOrder.push(method);
if (method === "Accessibility.enable") return {};
if (method === "Accessibility.getFullAXTree") return { nodes: [root, button] };
if (method === "Page.getLayoutMetrics") {
return { cssLayoutViewport: { clientWidth: 1000, clientHeight: 800, pageX: 0, pageY: 0 } };
}
if (method === "DOMSnapshot.enable") return {};
if (method === "DOMSnapshot.captureSnapshot") {
return {
strings,
documents: [
{
nodes: {
parentIndex: [-1, 0],
nodeName: [i("body"), i("button")],
backendNodeId: [100, 200],
attributes: [[], []],
},
layout: {
nodeIndex: [0, 1],
styles: [
[i("static"), i("auto"), i("auto")],
[i("static"), i("auto"), i("auto")],
],
bounds: [
[0, 0, 1000, 800],
[20, 20, 120, 40],
],
paintOrders: [0, 1],
},
},
],
};
}
if (method === "Input.dispatchMouseEvent") return {};
if (method === "Runtime.evaluate") {
const expression = (params as { expression?: string } | undefined)?.expression ?? "";
// Report the button as a `:hover` rule target so a real hover fires.
if (expression.includes("document.styleSheets")) {
return { result: { value: [{ x: 80, y: 40 }] } };
}
return { result: { value: [] } };
}
throw new Error(`unexpected CDP method ${method}`);
});
const res = await handleObserve(
sm,
{ session_id: "aa11", probe_hover: true },
{
cdp: {
send: send as unknown as <T = unknown>(
tabId: number,
method: string,
params?: object,
) => Promise<T>,
trackSessionTab: vi.fn(),
},
tabsApi: {
get: vi.fn(
async (tabId: number) =>
({ id: tabId, windowId: 100, active: true }) as chrome.tabs.Tab,
),
query: vi.fn(async () => [{ id: 4, windowId: 100, active: true } as chrome.tabs.Tab]),
},
},
);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
const firstHover = methodOrder.indexOf("Input.dispatchMouseEvent");
expect(firstHover).toBeGreaterThan(-1);
expect(methodOrder.lastIndexOf("DOMSnapshot.captureSnapshot")).toBeLessThan(firstHover);
expect(methodOrder.lastIndexOf("Accessibility.getFullAXTree")).toBeLessThan(firstHover);
});
it("runs conditional surface probing only when observe opts in", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
await sm.start("aa11");
const root: CdpAxNode = {
@@ -2662,30 +2786,46 @@ describe("handleSnapshot", () => {
if (method === "Runtime.evaluate") return { result: { value: [] } };
throw new Error(`unexpected CDP method ${method}`);
});
const res = await handleObserve(
const deps = {
cdp: {
send: send as unknown as <T = unknown>(
tabId: number,
method: string,
params?: object,
) => Promise<T>,
trackSessionTab: vi.fn(),
},
tabsApi: {
get: vi.fn(
async (tabId: number) => ({ id: tabId, windowId: 100, active: true }) as chrome.tabs.Tab,
),
query: vi.fn(async () => [{ id: 4, windowId: 100, active: true } as chrome.tabs.Tab]),
},
};
const withoutProbe = await handleObserve(
sm,
{ session_id: "aa11", debug_surfaces: true },
{
cdp: {
send: send as unknown as <T = unknown>(
tabId: number,
method: string,
params?: object,
) => Promise<T>,
trackSessionTab: vi.fn(),
},
tabsApi: {
get: vi.fn(
async (tabId: number) =>
({ id: tabId, windowId: 100, active: true }) as chrome.tabs.Tab,
),
query: vi.fn(async () => [{ id: 4, windowId: 100, active: true } as chrome.tabs.Tab]),
},
},
deps,
);
if ("code" in withoutProbe) {
throw new Error(`unexpected error: ${JSON.stringify(withoutProbe)}`);
}
expect(withoutProbe.hover_probe).toBeUndefined();
expect(send).not.toHaveBeenCalledWith(
4,
"Runtime.evaluate",
expect.objectContaining({ returnByValue: true }),
);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
expect(res.debug).toEqual({ surface_probes: [] });
const withProbe = await handleObserve(
sm,
{ session_id: "aa11", debug_surfaces: true, probe_hover: true },
deps,
);
if ("code" in withProbe) throw new Error(`unexpected error: ${JSON.stringify(withProbe)}`);
expect(withProbe.debug).toEqual({ surface_probes: [] });
expect(withProbe.hover_probe).toEqual({ performed: true, revealed_content: false });
expect(send).toHaveBeenCalledWith(
4,
"Runtime.evaluate",
@@ -1,8 +1,26 @@
import { describe, expect, it, vi } from "vitest";
import { SessionManager } from "@/session-manager/manager";
import { handleSessionStop } from "../session";
import type { ChromeTabsApi } from "../shared";
import { type AgentOverlayResetApi, type ChromeWindowsApi, type TabMutationApi } from "../tabs";
/** Build a read-only query api over a FakeState. */
function makeQuery(state: FakeState): ChromeTabsApi {
return {
get: vi.fn(async (id) => {
const t = state.tabs.get(id);
if (!t) throw new Error(`tab ${id} not found`);
return t;
}),
query: vi.fn(async (q: chrome.tabs.QueryInfo) => {
const w = q.windowId;
return Array.from(state.tabs.values()).filter(
(t) => typeof w !== "number" || t.windowId === w,
);
}),
};
}
function fakeAgentWindow(ids: number[]) {
let i = 0;
const create = vi.fn(async () => {
@@ -11,7 +29,7 @@ function fakeAgentWindow(ids: number[]) {
return id;
});
const remove = vi.fn(async () => {});
const ensureActiveTab = vi.fn(async () => {});
const ensureActiveTab = vi.fn(async () => 0);
return { create, remove, ensureActiveTab };
}
@@ -30,7 +48,9 @@ function makeApis(
} {
const tabs: TabMutationApi = {
create: vi.fn(),
remove: vi.fn(async () => {}),
remove: vi.fn(async (id: number) => {
state.tabs.delete(id);
}),
update: vi.fn(async (_id, _p) => undefined),
get: vi.fn(async (id) => {
const t = state.tabs.get(id);
@@ -267,4 +287,275 @@ describe("handleSessionStop with auto-return", () => {
await handleSessionStop(sm, { session_id: "aa11" }, { tabManagement: { tabs, windows } });
expect(ctx.refStore.isEmpty()).toBe(true);
});
it.each([
"before teardown",
"during its return",
"after its return failed",
"before its turn in the return loop",
])("finishes stopping when a borrowed tab closes %s", async (timing) => {
const aw = fakeAgentWindow([100]);
const sm = new SessionManager({ agentWindow: aw });
const ctx = await sm.start("aa11");
ctx.borrowedTabs.set(1, { tabId: 1, originalWindowId: 200, originalIndex: 0 });
ctx.borrowedTabs.set(2, { tabId: 2, originalWindowId: 200, originalIndex: 1 });
const state: FakeState = {
tabs: new Map([
[1, { id: 1, windowId: 100 } as chrome.tabs.Tab],
[2, { id: 2, windowId: 100 } as chrome.tabs.Tab],
]),
windowsClosed: new Set(),
moves: [],
};
const { tabs, windows } = makeApis(state);
const cdp = { detachSession: vi.fn(async () => {}) };
const closeTab = (tabId: number) => {
state.tabs.delete(tabId);
sm.forgetClosedTab(tabId);
};
const baseMove = tabs.move;
tabs.move = vi.fn(async (tabId, props) => {
if (tabId === 1 && timing === "during its return") closeTab(1);
if (tabId === 1 && timing === "after its return failed") {
throw new Error("simulated move failure");
}
if (tabId === 2 && timing === "after its return failed") closeTab(1);
if (tabId === 1 && timing === "before its turn in the return loop") closeTab(2);
if (!state.tabs.has(tabId)) throw new Error(`tab ${tabId} not found`);
return baseMove(tabId, props);
});
if (timing === "before teardown") closeTab(1);
const res = await handleSessionStop(
sm,
{ session_id: "aa11" },
{
cdp,
tabManagement: {
tabs,
windows,
agentOverlayReset: { resetAgentOverlays: vi.fn(async () => {}) },
},
tabsQuery: makeQuery(state),
},
);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
const closedTabId = timing === "before its turn in the return loop" ? 2 : 1;
expect(res.returned_tab_ids).toEqual([closedTabId === 1 ? 2 : 1]);
expect(res.return_failures).toBeUndefined();
expect(ctx.borrowedTabs.size).toBe(0);
expect(sm.has("aa11")).toBe(false);
expect(cdp.detachSession).toHaveBeenCalledWith("aa11");
expect(aw.remove).toHaveBeenCalledWith(100);
if (timing === "before teardown" || timing === "before its turn in the return loop") {
expect(tabs.move).not.toHaveBeenCalledWith(closedTabId, expect.anything());
}
});
});
describe("handleSessionStop window release (issue #57)", () => {
const agentWindowId = 100;
it("releases the window when user-created tabs remain", async () => {
const aw = fakeAgentWindow([agentWindowId]);
// ensureActiveTab returns the home tab id (10). We override the fake to
// return a known id.
aw.ensureActiveTab = vi.fn(async () => 10);
const sm = new SessionManager({ agentWindow: aw });
const ctx = await sm.start("aa11");
ctx.agentCreatedTabs.add(11);
ctx.agentCreatedTabs.add(12);
// user tab 99 (created via Chrome UI, NOT in agentCreatedTabs)
const state: FakeState = {
tabs: new Map([
[10, { id: 10, windowId: agentWindowId } as chrome.tabs.Tab],
[11, { id: 11, windowId: agentWindowId } as chrome.tabs.Tab],
[12, { id: 12, windowId: agentWindowId } as chrome.tabs.Tab],
[99, { id: 99, windowId: agentWindowId } as chrome.tabs.Tab],
]),
windowsClosed: new Set(),
moves: [],
};
const { tabs, windows } = makeApis(state);
const query = makeQuery(state);
const res = await handleSessionStop(
sm,
{ session_id: "aa11" },
{ tabManagement: { tabs, windows }, tabsQuery: query },
);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
expect(res.window_released).toBe(true);
// agent tabs + home removed, user tab 99 kept
expect((tabs.remove as ReturnType<typeof vi.fn>).mock.calls.map((c) => c[0]).sort()).toEqual([
10, 11, 12,
]);
expect(state.tabs.has(99)).toBe(true);
// window released (dropOnly), not closed
expect(aw.remove).not.toHaveBeenCalled();
expect(sm.has("aa11")).toBe(false);
});
it("closes the window (not release) when no user tabs remain", async () => {
const aw = fakeAgentWindow([agentWindowId]);
aw.ensureActiveTab = vi.fn(async () => 10);
const sm = new SessionManager({ agentWindow: aw });
const ctx = await sm.start("aa11");
ctx.agentCreatedTabs.add(11);
const state: FakeState = {
tabs: new Map([
[10, { id: 10, windowId: agentWindowId } as chrome.tabs.Tab],
[11, { id: 11, windowId: agentWindowId } as chrome.tabs.Tab],
]),
windowsClosed: new Set(),
moves: [],
};
const { tabs, windows } = makeApis(state);
const query = makeQuery(state);
const res = await handleSessionStop(
sm,
{ session_id: "aa11" },
{ tabManagement: { tabs, windows }, tabsQuery: query },
);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
expect(res.window_released).toBeFalsy();
expect(aw.remove).toHaveBeenCalledWith(agentWindowId);
expect(sm.has("aa11")).toBe(false);
});
it("is non-fatal when an agent tab is already gone", async () => {
const aw = fakeAgentWindow([agentWindowId]);
aw.ensureActiveTab = vi.fn(async () => 10);
const sm = new SessionManager({ agentWindow: aw });
const ctx = await sm.start("aa11");
ctx.agentCreatedTabs.add(11); // no longer in state → remove throws
const state: FakeState = {
tabs: new Map([[10, { id: 10, windowId: agentWindowId } as chrome.tabs.Tab]]),
windowsClosed: new Set(),
moves: [],
};
const { tabs, windows } = makeApis(state);
(tabs.remove as ReturnType<typeof vi.fn>).mockImplementation(async (id: number) => {
if (id === 11) throw new Error("tab 11 already closed");
state.tabs.delete(id); // home (10) removed normally
});
const query = makeQuery(state);
const res = await handleSessionStop(
sm,
{ session_id: "aa11" },
{ tabManagement: { tabs, windows }, tabsQuery: query },
);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
// user tab 10? no — 10 is home, removed. window would be empty → close.
expect(res.window_released).toBeFalsy();
expect(aw.remove).toHaveBeenCalled();
});
it("degrades to close-window when tabsQuery is missing", async () => {
const aw = fakeAgentWindow([agentWindowId]);
aw.ensureActiveTab = vi.fn(async () => 10);
const sm = new SessionManager({ agentWindow: aw });
const ctx = await sm.start("aa11");
ctx.agentCreatedTabs.add(11);
const state: FakeState = {
tabs: new Map([
[10, { id: 10, windowId: agentWindowId } as chrome.tabs.Tab],
[11, { id: 11, windowId: agentWindowId } as chrome.tabs.Tab],
]),
windowsClosed: new Set(),
moves: [],
};
const { tabs, windows } = makeApis(state);
const res = await handleSessionStop(
sm,
{ session_id: "aa11" },
{ tabManagement: { tabs, windows } }, // no tabsQuery
);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
// No query available → conservatively close the window.
expect(res.window_released).toBeFalsy();
expect(aw.remove).toHaveBeenCalledWith(agentWindowId);
});
it("degrades to close-window when tabsQuery.query throws", async () => {
const aw = fakeAgentWindow([agentWindowId]);
aw.ensureActiveTab = vi.fn(async () => 10);
const sm = new SessionManager({ agentWindow: aw });
const ctx = await sm.start("aa11");
ctx.agentCreatedTabs.add(11);
const state: FakeState = {
tabs: new Map([[11, { id: 11, windowId: agentWindowId } as chrome.tabs.Tab]]),
windowsClosed: new Set(),
moves: [],
};
const { tabs, windows } = makeApis(state);
const query = makeQuery(state);
(query.query as ReturnType<typeof vi.fn>).mockImplementation(async () => {
throw new Error("query failed");
});
const res = await handleSessionStop(
sm,
{ session_id: "aa11" },
{ tabManagement: { tabs, windows }, tabsQuery: query },
);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
expect(res.window_released).toBeFalsy();
expect(aw.remove).toHaveBeenCalled();
});
it("tracks agentCreatedTabs across tab_create / tab_close", async () => {
const aw = fakeAgentWindow([agentWindowId]);
aw.ensureActiveTab = vi.fn(async () => 10);
const sm = new SessionManager({ agentWindow: aw });
const ctx = await sm.start("aa11");
// Simulate handleTabCreate adding, then handleTabClose removing.
ctx.agentCreatedTabs.add(11);
expect(ctx.agentCreatedTabs.has(11)).toBe(true);
ctx.agentCreatedTabs.delete(11); // as handleTabClose does on success
expect(ctx.agentCreatedTabs.has(11)).toBe(false);
// Unknown id delete is a safe no-op (user-closed tab).
expect(() => ctx.agentCreatedTabs.delete(999)).not.toThrow();
});
it("closes the window (not release) when an agent tab fails to close", async () => {
// Regression: a leaked agent tab (still in agentCreatedTabs, but
// still present because Step 4 remove() threw) must NOT be mistaken
// for a user tab. Otherwise the window would be released (dropOnly)
// and the agent tab would leak (issue #57 regression).
const aw = fakeAgentWindow([agentWindowId]);
aw.ensureActiveTab = vi.fn(async () => 10);
const sm = new SessionManager({ agentWindow: aw });
const ctx = await sm.start("aa11");
ctx.agentCreatedTabs.add(11); // Step 4 remove() will throw for this tab
const state: FakeState = {
tabs: new Map([
[10, { id: 10, windowId: agentWindowId } as chrome.tabs.Tab],
[11, { id: 11, windowId: agentWindowId } as chrome.tabs.Tab],
]),
windowsClosed: new Set(),
moves: [],
};
const { tabs, windows } = makeApis(state);
(tabs.remove as ReturnType<typeof vi.fn>).mockImplementation(async (id: number) => {
if (id === 11) throw new Error("tab 11 failed to close");
state.tabs.delete(id); // home (10) removed normally; 11 lingers
});
const query = makeQuery(state);
const res = await handleSessionStop(
sm,
{ session_id: "aa11" },
{ tabManagement: { tabs, windows }, tabsQuery: query },
);
if ("code" in res) throw new Error(`unexpected error: ${JSON.stringify(res)}`);
// The leaked agent tab must NOT keep the window open.
expect(res.window_released).toBeFalsy();
expect(aw.remove).toHaveBeenCalledWith(agentWindowId);
expect(sm.has("aa11")).toBe(false);
});
});
@@ -12,7 +12,7 @@ function fakeAgentWindow() {
return {
create: vi.fn(async () => 100),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
};
}
@@ -11,7 +11,7 @@ function fakeAgentWindow(ids: number[]) {
return id;
},
remove: async () => {},
ensureActiveTab: async () => {},
ensureActiveTab: async () => 1,
};
}
@@ -23,7 +23,7 @@ function fakeAgentWindow(ids: number[]) {
return id;
}),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
};
}
@@ -221,6 +221,7 @@ describe("handleTabCreate", () => {
active: true,
});
expect(res).toMatchObject({ tab_id: 10, window_id: 100 });
expect(sm.get("aa11")?.agentCreatedTabs.has(10)).toBe(true);
});
it("forwards explicit url + index + active=false", async () => {
@@ -256,6 +257,41 @@ describe("handleTabCreate", () => {
const res = await handleTabCreate(sm, { session_id: "aa11", index: -1 }, { tabs: api });
expect(res).toMatchObject({ code: "invalid_params" });
});
it("does not claim any tab when chrome.tabs.create fails", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
await sm.start("aa11");
const state: FakeTabState = { tabs: new Map(), nextTabId: 10, windowsClosed: new Set() };
const { api, spies } = makeTabMutationApi(state);
spies.create.mockRejectedValueOnce(new Error("chrome.tabs.create failed"));
const res = await handleTabCreate(sm, { session_id: "aa11" }, { tabs: api });
expect(res).toMatchObject({ code: "protocol_error" });
expect(sm.get("aa11")?.agentCreatedTabs).toEqual(new Set([1]));
});
it("claims the returned id even when a user tab appears during tab creation", async () => {
const sm = new SessionManager({ agentWindow: fakeAgentWindow([100]) });
const ctx = await sm.start("aa11");
const state: FakeTabState = { tabs: new Map(), nextTabId: 10, windowsClosed: new Set() };
const { api, spies } = makeTabMutationApi(state);
let resolveCreate!: (tab: chrome.tabs.Tab) => void;
const createResult = new Promise<chrome.tabs.Tab>((resolve) => {
resolveCreate = resolve;
});
spies.create.mockImplementationOnce(() => createResult);
const pending = handleTabCreate(sm, { session_id: "aa11" }, { tabs: api });
// A user can create an unrelated same-window tab while Chrome is still
// resolving the agent request. There is no pending slot for it to steal.
expect(ctx.agentCreatedTabs.has(99)).toBe(false);
resolveCreate({ id: 10, windowId: 100, active: true } as chrome.tabs.Tab);
await expect(pending).resolves.toMatchObject({ tab_id: 10 });
expect(ctx.agentCreatedTabs.has(10)).toBe(true);
expect(ctx.agentCreatedTabs.has(99)).toBe(false);
});
});
describe("handleTabClose", () => {
@@ -12,7 +12,7 @@ function fakeAgentWindow(ids: number[]) {
return id;
}),
remove: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => {}),
ensureActiveTab: vi.fn(async () => 1),
};
}
@@ -11,7 +11,7 @@ function fakeAgentWindow(ids: number[]) {
return id;
});
const remove = vi.fn(async () => {});
const ensureActiveTab = vi.fn(async () => {});
const ensureActiveTab = vi.fn(async () => 1);
return { create, remove, ensureActiveTab };
}
@@ -24,7 +24,7 @@ export interface BorrowResponseMessage {
allowed: boolean;
}
export const CONFIRMATION_TIMEOUT_MS = 5000;
export const CONFIRMATION_TIMEOUT_MS = 60_000;
const EXIT_ANIMATION_MS = 150;
/** Matches BorrowConfirmationOverlay progress ring/bar transition (duration-1000). */
export const PROGRESS_TRANSITION_MS = 1000;
@@ -480,6 +480,13 @@ export async function requestBorrowConfirmation(
settle(false);
}, BACKGROUND_TIMEOUT_MS);
// Bring the user window to the front so the overlay is visible even
// when the Agent Window has stolen focus. This is fire-and-forget: if
// it fails the overlay / OS notification still give the user a path.
void windowsApi.update(notificationAnchor.windowId, { focused: true }).catch((err) => {
console.debug("[bsk borrow] proactive focus of user window failed", err);
});
// Surface the OS notification *before* messaging any candidate so the
// user has a parallel signal even if every content script is missing.
// The Allow / Deny buttons let the user authorize explicitly when the
+71 -5
View File
@@ -4,6 +4,7 @@ import type { Transport } from "@/transport/transport";
import type {
ClickParams,
ConsoleParams,
DownloadParams,
EmulateParams,
EvaluateParams,
FillParams,
@@ -28,10 +29,12 @@ import type {
ScreenshotParams,
SelectParams,
SnapshotParams,
UploadParams,
WaitForNavigationParams,
} from "@/transport/types";
import { isRequestFrame } from "@/transport/types";
import { handleConsole } from "./console";
import { handleDownload } from "./download";
import { type EmulateCdpRunner, handleEmulate } from "./emulate";
import { handleEvaluate } from "./evaluate";
import { handleRequestHelp } from "./human-loop";
@@ -66,6 +69,7 @@ import {
import { chromeTabsApi, lookupSession, resolveTargetTab } from "./shared";
import {
type BorrowConfirmationApprover,
chromeTabMutationApi,
handleTabBorrow,
handleTabClose,
handleTabCreate,
@@ -79,6 +83,7 @@ import {
type TabReturnParams,
type TabSelectParams,
} from "./tabs";
import { handleUpload } from "./upload";
import { handleWaitForNavigation } from "./waits";
import { handleWindowResize, type WindowResizeParams } from "./window";
@@ -114,6 +119,8 @@ export interface DispatcherDeps {
onSessionsChanged?: () => void;
/** Invoked before a tool that dispatches page input or mutates browser state is forwarded. */
onBrowserControlResumed?: (sessionId: string) => void;
/** Invoked after a tab is explicitly claimed so its overlay can be refreshed immediately. */
onAgentTabClaimed?: (tabId: number, windowId: number) => void;
/** User approval for `tool.tab_borrow` (overlay in content script). */
approveBorrow?: BorrowConfirmationApprover;
/** i18n notification copy for `tool.request_help` (resolved per-call). */
@@ -143,6 +150,7 @@ export class ToolDispatcher {
private readonly recording?: RecordRuntimeDeps;
private readonly onSessionsChanged?: () => void;
private readonly onBrowserControlResumed?: (sessionId: string) => void;
private readonly onAgentTabClaimed?: (tabId: number, windowId: number) => void;
private readonly approveBorrow?: BorrowConfirmationApprover;
private readonly helpNotificationCopy?: () => { title: string; body: string };
private subscription: { dispose(): void } | null = null;
@@ -163,6 +171,7 @@ export class ToolDispatcher {
this.recording = deps.recording;
this.onSessionsChanged = deps.onSessionsChanged;
this.onBrowserControlResumed = deps.onBrowserControlResumed;
this.onAgentTabClaimed = deps.onAgentTabClaimed;
this.approveBorrow = deps.approveBorrow;
this.helpNotificationCopy = deps.helpNotificationCopy;
}
@@ -298,13 +307,25 @@ export class ToolDispatcher {
await this.releaseHoverLatch((req.params as SessionStopParams).session_id);
return handleSessionStop(this.sessions, req.params as SessionStopParams, {
cdp: this.cdp,
// Must be wired in production: the agent-tab cleanup and the
// window-release decision (issue #57) read these deps directly
// and silently no-op when they are absent.
tabManagement: { tabs: chromeTabMutationApi },
tabsQuery: chromeTabsApi,
signal,
});
}
case "tool.tab_list":
return handleTabList(this.sessions, req.params as TabListParams, chromeTabsApi, signal);
case "tool.tab_create":
return handleTabCreate(this.sessions, req.params as TabCreateParams, { signal });
case "tool.tab_create": {
const result = await handleTabCreate(this.sessions, req.params as TabCreateParams, {
signal,
});
if (!isRpcError(result)) {
this.onAgentTabClaimed?.(result.tab_id, result.window_id);
}
return result;
}
case "tool.tab_close":
return this.withHoverReleaseForRequest(
req.params as TabCloseParams,
@@ -313,11 +334,16 @@ export class ToolDispatcher {
);
case "tool.tab_select":
return handleTabSelect(this.sessions, req.params as TabSelectParams, { signal });
case "tool.tab_borrow":
return handleTabBorrow(this.sessions, req.params as TabBorrowParams, {
case "tool.tab_borrow": {
const result = await handleTabBorrow(this.sessions, req.params as TabBorrowParams, {
signal,
approveBorrow: this.approveBorrow,
});
if (!isRpcError(result)) {
this.onAgentTabClaimed?.(result.tab_id, result.agent_window_id);
}
return result;
}
case "tool.tab_return":
return handleTabReturn(this.sessions, req.params as TabReturnParams, { signal });
case "tool.window_resize":
@@ -383,7 +409,11 @@ export class ToolDispatcher {
? {
cdp: this.cdp,
tabsApi: chromeTabsCaptureApi,
conditionalSurfaceProbe: !this.hasHoverLatchForScope(hoverScope),
// Active hover probing is opt-in. A held hover latch still
// suppresses it, because probing would move the cursor off
// the element the caller is deliberately holding.
conditionalSurfaceProbe:
params.probe_hover === true && !this.hasHoverLatchForScope(hoverScope),
hoverProbeBypassOverlay: bypassOverlay,
}
: undefined,
@@ -513,6 +543,40 @@ export class ToolDispatcher {
),
signal,
);
case "tool.upload":
return this.withHoverReleaseForRequest(
req.params as UploadParams,
() =>
this.cdp
? handleUpload(this.sessions, req.params as UploadParams, {
cdp: this.cdp,
tabsApi: chromeTabsApi,
signal,
bypassOverlay,
})
: Promise.resolve({
code: "unsupported",
message: "upload requires CDP",
} satisfies RpcError),
signal,
);
case "tool.download":
return this.withHoverReleaseForRequest(
req.params as DownloadParams,
() =>
this.cdp
? handleDownload(this.sessions, req.params as DownloadParams, {
cdp: this.cdp,
tabsApi: chromeTabsApi,
signal,
bypassOverlay,
})
: Promise.resolve({
code: "unsupported",
message: "download requires CDP",
} satisfies RpcError),
signal,
);
case "tool.evaluate":
return handleEvaluate(
this.sessions,
@@ -719,6 +783,8 @@ function sessionIdForBrowserControlMethod(req: RequestFrame): string | null {
case "tool.fill":
case "tool.press":
case "tool.select":
case "tool.upload":
case "tool.download":
case "tool.evaluate":
case "tool.observe":
case "tool.request_help":
@@ -0,0 +1,364 @@
// Order-independent coordinator for one browser download. CDP supplies the
// exact target/frame intent while chrome.downloads supplies the download id
// and filename routing hook; neither event is assumed to arrive first.
import type { CdpTarget } from "@/browser-driver/frame-graph";
import type { ClickResult, RpcError, TransferEffectState } from "@/transport/types";
import { transferError } from "./errors";
import { type CdpRunner, isRpcError } from "./shared";
const CORRELATION_GRACE_MS = 750;
const UNIQUE_SETTLE_MS = 50;
const SIZE_POLL_MS = 250;
type DeterminingFilenameListener = (
item: chrome.downloads.DownloadItem,
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void,
) => void | true;
interface ListenerEvent<T> {
addListener(listener: T): void;
removeListener(listener: T): void;
}
export interface DownloadsApi {
onCreated: ListenerEvent<(item: chrome.downloads.DownloadItem) => void>;
onChanged: ListenerEvent<(delta: chrome.downloads.DownloadDelta) => void>;
onDeterminingFilename: ListenerEvent<DeterminingFilenameListener>;
search(query: chrome.downloads.DownloadQuery): Promise<chrome.downloads.DownloadItem[]>;
cancel(downloadId: number): Promise<void>;
removeFile(downloadId: number): Promise<void>;
}
export const chromeDownloadsApi: DownloadsApi = {
get onCreated() {
return chrome.downloads.onCreated;
},
get onChanged() {
return chrome.downloads.onChanged;
},
get onDeterminingFilename() {
return chrome.downloads.onDeterminingFilename;
},
search: (query) => chrome.downloads.search(query),
cancel: (id) => chrome.downloads.cancel(id),
removeFile: (id) => chrome.downloads.removeFile(id),
};
export interface DownloadCaptureOptions {
cdp: CdpRunner;
target: CdpTarget;
expectedFrameId?: string;
downloads: DownloadsApi;
browserRelativeDir: string;
maxByteSize?: number;
timeoutMs: number;
signal?: AbortSignal;
trigger(): Promise<ClickResult | RpcError>;
}
export interface DownloadCaptureResult {
click: ClickResult;
item: chrome.downloads.DownloadItem;
}
interface DownloadIntent {
url: string;
suggestedFilename: string;
frameId?: string;
}
interface DownloadCandidate {
item: chrome.downloads.DownloadItem;
suggest: (suggestion?: chrome.downloads.DownloadFilenameSuggestion) => void;
suggested: boolean;
graceTimer: ReturnType<typeof setTimeout>;
}
function safeBasename(filename: string): string {
const basename = filename.split(/[\\/]/).pop()?.trim();
return basename && basename !== "." && basename !== ".." ? basename : "download";
}
function sameTarget(source: { tabId?: number; sessionId?: string }, target: CdpTarget): boolean {
return source.tabId === target.tabId && source.sessionId === target.sessionId;
}
function matchesIntent(item: chrome.downloads.DownloadItem, intent: DownloadIntent): boolean {
const urlMatches = item.url === intent.url || item.finalUrl === intent.url;
return urlMatches && safeBasename(item.filename) === safeBasename(intent.suggestedFilename);
}
function knownSize(item: chrome.downloads.DownloadItem): number | undefined {
if (item.fileSize >= 0) return item.fileSize;
if (item.totalBytes >= 0) return item.totalBytes;
return undefined;
}
function captureError(
message: string,
effectState: TransferEffectState,
phase: string,
cleanupFailed = false,
): RpcError {
return transferError("cdp_failed", "download_capture_failed", message, {
effectState,
phase,
...(cleanupFailed ? { cleanupState: "failed" } : {}),
});
}
async function cleanupClaimedDownload(downloads: DownloadsApi, downloadId: number): Promise<void> {
const lookup = async () => (await downloads.search({ id: downloadId }))[0];
const item = await lookup();
if (!item || item.state === "interrupted") return;
if (item.state === "complete") {
await downloads.removeFile(downloadId);
return;
}
try {
await downloads.cancel(downloadId);
} catch (cancelError) {
// Completion can win the race after the lookup but before cancellation.
// Reconcile against Chrome's authoritative state before declaring cleanup
// failed so every terminal state has one explicit cleanup path.
const reconciled = await lookup();
if (!reconciled || reconciled.state === "interrupted") return;
if (reconciled.state === "complete") {
await downloads.removeFile(downloadId);
return;
}
throw cancelError;
}
}
export async function captureBrowserDownload(
options: DownloadCaptureOptions,
): Promise<DownloadCaptureResult | RpcError> {
let click: ClickResult | undefined;
let intent: DownloadIntent | undefined;
let capturedId: number | undefined;
let settled = false;
let succeeded = false;
let failureResult: RpcError | undefined;
let uniquenessTimer: ReturnType<typeof setTimeout> | undefined;
let operationTimer: ReturnType<typeof setTimeout> | undefined;
let sizePoll: ReturnType<typeof setInterval> | undefined;
const candidates = new Map<number, DownloadCandidate>();
const createdItems = new Map<number, chrome.downloads.DownloadItem>();
let resolveCompletion!: (item: chrome.downloads.DownloadItem) => void;
let rejectCompletion!: (error: Error) => void;
const completion = new Promise<chrome.downloads.DownloadItem>((resolve, reject) => {
resolveCompletion = resolve;
rejectCompletion = reject;
});
const fail = (error: Error) => {
if (settled) return;
settled = true;
rejectCompletion(error);
};
const complete = (item: chrome.downloads.DownloadItem) => {
if (settled) return;
const size = knownSize(item);
if (size !== undefined && options.maxByteSize !== undefined && size > options.maxByteSize) {
fail(new Error(`download exceeds transfer limit ${options.maxByteSize}`));
return;
}
settled = true;
resolveCompletion(item);
};
const suggestDefault = (candidate: DownloadCandidate) => {
if (candidate.suggested) return;
candidate.suggested = true;
candidate.suggest();
};
const matchingCandidates = (): DownloadCandidate[] => {
const currentIntent = intent;
return currentIntent
? [...candidates.values()].filter(
(candidate) => !candidate.suggested && matchesIntent(candidate.item, currentIntent),
)
: [];
};
const claimUnique = () => {
uniquenessTimer = undefined;
if (settled || capturedId !== undefined || !intent) return;
const matches = matchingCandidates();
if (matches.length !== 1) {
if (matches.length > 1) {
for (const candidate of matches) suggestDefault(candidate);
fail(new Error("download attribution is ambiguous"));
}
return;
}
const candidate = matches[0];
candidate.suggested = true;
clearTimeout(candidate.graceTimer);
capturedId = candidate.item.id;
candidate.suggest({
filename: `${options.browserRelativeDir}/${safeBasename(intent.suggestedFilename)}`,
conflictAction: "overwrite",
});
const size = knownSize(candidate.item);
if (size !== undefined && options.maxByteSize !== undefined && size > options.maxByteSize) {
fail(new Error(`download exceeds transfer limit ${options.maxByteSize}`));
return;
}
const created = createdItems.get(candidate.item.id);
if (created?.state === "interrupted") {
fail(new Error(created.error ?? "download interrupted"));
} else if (created?.state === "complete") {
complete(created);
}
};
const reconcile = () => {
if (settled || capturedId !== undefined || !intent) return;
const matches = matchingCandidates();
if (matches.length > 1) {
for (const candidate of matches) suggestDefault(candidate);
fail(new Error("download attribution is ambiguous"));
return;
}
if (matches.length === 1 && !uniquenessTimer) {
uniquenessTimer = setTimeout(claimUnique, UNIQUE_SETTLE_MS);
}
};
const determiningListener: DeterminingFilenameListener = (item, suggest) => {
const candidate: DownloadCandidate = {
item,
suggest,
suggested: false,
graceTimer: setTimeout(() => {
suggestDefault(candidate);
candidates.delete(item.id);
if (intent && matchesIntent(item, intent) && capturedId === undefined) {
fail(new Error("download correlation grace elapsed before unique attribution"));
}
}, CORRELATION_GRACE_MS),
};
candidates.set(item.id, candidate);
reconcile();
return true;
};
const createdListener = (item: chrome.downloads.DownloadItem) => {
createdItems.set(item.id, item);
if (capturedId !== item.id) return;
if (item.state === "interrupted") {
fail(new Error(item.error ?? "download interrupted"));
} else if (item.state === "complete") {
complete(item);
}
};
const changedListener = async (delta: chrome.downloads.DownloadDelta) => {
if (capturedId === undefined || delta.id !== capturedId || settled) return;
if (delta.state?.current === "interrupted" || delta.error?.current) {
fail(new Error(delta.error?.current ?? "download interrupted"));
return;
}
if (delta.state?.current === "complete") {
try {
const [item] = await options.downloads.search({ id: delta.id });
if (item) complete(item);
else fail(new Error("completed download disappeared"));
} catch (err) {
fail(err instanceof Error ? err : new Error(String(err)));
}
}
};
const onAbort = () => fail(new DOMException("aborted", "AbortError"));
const cdpSubscription = options.cdp.onEvent?.((source, method, raw) => {
if (method !== "Page.downloadWillBegin" || !sameTarget(source, options.target)) return;
const event = raw as { url?: unknown; suggestedFilename?: unknown; frameId?: unknown };
if (typeof event.url !== "string" || typeof event.suggestedFilename !== "string") return;
if (options.expectedFrameId && event.frameId !== options.expectedFrameId) {
fail(new Error("download originated from a different frame"));
return;
}
if (intent) {
fail(new Error("download trigger produced more than one browser download intent"));
return;
}
intent = {
url: event.url,
suggestedFilename: event.suggestedFilename,
...(typeof event.frameId === "string" ? { frameId: event.frameId } : {}),
};
reconcile();
});
if (!cdpSubscription) {
return captureError("CDP download intent subscription unavailable", "none", "arm");
}
options.downloads.onDeterminingFilename.addListener(determiningListener);
options.downloads.onCreated.addListener(createdListener);
options.downloads.onChanged.addListener(changedListener);
options.signal?.addEventListener("abort", onAbort, { once: true });
operationTimer = setTimeout(
() => fail(new Error("download did not complete before timeout")),
options.timeoutMs,
);
sizePoll = setInterval(() => {
if (capturedId === undefined || settled || options.maxByteSize === undefined) return;
void options.downloads
.search({ id: capturedId })
.then(([item]) => {
if (!item || settled) return;
if (item.bytesReceived > (options.maxByteSize as number)) {
fail(new Error(`download exceeds transfer limit ${options.maxByteSize}`));
}
})
.catch((err) => fail(err instanceof Error ? err : new Error(String(err))));
}, SIZE_POLL_MS);
try {
const triggered = await options.trigger();
if (isRpcError(triggered)) {
void completion.catch(() => undefined);
const effect: TransferEffectState =
capturedId !== undefined ? "committed" : intent ? "unknown" : "none";
failureResult = {
...triggered,
data: { ...triggered.data, effect_state: effect, phase: "trigger" },
};
return failureResult;
}
click = triggered;
const item = await completion;
succeeded = true;
return { click, item };
} catch (err) {
const effect: TransferEffectState =
capturedId !== undefined ? "committed" : click ? "unknown" : "none";
failureResult = captureError(
err instanceof Error ? err.message : String(err),
effect,
capturedId !== undefined ? "download" : "attribution",
);
return failureResult;
} finally {
settled = true;
if (operationTimer) clearTimeout(operationTimer);
if (uniquenessTimer) clearTimeout(uniquenessTimer);
if (sizePoll) clearInterval(sizePoll);
options.signal?.removeEventListener("abort", onAbort);
options.downloads.onDeterminingFilename.removeListener(determiningListener);
options.downloads.onCreated.removeListener(createdListener);
options.downloads.onChanged.removeListener(changedListener);
cdpSubscription.dispose();
for (const candidate of candidates.values()) {
clearTimeout(candidate.graceTimer);
if (candidate.item.id !== capturedId) suggestDefault(candidate);
}
if (!succeeded && capturedId !== undefined) {
try {
await cleanupClaimedDownload(options.downloads, capturedId);
} catch {
if (failureResult?.data) failureResult.data.cleanup_state = "failed";
}
}
}
}
+64
View File
@@ -0,0 +1,64 @@
// Download orchestration: validate session/tab ownership, then delegate one
// browser-global chrome.downloads transaction to download-capture.ts.
import type { SessionManager } from "@/session-manager/manager";
import type { DownloadParams, DownloadResult, RpcError } from "@/transport/types";
import { captureBrowserDownload, chromeDownloadsApi, type DownloadsApi } from "./download-capture";
import { clickResolvedTarget, type InteractionDeps, resolveActionTarget } from "./interaction";
import { enforceAgentWindow, isRpcError, lookupSession, resolveTargetTab } from "./shared";
let downloadActive = false;
export type { DownloadsApi } from "./download-capture";
export interface DownloadDeps extends InteractionDeps {
downloads?: DownloadsApi;
}
export async function handleDownload(
manager: SessionManager,
params: DownloadParams,
deps: DownloadDeps,
): Promise<DownloadResult | RpcError> {
if (downloadActive) return { code: "invalid_params", message: "another bsk download is active" };
downloadActive = true;
try {
const ctx = lookupSession(manager, params, "download");
if (isRpcError(ctx)) return ctx;
const target = await resolveTargetTab(manager, ctx, params.tab_id, deps.tabsApi);
if (isRpcError(target)) return target;
const denied = enforceAgentWindow(ctx, target, "download");
if (denied) return denied;
if (!params.browser_relative_dir) {
return { code: "invalid_params", message: "download requires a daemon capability directory" };
}
const address = await resolveActionTarget(deps.cdp, ctx, target, params, "download");
if (isRpcError(address)) return address;
const capture = await captureBrowserDownload({
cdp: deps.cdp,
target: address.cdpTarget,
downloads: deps.downloads ?? chromeDownloadsApi,
browserRelativeDir: params.browser_relative_dir,
maxByteSize: params.max_byte_size,
timeoutMs: params.timeout_ms ?? 120_000,
signal: deps.signal,
expectedFrameId: address.frameId,
trigger: () => clickResolvedTarget(ctx, address, {}, deps),
});
if (isRpcError(capture)) return capture;
const { click, item } = capture;
return {
tab_id: target.tabId,
used_ref: click.used_ref,
used_selector: click.used_selector,
suggested_filename: item.filename.split(/[\\/]/).pop() ?? "download",
byte_size: item.fileSize >= 0 ? item.fileSize : item.totalBytes,
mime: item.mime || undefined,
danger: item.danger,
browser_path: item.filename,
};
} finally {
downloadActive = false;
}
}
+27 -1
View File
@@ -2,10 +2,23 @@
// rendering. Extension handlers attach reasons here; human-facing copy
// lives in bsk-cli `render_error.rs`.
import type { ErrorCode, RpcError, RpcErrorData, RpcErrorReason } from "@/transport/types";
import type {
ErrorCode,
RpcError,
RpcErrorData,
RpcErrorReason,
TransferCleanupState,
TransferEffectState,
} from "@/transport/types";
export type { RpcErrorData, RpcErrorReason };
export interface TransferErrorOptions {
effectState: TransferEffectState;
phase: string;
cleanupState?: TransferCleanupState;
}
export function rpcError(
code: ErrorCode,
reason: RpcErrorReason,
@@ -15,3 +28,16 @@ export function rpcError(
const data: RpcErrorData = { reason, ...extra };
return { code, message, data };
}
export function transferError(
code: ErrorCode,
reason: RpcErrorReason,
message: string,
options: TransferErrorOptions,
): RpcError {
return rpcError(code, reason, message, {
effect_state: options.effectState,
phase: options.phase,
...(options.cleanupState ? { cleanup_state: options.cleanupState } : {}),
});
}
@@ -0,0 +1,319 @@
// One native file-drop transaction bound to an explicit page target. CDP drag
// events are sent to the target that owns the node: root targets use top-level
// coordinates, while OOPIF targets require coordinates local to their session.
import type { RpcError, TransferEffectState } from "@/transport/types";
import { transferError } from "./errors";
import { resolveNodeGeometry } from "./frame-geometry";
import type { ResolvedActionTarget } from "./interaction";
import { type CdpRunner, isRpcError, sendToCdpTarget } from "./shared";
import { BoundedWaitError, waitBounded } from "./transfer-transaction";
const CLEANUP_TIMEOUT_MS = 1_000;
type FileDropPhase = "resolve_target" | "drag_enter" | "drag_over" | "drop" | "cleanup";
interface RuntimeReply {
result?: { value?: unknown; objectId?: string };
exceptionDetails?: { text?: string; exception?: { description?: string } };
}
interface DragData {
items: [];
files: string[];
dragOperationsMask: number;
}
export interface FileDropTransactionOptions {
cdp: CdpRunner;
actionTarget: ResolvedActionTarget;
files: string[];
timeoutMs: number;
signal?: AbortSignal;
withOverlayHidden: <T>(operation: () => Promise<T>) => Promise<T>;
}
export interface FileDropTransactionResult {
x: number;
y: number;
}
function failure(
code: RpcError["code"],
reason: "upload_mechanism_unsupported" | "file_drop_target_unavailable" | "file_drop_failed",
message: string,
effectState: TransferEffectState,
phase: FileDropPhase,
): RpcError {
const error = transferError(code, reason, message, { effectState, phase });
error.data = { ...error.data, mechanism: "drop" };
return error;
}
function enrichFailure(
error: RpcError,
effectState: TransferEffectState,
phase: FileDropPhase,
): RpcError {
return {
...error,
data: {
...error.data,
mechanism: "drop",
effect_state: effectState,
phase,
},
};
}
function messageOf(error: unknown): string {
return error instanceof Error ? error.message : String(error);
}
function methodUnavailable(error: unknown): boolean {
return /(?:method|command).*(?:not found|wasn't found|unsupported)|wasn't found|-32601/i.test(
messageOf(error),
);
}
function runtimeError(reply: RuntimeReply, fallback: string): Error | null {
if (!reply.exceptionDetails) return null;
return new Error(
reply.exceptionDetails.exception?.description ?? reply.exceptionDetails.text ?? fallback,
);
}
async function pointBelongsToTarget(
options: FileDropTransactionOptions,
x: number,
y: number,
objectGroup: string,
): Promise<boolean> {
const target = options.actionTarget.cdpTarget;
const hit = await sendToCdpTarget<{ backendNodeId?: number; frameId?: string }>(
options.cdp,
target,
"DOM.getNodeForLocation",
{ x: Math.round(x), y: Math.round(y), includeUserAgentShadowDOM: true },
);
if (typeof hit.backendNodeId !== "number") return false;
if (options.actionTarget.frameId && hit.frameId !== options.actionTarget.frameId) return false;
const [resolvedTarget, resolvedHit] = await Promise.all([
sendToCdpTarget<{ object?: { objectId?: string } }>(options.cdp, target, "DOM.resolveNode", {
backendNodeId: options.actionTarget.backendNodeId,
objectGroup,
}),
sendToCdpTarget<{ object?: { objectId?: string } }>(options.cdp, target, "DOM.resolveNode", {
backendNodeId: hit.backendNodeId,
objectGroup,
}),
]);
const targetObjectId = resolvedTarget.object?.objectId;
const hitObjectId = resolvedHit.object?.objectId;
if (!targetObjectId || !hitObjectId) return false;
const reply = await sendToCdpTarget<RuntimeReply>(options.cdp, target, "Runtime.callFunctionOn", {
objectId: targetObjectId,
functionDeclaration: `function(hit) {
for (let node = hit; node; ) {
if (node === this) return true;
if (this.contains?.(node)) return true;
const root = node.getRootNode?.();
node = node.parentNode || (root && root !== node ? root.host : null);
}
return false;
}`,
arguments: [{ objectId: hitObjectId }],
returnByValue: true,
});
const error = runtimeError(reply, "failed to verify file-drop target");
if (error) throw error;
return reply.result?.value === true;
}
export async function uploadThroughFileDrop(
options: FileDropTransactionOptions,
): Promise<FileDropTransactionResult | RpcError> {
const deadline = Date.now() + options.timeoutMs;
const target = options.actionTarget.cdpTarget;
const objectGroup = `bsk-file-drop-${crypto.randomUUID()}`;
const dragData: DragData = {
items: [],
files: options.files,
dragOperationsMask: 1,
};
let dragStarted = false;
let dropSent = false;
let dragPoint = { x: 0, y: 0 };
let cleanupFailed = false;
let outcome: FileDropTransactionResult | RpcError = failure(
"protocol_error",
"file_drop_failed",
"file-drop transaction ended without an outcome",
"none",
"cleanup",
);
try {
const geometry = await waitBounded(
resolveNodeGeometry(
options.cdp,
options.actionTarget.tab.tabId,
{
target,
backendNodeId: options.actionTarget.backendNodeId,
...(options.actionTarget.frameId ? { frameId: options.actionTarget.frameId } : {}),
},
{ scrollIntoView: true },
),
deadline,
options.signal,
"resolving file-drop target timed out",
);
if (isRpcError(geometry)) {
outcome = enrichFailure(geometry, "none", "resolve_target");
return outcome;
}
const point = geometry.targetActionPoint;
dragPoint = point;
outcome = await options.withOverlayHidden(async () => {
try {
try {
const belongs = await waitBounded(
pointBelongsToTarget(options, point.x, point.y, objectGroup),
deadline,
options.signal,
"verifying file-drop target timed out",
);
if (!belongs) {
return failure(
"permission_denied",
"file_drop_target_unavailable",
"file-drop target is not the topmost element at its action point",
"none",
"resolve_target",
);
}
} catch (error) {
return failure(
error instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"file_drop_target_unavailable",
messageOf(error),
"none",
"resolve_target",
);
}
for (const [type, phase] of [
["dragEnter", "drag_enter"],
["dragOver", "drag_over"],
] as const) {
try {
if (type === "dragEnter") dragStarted = true;
await waitBounded(
sendToCdpTarget(options.cdp, target, "Input.dispatchDragEvent", {
type,
x: point.x,
y: point.y,
data: dragData,
}),
deadline,
options.signal,
`${type} timed out`,
);
} catch (error) {
const unsupported = methodUnavailable(error);
if (type === "dragEnter" && unsupported) dragStarted = false;
return failure(
unsupported
? "unsupported"
: error instanceof BoundedWaitError
? "timeout"
: "cdp_failed",
unsupported ? "upload_mechanism_unsupported" : "file_drop_failed",
messageOf(error),
"none",
phase,
);
}
}
try {
dropSent = true;
await waitBounded(
sendToCdpTarget(options.cdp, target, "Input.dispatchDragEvent", {
type: "drop",
x: point.x,
y: point.y,
data: dragData,
}),
deadline,
options.signal,
"file drop timed out",
);
return { x: point.x, y: point.y };
} catch (error) {
outcome = failure(
error instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"file_drop_failed",
messageOf(error),
"unknown",
"drop",
);
return outcome;
}
} finally {
if (dragStarted && !dropSent) {
try {
await waitBounded(
sendToCdpTarget(options.cdp, target, "Input.dispatchDragEvent", {
type: "dragCancel",
x: dragPoint.x,
y: dragPoint.y,
data: dragData,
}),
Date.now() + CLEANUP_TIMEOUT_MS,
undefined,
"cancelling file drag timed out",
);
} catch {
cleanupFailed = true;
}
}
}
});
return outcome;
} catch (error) {
outcome = failure(
error instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"file_drop_target_unavailable",
messageOf(error),
"none",
"resolve_target",
);
return outcome;
} finally {
try {
await waitBounded(
sendToCdpTarget(options.cdp, target, "Runtime.releaseObjectGroup", { objectGroup }),
Date.now() + CLEANUP_TIMEOUT_MS,
undefined,
"releasing file-drop object group timed out",
);
} catch {
cleanupFailed = true;
}
const effect = isRpcError(outcome)
? (outcome.data?.effect_state as TransferEffectState | undefined)
: "committed";
if (effect === "unknown" || cleanupFailed) {
await options.cdp.detach?.(target.tabId);
}
if (cleanupFailed && isRpcError(outcome)) {
outcome.data = { ...outcome.data, cleanup_state: "failed" };
}
}
}
@@ -0,0 +1,445 @@
// One upload transaction with an explicit browser-side commit boundary.
// Chrome interception prevents a native chooser from escaping automation;
// chooser events and a frame-scoped DOM probe are independent input-location
// signals, so event delivery is not required for standard file inputs.
import type { ClickResult, RpcError, TransferEffectState } from "@/transport/types";
import { transferError } from "./errors";
import type { ResolvedActionTarget } from "./interaction";
import { type CdpRunner, isRpcError, sendToCdpTarget } from "./shared";
import { BoundedWaitError, remainingMs, waitBounded } from "./transfer-transaction";
const CLEANUP_TIMEOUT_MS = 1_000;
const ACTIVATION_GRACE_MS = 1_000;
const PROBE_INTERVAL_MS = 20;
type UploadPhase =
| "arm_interception"
| "arm_input_probe"
| "trigger"
| "resolve_input"
| "set_files"
| "cleanup";
interface RuntimeReply {
result?: { value?: unknown; objectId?: string };
exceptionDetails?: { text?: string; exception?: { description?: string } };
}
interface ChooserEvent {
frameId?: string;
backendNodeId?: number;
mode?: "selectSingle" | "selectMultiple";
}
export interface FileInputTransactionOptions {
cdp: CdpRunner;
actionTarget: ResolvedActionTarget;
files: string[];
timeoutMs: number;
signal?: AbortSignal;
trigger(): Promise<ClickResult | RpcError>;
}
export interface FileInputTransactionResult {
click: ClickResult;
multiple: boolean;
}
function runtimeError(reply: RuntimeReply, fallback: string): Error | null {
if (!reply.exceptionDetails) return null;
return new Error(
reply.exceptionDetails.exception?.description ?? reply.exceptionDetails.text ?? fallback,
);
}
function transferFailure(
code: RpcError["code"],
reason: "file_input_probe_failed" | "file_input_not_activated" | "set_file_input_failed",
message: string,
effectState: TransferEffectState,
phase: UploadPhase,
): RpcError {
return transferError(code, reason, message, { effectState, phase });
}
function enrichFailure(
error: RpcError,
effectState: TransferEffectState,
phase: UploadPhase,
): RpcError {
return {
...error,
data: { ...error.data, effect_state: effectState, phase },
};
}
function sameTarget(
source: { tabId?: number; sessionId?: string },
target: { tabId: number; sessionId?: string },
): boolean {
return source.tabId === target.tabId && source.sessionId === target.sessionId;
}
async function delay(ms: number, signal?: AbortSignal): Promise<void> {
await waitBounded(
new Promise((resolve) => setTimeout(resolve, ms)),
Date.now() + ms + 1,
signal,
"upload activation probe timed out",
);
}
async function callOnTrigger<T = RuntimeReply>(
options: FileInputTransactionOptions,
objectId: string,
functionDeclaration: string,
args: unknown[],
returnByValue: boolean,
): Promise<T> {
return sendToCdpTarget<T>(options.cdp, options.actionTarget.cdpTarget, "Runtime.callFunctionOn", {
objectId,
functionDeclaration,
arguments: args.map((value) => ({ value })),
returnByValue,
awaitPromise: false,
});
}
export async function uploadThroughActivatedFileInput(
options: FileInputTransactionOptions,
): Promise<FileInputTransactionResult | RpcError> {
const deadline = Date.now() + options.timeoutMs;
const target = options.actionTarget.cdpTarget;
const objectGroup = `bsk-upload-${crypto.randomUUID()}`;
const stateKey = `__bskUpload_${crypto.randomUUID().replaceAll("-", "")}`;
const chooserEvents: ChooserEvent[] = [];
let interceptionArmed = false;
let probeArmed = false;
let triggerObjectId: string | undefined;
let outcome: FileInputTransactionResult | RpcError = transferFailure(
"protocol_error",
"file_input_probe_failed",
"upload transaction ended without an outcome",
"none",
"cleanup",
);
const chooserSubscription = options.cdp.onEvent?.((source, method, raw) => {
if (method !== "Page.fileChooserOpened" || !sameTarget(source, target)) return;
const event = raw as ChooserEvent;
chooserEvents.push(event);
});
try {
try {
await waitBounded(
sendToCdpTarget(options.cdp, target, "Page.setInterceptFileChooserDialog", {
enabled: true,
}),
deadline,
options.signal,
"arming native file chooser interception timed out",
);
interceptionArmed = true;
} catch (err) {
outcome = transferFailure(
err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"file_input_probe_failed",
err instanceof Error ? err.message : String(err),
"none",
"arm_interception",
);
return outcome;
}
try {
const resolved = await waitBounded(
sendToCdpTarget<{ object?: { objectId?: string } }>(
options.cdp,
target,
"DOM.resolveNode",
{
backendNodeId: options.actionTarget.backendNodeId,
objectGroup,
},
),
deadline,
options.signal,
"resolving upload trigger timed out",
);
triggerObjectId = resolved.object?.objectId;
if (!triggerObjectId) throw new Error("DOM.resolveNode returned no trigger objectId");
const armed = await waitBounded(
callOnTrigger<RuntimeReply>(
options,
triggerObjectId,
`function(key) {
const doc = this.ownerDocument;
const owner = doc.defaultView;
if (!owner) return false;
const state = { inputs: [], listener: null };
Object.defineProperty(owner, key, { value: state, configurable: true });
state.listener = event => {
const path = typeof event.composedPath === "function" ? event.composedPath() : [];
const candidate = path[0] || event.target;
if (candidate && candidate.nodeType === 1 &&
candidate.localName === "input" && candidate.type === "file") {
if (!state.inputs.includes(candidate)) state.inputs.push(candidate);
event.preventDefault();
}
};
doc.addEventListener("click", state.listener, true);
return true;
}`,
[stateKey],
true,
),
deadline,
options.signal,
"arming frame-scoped file input probe timed out",
);
const armError = runtimeError(armed, "failed to arm file input probe");
if (armError || armed.result?.value !== true) {
throw armError ?? new Error("file input probe did not arm");
}
probeArmed = true;
} catch (err) {
outcome = transferFailure(
err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"file_input_probe_failed",
err instanceof Error ? err.message : String(err),
"none",
"arm_input_probe",
);
return outcome;
}
let click: ClickResult | RpcError;
try {
click = await waitBounded(
options.trigger(),
deadline,
options.signal,
"upload trigger timed out",
);
} catch (err) {
outcome = transferFailure(
err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"file_input_probe_failed",
err instanceof Error ? err.message : String(err),
"none",
"trigger",
);
return outcome;
}
if (isRpcError(click)) {
outcome = enrichFailure(click, "none", "trigger");
return outcome;
}
try {
const activationDeadline = Math.min(deadline, Date.now() + ACTIVATION_GRACE_MS);
let summary: { count: number; multiple: boolean } = { count: 0, multiple: false };
while (Date.now() < activationDeadline) {
if (chooserEvents.length > 0) break;
const reply = await callOnTrigger<RuntimeReply>(
options,
triggerObjectId,
`function(key) {
const state = this.ownerDocument.defaultView?.[key];
return state
? { count: state.inputs.length, multiple: state.inputs[0]?.multiple === true }
: { count: 0, multiple: false };
}`,
[stateKey],
true,
);
const summaryError = runtimeError(reply, "failed to inspect activated file input");
if (summaryError) throw summaryError;
const value = reply.result?.value as { count?: unknown; multiple?: unknown } | undefined;
summary = {
count: typeof value?.count === "number" ? value.count : 0,
multiple: value?.multiple === true,
};
if (summary.count > 0) break;
await delay(Math.min(PROBE_INTERVAL_MS, remainingMs(activationDeadline)), options.signal);
}
if (chooserEvents.length > 1) {
outcome = transferFailure(
"unsupported",
"file_input_not_activated",
"upload trigger activated more than one file chooser",
"none",
"resolve_input",
);
return outcome;
}
const chooser = chooserEvents[0];
let backendNodeId: number | undefined;
let multiple = summary.multiple;
if (chooser) {
if (options.actionTarget.frameId && chooser.frameId !== options.actionTarget.frameId) {
outcome = transferFailure(
"unsupported",
"file_input_not_activated",
"upload trigger activated a file chooser in a different frame",
"none",
"resolve_input",
);
return outcome;
}
if (typeof chooser.backendNodeId !== "number") {
outcome = transferFailure(
"unsupported",
"file_input_not_activated",
"upload trigger invoked a non-input file picker",
"none",
"resolve_input",
);
return outcome;
}
backendNodeId = chooser.backendNodeId;
multiple = chooser.mode === "selectMultiple";
} else {
if (summary.count !== 1) {
outcome = transferFailure(
"unsupported",
"file_input_not_activated",
summary.count === 0
? "upload trigger did not activate an input[type=file]"
: "upload trigger activated more than one input[type=file]",
"none",
"resolve_input",
);
return outcome;
}
const input = await callOnTrigger<RuntimeReply>(
options,
triggerObjectId,
`function(key) { return this.ownerDocument.defaultView?.[key]?.inputs[0]; }`,
[stateKey],
false,
);
const inputError = runtimeError(input, "failed to resolve activated file input object");
if (inputError) throw inputError;
if (!input.result?.objectId) throw new Error("activated file input returned no objectId");
const described = await sendToCdpTarget<{ node?: { backendNodeId?: number } }>(
options.cdp,
target,
"DOM.describeNode",
{ objectId: input.result.objectId },
);
backendNodeId = described.node?.backendNodeId;
if (typeof backendNodeId !== "number") {
throw new Error("DOM.describeNode returned no file input backendNodeId");
}
}
if (!multiple && options.files.length !== 1) {
outcome = enrichFailure(
{ code: "invalid_params", message: "file input accepts exactly one file" },
"none",
"resolve_input",
);
return outcome;
}
try {
await waitBounded(
sendToCdpTarget(options.cdp, target, "DOM.setFileInputFiles", {
files: options.files,
backendNodeId,
}),
deadline,
options.signal,
"setting file input files timed out",
);
} catch (err) {
outcome = transferFailure(
err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"set_file_input_failed",
err instanceof Error ? err.message : String(err),
"unknown",
"set_files",
);
return outcome;
}
outcome = { click, multiple };
return outcome;
} catch (err) {
outcome = transferFailure(
err instanceof BoundedWaitError ? "timeout" : "cdp_failed",
"file_input_probe_failed",
err instanceof Error ? err.message : String(err),
"none",
"resolve_input",
);
return outcome;
}
} finally {
chooserSubscription?.dispose();
let cleanupFailed = false;
if (probeArmed && triggerObjectId) {
try {
await waitBounded(
callOnTrigger(
options,
triggerObjectId,
`function(key) {
const owner = this.ownerDocument.defaultView;
const state = owner?.[key];
if (state?.listener) this.ownerDocument.removeEventListener("click", state.listener, true);
if (owner) delete owner[key];
}`,
[stateKey],
true,
),
Date.now() + CLEANUP_TIMEOUT_MS,
undefined,
"cleaning file input probe timed out",
);
} catch {
cleanupFailed = true;
}
}
if (interceptionArmed) {
try {
await waitBounded(
sendToCdpTarget(options.cdp, target, "Page.setInterceptFileChooserDialog", {
enabled: false,
cancel: true,
}),
Date.now() + CLEANUP_TIMEOUT_MS,
undefined,
"disabling file chooser interception timed out",
);
} catch {
cleanupFailed = true;
}
}
try {
await waitBounded(
sendToCdpTarget(options.cdp, target, "Runtime.releaseObjectGroup", { objectGroup }),
Date.now() + CLEANUP_TIMEOUT_MS,
undefined,
"releasing upload object group timed out",
);
} catch {
cleanupFailed = true;
}
const effect = isRpcError(outcome)
? (outcome.data?.effect_state as TransferEffectState | undefined)
: "committed";
if (effect === "unknown" || cleanupFailed) {
await options.cdp.detach?.(target.tabId);
}
if (cleanupFailed && isRpcError(outcome)) {
outcome.data = { ...outcome.data, cleanup_state: "failed" };
}
}
}
+27 -1
View File
@@ -29,7 +29,10 @@ export interface NodeAddress {
export interface ResolvedNodeGeometry {
topVisibleRegions: Region;
topBounds: ViewportRect;
/** Point in the top-level tab viewport, used by root-target input events. */
actionPoint: Point;
/** Point in the addressed CDP target's viewport, used by OOPIF-local input events. */
targetActionPoint: Point;
}
function geometryError(message: string): RpcError {
@@ -262,11 +265,30 @@ export async function resolveNodeGeometry(
if (isRpcError(localRegion)) return localRegion;
let topVisibleRegions: Region;
let targetActionPoint: Point | null = null;
if (address.frameId && graph) {
const projection = await resolveFrameProjection(cdp, graph, address.frameId);
if (!projection)
return geometryError(`could not resolve frame geometry for ${address.frameId}`);
topVisibleRegions = projectRegionToViewport(localRegion, projection);
if (address.target.sessionId) {
const localViewport = projection.edges[0]?.sourceViewport ?? projection.topViewport;
const localVisibleRegions = localRegion
.map((polygon) =>
clipPolygon(
polygon,
rectPolygon({
x: 0,
y: 0,
w: localViewport.width,
h: localViewport.height,
}),
),
)
.filter((polygon) => polygon.length >= 3);
const localActionRegion = largestRegion(localVisibleRegions);
targetActionPoint = localActionRegion ? polygonCentroid(localActionRegion) : null;
}
} else {
const viewport = await targetViewport(cdp, address.target);
if (!viewport) return geometryError("could not resolve top viewport geometry");
@@ -283,7 +305,11 @@ export async function resolveNodeGeometry(
if (!topBounds || !actionPoint) {
return { code: "permission_denied", message: "element not visible" };
}
return { topVisibleRegions, topBounds, actionPoint };
if (address.target.sessionId && !targetActionPoint) {
return { code: "permission_denied", message: "element not visible in its target" };
}
targetActionPoint ??= actionPoint;
return { topVisibleRegions, topBounds, actionPoint, targetActionPoint };
} catch (error) {
return geometryError(error instanceof Error ? error.message : String(error));
}
+39 -9
View File
@@ -40,6 +40,7 @@ import {
enforceAgentWindow,
isRpcError,
lookupSession,
type ResolvedTargetTab,
resolveTargetTab,
} from "./shared";
import { resolveSnapshotRef } from "./snapshot-ref";
@@ -56,6 +57,15 @@ export interface InteractionDeps {
keepOverlayBypassAfterHover?: boolean;
}
export interface ResolvedActionTarget {
tab: ResolvedTargetTab;
backendNodeId: number;
cdpTarget: CdpTarget;
frameId?: string;
usedRef?: string;
usedSelector?: string;
}
const DEFAULT_TIMEOUT_MS = 30_000;
const DEFAULT_HOVER_SETTLE_MS = 200;
@@ -128,7 +138,7 @@ async function wait(ms: number, signal?: AbortSignal): Promise<void> {
* `RpcError` if the caller supplied neither (or both), or if neither
* lookup matched.
*/
async function resolveBackendNode(
export async function resolveBackendNode(
cdp: CdpRunner,
ctx: SessionContext,
target: { tabId: number },
@@ -215,6 +225,17 @@ async function resolveBackendNode(
}
}
export async function resolveActionTarget(
cdp: CdpRunner,
ctx: SessionContext,
target: ResolvedTargetTab,
params: { ref?: string; selector?: string },
toolName: string,
): Promise<ResolvedActionTarget | RpcError> {
const node = await resolveBackendNode(cdp, ctx, target, params, toolName);
return isRpcError(node) ? node : { tab: target, ...node };
}
// ---------------------------------------------------------------------------
// tool.click
// ---------------------------------------------------------------------------
@@ -233,10 +254,19 @@ export async function handleClick(
if (isRpcError(target)) return target;
const denied = enforceAgentWindow(ctx, target, "click");
if (denied) return denied;
const dialogCursor = markDialogCursor(deps.cdp, target.tabId);
const resolved = await resolveActionTarget(deps.cdp, ctx, target, params, "click");
if (isRpcError(resolved)) return resolved;
return clickResolvedTarget(ctx, resolved, params, deps);
}
const node = await resolveBackendNode(deps.cdp, ctx, target, params, "click");
if (isRpcError(node)) return node;
export async function clickResolvedTarget(
ctx: SessionContext,
resolved: ResolvedActionTarget,
params: Pick<ClickParams, "button" | "click_count" | "modifiers">,
deps: InteractionDeps,
): Promise<ClickResult | RpcError> {
const { tab: target } = resolved;
const dialogCursor = markDialogCursor(deps.cdp, target.tabId);
if (throwIfAborted(deps.signal)) {
return { code: "cancelled", message: "click aborted" };
@@ -247,9 +277,9 @@ export async function handleClick(
deps.cdp,
target.tabId,
{
target: node.cdpTarget,
backendNodeId: node.backendNodeId,
...(node.frameId ? { frameId: node.frameId } : {}),
target: resolved.cdpTarget,
backendNodeId: resolved.backendNodeId,
...(resolved.frameId ? { frameId: resolved.frameId } : {}),
},
{ scrollIntoView: true },
);
@@ -337,8 +367,8 @@ export async function handleClick(
return attachDialogs(deps.cdp, target.tabId, dialogCursor, {
tab_id: target.tabId,
used_ref: node.usedRef,
used_selector: node.usedSelector,
used_ref: resolved.usedRef,
used_selector: resolved.usedSelector,
x: centre.x,
y: centre.y,
});
+109 -22
View File
@@ -17,7 +17,7 @@ import { ChromiumCdp } from "@/browser-driver/chromium-cdp";
import type { CdpTarget } from "@/browser-driver/frame-graph";
import {
type CaptureSuppressSendToTab,
withOverlaysHiddenForCapture,
withExtensionOverlayHidden,
} from "@/lib/capture-suppress-bridge";
import type { SessionContext, SessionManager } from "@/session-manager/manager";
import type {
@@ -49,11 +49,14 @@ import {
import { resolveSnapshotRef } from "./snapshot-ref";
import {
type CapturedNode,
type CapturedSurfaceProbe,
type CapturedViewModel,
captureViewModel,
collectOverlayExcludedBackendIds,
probeHoverSurfaces,
} from "./vom/capture";
import { type CapturedFrameDocument, captureFrameData } from "./vom/frame-capture";
import { withOverlayBypass } from "./vom/hover-perception";
import { probeTooltipNames } from "./vom/name-enrichment";
import {
type CaptureVomObservationResult,
@@ -321,7 +324,7 @@ export async function handleScreenshot(
tabId: target.tabId,
...(node.cdpSessionId ? { sessionId: node.cdpSessionId } : {}),
};
const captured = await withOverlaysHiddenForCapture(
const captured = await withExtensionOverlayHidden(
target.tabId,
() =>
captureElementScreenshot(
@@ -353,7 +356,7 @@ export async function handleScreenshot(
);
}
const captured = await withOverlaysHiddenForCapture(
const captured = await withExtensionOverlayHidden(
target.tabId,
() => captureFullTabPng(deps, ctx, target, signal),
deps.sendToTab,
@@ -525,8 +528,11 @@ function buildActiveScopeBlocks(nodes: VomNode[], signals: VomNodeDomSignals): A
return blocks;
}
function buildConditionalSurfaces(nodes: VomNode[], captured: CapturedViewModel): CondSurface[] {
const probes = captured.surfaceProbes ?? [];
function buildConditionalSurfaces(
nodes: VomNode[],
captured: CapturedViewModel,
probes: CapturedSurfaceProbe[],
): CondSurface[] {
if (probes.length === 0) return [];
const surfaces: CondSurface[] = [];
@@ -649,6 +655,7 @@ function findSurfaceNodeByPoint(
export interface BuildVomSceneOptions {
pageUrl?: string;
supplementalNames?: ReadonlyMap<string, string>;
surfaceProbes?: CapturedSurfaceProbe[];
}
export type VomFrameDocument = CapturedFrameDocument<CdpAxNode>;
@@ -817,18 +824,19 @@ export function buildFrameVomScene(
excludedBackendNodeIds: captured.excludedBackendNodeIds,
supplementalNames: options.supplementalNames,
});
return attachCapturedSceneAnnotations(scene, documents, captured);
return attachCapturedSceneAnnotations(scene, documents, captured, options.surfaceProbes ?? []);
}
function attachCapturedSceneAnnotations(
scene: VomScene,
documents: VomFrameDocument[],
captured: CapturedViewModel,
surfaceProbes: CapturedSurfaceProbe[],
): VomScene {
const rootDocument = documents.find((document) => document.frameId === scene.rootFrameId);
const signals = capturedOnlySignals(rootDocument?.domNodes ?? captured.nodes);
const activeScopeBlocks = buildActiveScopeBlocks(scene.nodes, signals);
const surfaces = buildConditionalSurfaces(scene.nodes, captured);
const surfaces = buildConditionalSurfaces(scene.nodes, captured, surfaceProbes);
return {
...scene,
...(surfaces.length > 0 ? { surfaces } : {}),
@@ -1010,17 +1018,69 @@ async function captureForVom(
options: CaptureVomObservationOptions,
): Promise<CapturedViewModel> {
try {
return await captureViewModel(cdp, tabId, {
conditionalSurfaceProbe: options.conditionalSurfaceProbe ?? false,
hoverProbeBypassOverlay: options.hoverProbeBypassOverlay,
signal: options.signal,
});
return await captureViewModel(cdp, tabId, { signal: options.signal });
} catch (error) {
if (isAbortError(error)) throw error;
return fallbackCapturedViewModel(cdp, tabId, options.signal);
}
}
export interface HoverProbeOutcome {
/** Whether any active hover was dispatched during this observation. */
performed: boolean;
/**
* Whether hovering surfaced content absent from the static snapshot. Cached
* tooltip names count, so this may over-report on repeat observations — it
* errs towards telling the caller the snapshot is less fresh than it looks.
*/
revealedContent: boolean;
surfaceProbes: CapturedSurfaceProbe[];
tooltipNames: Map<string, string>;
}
const NO_HOVER_PROBES: HoverProbeOutcome = {
performed: false,
revealedContent: false,
surfaceProbes: [],
tooltipNames: new Map(),
};
/**
* Runs both active-hover chains back to back.
*
* Ordering matters: this happens after DOM *and* accessibility capture so a
* hover that opens a menu cannot leave one half of the observation describing
* the page before the change and the other half after it. Sharing one overlay
* bypass span also means the agent overlay toggles once per observation
* instead of once per chain.
*/
async function runHoverProbes(
cdp: CdpRunner,
tabId: number,
captured: CapturedViewModel,
documents: VomFrameDocument[],
staticSemantics: ReturnType<typeof resolveSemanticGraph>,
options: CaptureVomObservationOptions,
): Promise<HoverProbeOutcome> {
if (!options.conditionalSurfaceProbe) return NO_HOVER_PROBES;
return withOverlayBypass(options.hoverProbeBypassOverlay, tabId, async () => {
const surfaceProbes = await probeHoverSurfaces(cdp, tabId, captured.nodes, {
signal: options.signal,
});
throwIfAborted(options.signal, "observation");
const tooltipNames = await probeTooltipNames(cdp, tabId, documents, staticSemantics, {
signal: options.signal,
});
return {
performed: true,
revealedContent: surfaceProbes.length > 0 || tooltipNames.size > 0,
surfaceProbes,
tooltipNames,
};
});
}
export interface CaptureVomObservationOptions extends VomOptions {
conditionalSurfaceProbe?: boolean;
hoverProbeBypassOverlay?: (tabId: number, enabled: boolean) => Promise<void>;
@@ -1051,19 +1111,26 @@ export async function captureVomObservation(
excludedBackendNodeIds: captured.excludedBackendNodeIds,
});
const staticSemantics = resolveSemanticGraph(semanticGraph, { identifierFallback: false });
const tooltipNames = options.conditionalSurfaceProbe
? await probeTooltipNames(cdp, tabId, normalizedDocuments, staticSemantics, {
signal: options.signal,
hoverProbeBypassOverlay: options.hoverProbeBypassOverlay,
})
: new Map<string, string>();
const hoverProbes = await runHoverProbes(
cdp,
tabId,
captured,
normalizedDocuments,
staticSemantics,
options,
);
throwIfAborted(options.signal, "observation");
const scene = projectSemanticGraph(
normalizeSemanticStructure(
resolveSemanticGraph(semanticGraph, { supplementalNames: tooltipNames }),
resolveSemanticGraph(semanticGraph, { supplementalNames: hoverProbes.tooltipNames }),
),
);
const decoratedScene = attachCapturedSceneAnnotations(scene, normalizedDocuments, captured);
const decoratedScene = attachCapturedSceneAnnotations(
scene,
normalizedDocuments,
captured,
hoverProbes.surfaceProbes,
);
const rendered = renderVom(decoratedScene, {
maxDepth: options.maxDepth,
maxTokens: options.maxTokens,
@@ -1075,7 +1142,11 @@ export async function captureVomObservation(
rootFrameId: captured.rootFrameId ?? normalizedDocuments[0]?.frameId ?? "root",
frameDocuments: normalizedDocuments,
rendered,
surfaceProbes: captured.surfaceProbes,
surfaceProbes: hoverProbes.surfaceProbes,
hoverProbe: {
performed: hoverProbes.performed,
revealedContent: hoverProbes.revealedContent,
},
});
}
@@ -1141,6 +1212,14 @@ async function handleVomObservation(
ref_count: observation.refs.length,
tab_id: target.tabId,
truncated: observation.truncated,
...(toolName === "observe" && observation.hoverProbe?.performed
? {
hover_probe: {
performed: true,
revealed_content: observation.hoverProbe.revealedContent,
},
}
: {}),
...(toolName === "observe" && (params as ObserveParams).debug_surfaces
? {
debug: {
@@ -1179,5 +1258,13 @@ export async function handleObserve(
deps: SnapshotDeps = getDefaultDeps(),
signal?: AbortSignal,
): Promise<ObserveResult | RpcError> {
return handleVomObservation(manager, params, "observe", "transient_input", true, deps, signal);
return handleVomObservation(
manager,
params,
"observe",
"transient_input",
params.probe_hover === true,
deps,
signal,
);
}
+92 -7
View File
@@ -2,6 +2,7 @@ import { type SessionManager, SessionStartCleanupError } from "@/session-manager
import type { RpcError } from "@/transport/types";
import { rpcError } from "./errors";
import { clearRecordingForSession } from "./record";
import type { ChromeTabsApi } from "./shared";
import { isRpcError } from "./shared";
import { returnBorrowedTab, type TabManagementDeps } from "./tabs";
@@ -73,6 +74,12 @@ export interface SessionStopResult {
/** Tab ids whose return path failed; those entries remain borrowed so
* shutdown can be retried without closing the Agent Window. */
return_failures?: Array<{ tab_id: number; code: string; message: string }>;
/**
* True when the Agent Window was *released* to the user (instead of being
* closed) because it still contained user-created tabs after the agent's own
* tabs were closed. Single-session scope.
*/
window_released?: boolean;
}
export interface SessionStopDeps {
@@ -87,6 +94,13 @@ export interface SessionStopDeps {
* without a real browser.
*/
tabManagement?: TabManagementDeps;
/**
* Read-only Chrome tabs API used to *query* remaining tabs before deciding
* whether to release the window. Kept separate from `tabManagement.tabs`
* (a `TabMutationApi` that has no `query`) so the mutation interface stays
* pure. Tests can inject a fake `ChromeTabsApi` too.
*/
tabsQuery?: ChromeTabsApi;
}
/**
@@ -152,8 +166,10 @@ export async function handleSessionStart(
* cleanly (review parity with M6).
* 3. Detach CDP sessions the extension still holds for this
* session (no-op if M6/M7 didn't attach to any tab).
* 4. Close the Agent Window. SessionManager.stop() removes the
* Chrome window and forgets the context.
* 4. Close the agent-created tabs and the home tab (design §3.2).
* 5. If any tab remains — those are user-created tabs — release the
* window to the user (dropOnly) instead of closing it, so user
* tabs survive. Otherwise close the window as before.
*
* Failures in step 1 keep the Agent Window open: a failed borrowed tab
* may still be there, so closing the window would risk losing user
@@ -188,6 +204,8 @@ export async function handleSessionStop(
const returnFailures: SessionStopResult["return_failures"] = [];
const borrowedIds = Array.from(ctx.borrowedTabs.keys());
for (const tabId of borrowedIds) {
// A close event may have removed this borrow while another tab returned.
if (!ctx.borrowedTabs.has(tabId)) continue;
try {
const tabManagement = {
...(deps.tabManagement ?? {}),
@@ -197,6 +215,7 @@ export async function handleSessionStop(
((windowId: number) => manager.findByWindowId(windowId) !== null),
};
const outcome = await returnBorrowedTab(ctx, tabId, tabManagement);
if (!ctx.borrowedTabs.has(tabId)) continue;
if (typeof outcome === "object" && "code" in outcome) {
console.warn(`[bsk session_stop] auto-return failed for tab ${tabId}`, outcome);
returnFailures?.push({
@@ -209,6 +228,7 @@ export async function handleSessionStop(
ctx.borrowedTabs.delete(tabId);
}
} catch (err) {
if (!ctx.borrowedTabs.has(tabId)) continue;
const message = err instanceof Error ? err.message : String(err);
console.warn(`[bsk session_stop] auto-return threw for tab ${tabId}: ${message}`);
returnFailures?.push({
@@ -219,8 +239,15 @@ export async function handleSessionStop(
}
}
// A previously failed tab may have closed during a later return. Only
// confirmed close events remove borrows; ordinary move failures stay retryable.
const pendingReturnFailures = returnFailures.filter((failure) =>
ctx.borrowedTabs.has(failure.tab_id),
);
if (deps.signal?.aborted) {
const nonCancelFailures = returnFailures?.filter((failure) => failure.code !== "cancelled");
const nonCancelFailures = pendingReturnFailures.filter(
(failure) => failure.code !== "cancelled",
);
if (nonCancelFailures && nonCancelFailures.length > 0) {
return {
...(returnedTabIds.length > 0 ? { returned_tab_ids: returnedTabIds } : {}),
@@ -232,8 +259,8 @@ export async function handleSessionStop(
const result: SessionStopResult = {};
if (returnedTabIds.length > 0) result.returned_tab_ids = returnedTabIds;
if (returnFailures && returnFailures.length > 0) {
result.return_failures = returnFailures;
if (pendingReturnFailures.length > 0) {
result.return_failures = pendingReturnFailures;
// A failed return means at least one borrowed user tab may still be
// inside the Agent Window. Keep the session/window alive so the user
// can retry `bsk session stop` or explicitly `bsk tab return` after the
@@ -253,8 +280,66 @@ export async function handleSessionStop(
return { code: "cancelled", message: "session_stop aborted before window close" };
}
// Step 4: close the Agent Window and drop the context.
await manager.stop(params.session_id);
// Step 4: close every tab explicitly created by the agent, including the
// home tab. `tabsApi` is a
// TabMutationApi (remove only); `queryApi` is a separate read-only
// ChromeTabsApi. If neither is injected, we conservatively fall back to
// closing the window (see Step 5).
const tabsApi = deps.tabManagement?.tabs;
const queryApi = deps.tabsQuery;
if (tabsApi) {
// Close each agent-created tab that still exists.
const agentCreatedTabIds = Array.from(ctx.agentCreatedTabs);
for (const tabId of agentCreatedTabIds) {
try {
await tabsApi.remove(tabId);
ctx.agentCreatedTabs.delete(tabId);
} catch (err) {
// Tab may already be gone (closed by the user). Non-fatal.
console.warn(`[bsk session_stop] failed to close agent tab ${tabId}`, err);
}
}
}
// Step 5: decide whether to release (keep) the window or close it.
let shouldRelease = false;
if (queryApi) {
try {
const liveWindowTabs = await queryApi.query({ windowId: ctx.agentWindowId });
// Only genuine *user* tabs count toward keeping the window open.
// An agent tab that failed to close in Step 4 may still be present
// here; if we counted it as a reason to release (dropOnly), the
// window would be kept and that agent tab would leak (issue #57
// regression). Exclude any id still tracked in agentCreatedTabs.
const userTabs = liveWindowTabs.filter((t) => {
if (t.id === undefined) return false;
return !ctx.agentCreatedTabs.has(t.id);
});
const leakedAgentTabs = liveWindowTabs.filter(
(t) => t.id !== undefined && ctx.agentCreatedTabs.has(t.id),
);
if (leakedAgentTabs.length > 0) {
console.warn(
`[bsk session_stop] ${leakedAgentTabs.length} agent tab(s) failed to close; forcing window close instead of release`,
leakedAgentTabs.map((t) => t.id),
);
}
shouldRelease = userTabs.length > 0;
} catch {
// Query failed (e.g. window already gone) — conservatively close it.
shouldRelease = false;
}
}
if (shouldRelease) {
// Keep the window + its user tabs; only drop the session binding.
await manager.stop(params.session_id, { dropOnly: true });
result.window_released = true;
} else {
// Window is empty (or we couldn't verify state) — close it.
await manager.stop(params.session_id);
}
return result;
}
+1
View File
@@ -53,6 +53,7 @@ export type { DialogCursor };
export interface CdpRunner {
send<T = unknown>(tabId: number, method: string, params?: object): Promise<T>;
sendToTarget?<T = unknown>(target: CdpTarget, method: string, params?: object): Promise<T>;
detach?(tabId: number): Promise<void>;
getFrameGraph?(tabId: number): Promise<CdpFrameGraph>;
ensureAttachedToUrl?(tabId: number, expectedUrl: string | undefined): Promise<void>;
trackSessionTab?(sessionId: string, tabId: number): void;
+30 -24
View File
@@ -369,6 +369,7 @@ function buildCreateProps(
* Returns the created tab on success, or an `RpcError` on failure.
*/
async function createTabAndCleanup(
ctx: SessionContext,
deps: TabManagementDeps,
createProps: chrome.tabs.CreateProperties,
): Promise<CreatedChromeTab | RpcError> {
@@ -381,25 +382,28 @@ async function createTabAndCleanup(
message: err instanceof Error ? err.message : String(err),
};
}
if (aborted(deps.signal, "tab_create")) {
// We already opened the tab; close it on abort so we don't leak.
if (typeof tab.id === "number") {
try {
await getTabsApi(deps).remove(tab.id);
} catch (cleanupErr) {
return rpcError(
"protocol_error",
"cleanup_failed",
`tab_create aborted but cleanup of tab ${tab.id} failed: ${describeError(cleanupErr)}`,
{ resource_type: "tab", resource_id: tab.id },
);
}
}
return { code: "cancelled", message: "tab_create aborted" };
}
if (typeof tab.id !== "number") {
return { code: "protocol_error", message: "chrome.tabs.create returned no tab id" };
}
// Claim the concrete id returned by Chrome. Ownership never depends on
// matching this request to an asynchronous onCreated event.
ctx.agentCreatedTabs.add(tab.id);
if (aborted(deps.signal, "tab_create")) {
try {
await getTabsApi(deps).remove(tab.id);
ctx.agentCreatedTabs.delete(tab.id);
} catch (cleanupErr) {
// Keep the claim when cleanup fails so session_stop can retry instead
// of releasing an agent-owned tab to the user.
return rpcError(
"protocol_error",
"cleanup_failed",
`tab_create aborted but cleanup of tab ${tab.id} failed: ${describeError(cleanupErr)}`,
{ resource_type: "tab", resource_id: tab.id },
);
}
return { code: "cancelled", message: "tab_create aborted" };
}
return { ...tab, id: tab.id };
}
@@ -426,7 +430,7 @@ export async function handleTabCreate(
const paramErr = validateTabCreateParams(params);
if (paramErr) return paramErr;
const tab = await createTabAndCleanup(deps, buildCreateProps(ctx, params));
const tab = await createTabAndCleanup(ctx, deps, buildCreateProps(ctx, params));
if (isRpcError(tab)) return tab;
return {
@@ -526,6 +530,9 @@ export async function handleTabClose(
}
try {
await getTabsApi(deps).remove(params.tab_id);
// Keep the tracking set accurate so session_stop won't try to close a
// tab that's already gone (design §3.1).
ctx.agentCreatedTabs.delete(params.tab_id);
} catch (err) {
return {
code: "protocol_error",
@@ -753,13 +760,6 @@ async function executeBorrowCore(
);
if (moveErr) return moveErr;
// Best-effort activation — failure is non-fatal.
try {
await p.tabsApi.update(p.tabId, { active: true });
} catch (err) {
console.debug("[bsk tab_borrow] activate after move failed", err);
}
return { originalWindowId, originalIndex };
}
@@ -823,6 +823,12 @@ export async function handleTabBorrow(
message: err instanceof Error ? err.message : String(err),
};
}
// Activate after commit so overlay/event observers see the tab as claimed.
try {
await tabsApi.update(params.tab_id, { active: true });
} catch (err) {
console.debug("[bsk tab_borrow] activate after move failed", err);
}
return {
tab_id: params.tab_id,
original_window_id: coreResult.originalWindowId,
@@ -0,0 +1,42 @@
// Shared time boundary for browser-side file-transfer transactions. Concrete
// upload/download mechanisms own their state machines and effect semantics.
export class BoundedWaitError extends Error {
constructor(
readonly kind: "timeout" | "aborted",
message: string,
) {
super(message);
}
}
export function remainingMs(deadline: number): number {
return Math.max(0, deadline - Date.now());
}
export async function waitBounded<T>(
promise: Promise<T>,
deadline: number,
signal: AbortSignal | undefined,
timeoutMessage: string,
): Promise<T> {
const remaining = remainingMs(deadline);
if (signal?.aborted) throw new BoundedWaitError("aborted", "file transfer aborted");
if (remaining === 0) throw new BoundedWaitError("timeout", timeoutMessage);
let timer: ReturnType<typeof setTimeout> | undefined;
let onAbort: (() => void) | undefined;
const boundary = new Promise<never>((_resolve, reject) => {
timer = setTimeout(() => reject(new BoundedWaitError("timeout", timeoutMessage)), remaining);
if (signal) {
onAbort = () => reject(new BoundedWaitError("aborted", "file transfer aborted"));
signal.addEventListener("abort", onAbort, { once: true });
}
});
try {
return await Promise.race([promise, boundary]);
} finally {
if (timer) clearTimeout(timer);
if (signal && onAbort) signal.removeEventListener("abort", onAbort);
}
}
+85
View File
@@ -0,0 +1,85 @@
// Upload orchestration: validate the session-scoped request, resolve its click
// target, then delegate the browser protocol transaction to the file-input
// transaction module.
import {
type CaptureSuppressSendToTab,
withExtensionOverlayHidden,
} from "@/lib/capture-suppress-bridge";
import type { SessionManager } from "@/session-manager/manager";
import type { RpcError, UploadParams, UploadResult } from "@/transport/types";
import { uploadThroughFileDrop } from "./file-drop-transaction";
import { uploadThroughActivatedFileInput } from "./file-input-transaction";
import { clickResolvedTarget, type InteractionDeps, resolveActionTarget } from "./interaction";
import {
type CdpRunner,
enforceAgentWindow,
isRpcError,
lookupSession,
resolveTargetTab,
} from "./shared";
const DEFAULT_TIMEOUT_MS = 120_000;
export interface UploadDeps extends InteractionDeps {
cdp: CdpRunner;
sendToTab?: CaptureSuppressSendToTab;
}
export async function handleUpload(
manager: SessionManager,
params: UploadParams,
deps: UploadDeps,
): Promise<UploadResult | RpcError> {
const ctx = lookupSession(manager, params, "upload");
if (isRpcError(ctx)) return ctx;
const target = await resolveTargetTab(manager, ctx, params.tab_id, deps.tabsApi);
if (isRpcError(target)) return target;
const denied = enforceAgentWindow(ctx, target, "upload");
if (denied) return denied;
if (
params.files.length === 0 ||
params.files.length > 20 ||
params.files.some((file) => !file.staged_path)
) {
return { code: "invalid_params", message: "upload requires daemon-staged files" };
}
const address = await resolveActionTarget(deps.cdp, ctx, target, params, "upload");
if (isRpcError(address)) return address;
const timeoutMs = params.timeout_ms ?? DEFAULT_TIMEOUT_MS;
const files = params.files.map((file) => file.staged_path as string);
const mode = params.mode ?? "input";
let usedRef = address.usedRef;
let usedSelector = address.usedSelector;
if (mode === "drop") {
deps.cdp.trackSessionTab?.(ctx.sessionId, target.tabId);
const transaction = await uploadThroughFileDrop({
cdp: deps.cdp,
actionTarget: address,
files,
timeoutMs,
signal: deps.signal,
withOverlayHidden: (operation) =>
withExtensionOverlayHidden(target.tabId, operation, deps.sendToTab),
});
if (isRpcError(transaction)) return transaction;
} else {
const transaction = await uploadThroughActivatedFileInput({
cdp: deps.cdp,
actionTarget: address,
files,
timeoutMs,
signal: deps.signal,
trigger: () => clickResolvedTarget(ctx, address, {}, deps),
});
if (isRpcError(transaction)) return transaction;
usedRef = transaction.click.used_ref;
usedSelector = transaction.click.used_selector;
}
return {
tab_id: target.tabId,
used_ref: usedRef,
used_selector: usedSelector,
file_names: params.files.map((file) => file.name),
};
}
@@ -1,6 +1,6 @@
import { describe, expect, it, vi } from "vitest";
import { OVERLAY_HOST_MARKER_ATTR, OVERLAY_HOST_NAME } from "../../../lib/overlay-bridge";
import { captureViewModel, collectOverlayExcludedBackendIds } from "../capture";
import { captureViewModel, collectOverlayExcludedBackendIds, probeHoverSurfaces } from "../capture";
// Minimal but format-accurate captureSnapshot reply: a body with one
// fixed full-screen overlay div carrying a password input.
@@ -251,11 +251,10 @@ describe("captureViewModel", () => {
expect(input?.attrs).toEqual({ type: "password" });
});
it("does not run conditional surface probes by default", async () => {
it("never hovers the page while capturing", async () => {
const cdp = makeCdp(fakeSnapshotReply());
const result = await captureViewModel(cdp, 4);
await captureViewModel(cdp, 4);
expect(result.surfaceProbes).toEqual([]);
expect(cdp.send).not.toHaveBeenCalledWith(4, "Input.dispatchMouseEvent", expect.anything());
});
@@ -319,9 +318,10 @@ describe("captureViewModel", () => {
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
const result = await captureViewModel(cdp, 4, { conditionalSurfaceProbe: true });
const captured = await captureViewModel(cdp, 4);
const surfaceProbes = await probeHoverSurfaces(cdp, 4, captured.nodes);
expect(result.surfaceProbes).toEqual([
expect(surfaceProbes).toEqual([
{
triggerBackendNodeId: 12,
triggerPoint: { x: 956, y: 32 },
@@ -378,9 +378,10 @@ describe("captureViewModel", () => {
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
const result = await captureViewModel(cdp, 4, { conditionalSurfaceProbe: true });
const captured = await captureViewModel(cdp, 4);
const surfaceProbes = await probeHoverSurfaces(cdp, 4, captured.nodes);
expect(result.surfaceProbes).toEqual([
expect(surfaceProbes).toEqual([
expect.objectContaining({
triggerBackendNodeId: 12,
subItems: ["My profile", "Sign out"],
@@ -422,7 +423,8 @@ describe("captureViewModel", () => {
}) as unknown as <T>(tabId: number, method: string, params?: object) => Promise<T>,
};
await captureViewModel(cdp, 4, { conditionalSurfaceProbe: true });
const captured = await captureViewModel(cdp, 4);
await probeHoverSurfaces(cdp, 4, captured.nodes);
expect(hoverMoves).toBe(1);
});
@@ -0,0 +1,84 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { ProbeBudget, resetOverlayBypassDepth, withOverlayBypass } from "../hover-perception";
afterEach(() => {
resetOverlayBypassDepth();
});
describe("ProbeBudget", () => {
it("refuses a candidate that cannot finish inside the budget", () => {
const budget = new ProbeBudget(500);
expect(budget.canAfford(400)).toBe(true);
expect(budget.canAfford(600)).toBe(false);
});
it("stops admitting candidates once the remaining budget is smaller than one", async () => {
const budget = new ProbeBudget(60);
await new Promise((resolve) => setTimeout(resolve, 40));
// A top-of-loop `elapsed > total` check would still admit this candidate
// and then overshoot by its full cost.
expect(budget.canAfford(50)).toBe(false);
});
});
describe("withOverlayBypass", () => {
it("toggles the overlay once around nested probe phases", async () => {
const bypass = vi.fn(async () => {});
await withOverlayBypass(bypass, 4, async () => {
await withOverlayBypass(bypass, 4, async () => {
expect(bypass).toHaveBeenCalledTimes(1);
});
expect(bypass).toHaveBeenCalledTimes(1);
});
expect(bypass.mock.calls).toEqual([
[4, true],
[4, false],
]);
});
it("restores the overlay when the probe throws", async () => {
const bypass = vi.fn(async () => {});
await expect(
withOverlayBypass(bypass, 4, async () => {
throw new Error("probe exploded");
}),
).rejects.toThrow("probe exploded");
expect(bypass).toHaveBeenLastCalledWith(4, false);
});
it("reports a failed restore instead of swallowing it", async () => {
const onRestoreFailure = vi.fn();
const bypass = vi.fn(async (_tabId: number, enabled: boolean) => {
if (!enabled) throw new Error("tab is gone");
});
await withOverlayBypass(bypass, 4, async () => undefined, { onRestoreFailure });
expect(onRestoreFailure).toHaveBeenCalledTimes(1);
});
it("does not pin the overlay off for the tab when a restore fails", async () => {
const failing = vi.fn(async (_tabId: number, enabled: boolean) => {
if (!enabled) throw new Error("tab is gone");
});
await withOverlayBypass(failing, 4, async () => undefined, { onRestoreFailure: () => {} });
const bypass = vi.fn(async () => {});
await withOverlayBypass(bypass, 4, async () => undefined);
expect(bypass.mock.calls).toEqual([
[4, true],
[4, false],
]);
});
it("runs the probe untouched when no bypass is wired up", async () => {
await expect(withOverlayBypass(undefined, 4, async () => "done")).resolves.toBe("done");
});
});
@@ -132,6 +132,37 @@ describe("tooltip name enrichment", () => {
expect(cdp.sendToTarget).not.toHaveBeenCalled();
});
it("remembers controls that revealed no tooltip so the miss is paid once", async () => {
const cdp: CdpRunner = {
send: vi.fn(async () => ({})) as CdpRunner["send"],
sendToTarget: vi.fn(async (_target, method) => {
if (method === "Page.createIsolatedWorld") return { executionContextId: 31 };
if (method === "DOM.resolveNode") return { object: { objectId: "button-801" } };
if (method === "Runtime.callFunctionOn") return { result: { value: [] } };
return {};
}) as CdpRunner["sendToTarget"],
};
const documents = [
document({ tabId: 11, sessionId: "miss-session" }, 801, undefined, "miss-frame"),
];
const graph = semantics(documents);
const first = await probeTooltipNames(cdp, 11, documents, graph);
expect(first.size).toBe(0);
const hoversAfterFirst = vi
.mocked(cdp.send)
.mock.calls.filter(([, method]) => method === "Input.dispatchMouseEvent").length;
expect(hoversAfterFirst).toBeGreaterThan(0);
const second = await probeTooltipNames(cdp, 11, documents, graph);
expect(second.size).toBe(0);
expect(
vi.mocked(cdp.send).mock.calls.filter(([, method]) => method === "Input.dispatchMouseEvent")
.length,
).toBe(hoversAfterFirst);
});
it("keeps tooltip evidence isolated across sibling frame sessions", async () => {
let hoveredX = -10;
const cdp: CdpRunner = {
+80 -65
View File
@@ -9,7 +9,7 @@ import { evaluateHoverTrigger } from "@/lib/hover-trigger-policy";
import { isOverlayHostNode, OVERLAY_HOST_SELECTOR } from "../../lib/overlay-bridge";
import { childFrameProjection, type GeometryProjection, projectRectToViewport } from "../geometry";
import type { CdpRunner } from "../shared";
import { clearHover, waitForHover } from "./hover-perception";
import { clearHover, ProbeBudget, waitForHover } from "./hover-perception";
const REQUESTED_STYLES = ["position", "pointer-events", "cursor", "visibility", "opacity"] as const;
const STYLE_COL = Object.fromEntries(
@@ -71,14 +71,11 @@ export interface CapturedViewModel {
/** Explicit DOMSnapshot frame ancestry; never inferred from backend node ids. */
frameParentIds?: Map<string, string>;
rootFrameId?: string;
surfaceProbes?: CapturedSurfaceProbe[];
/** Backend node ids belonging to the agent overlay host + its shadow subtree. */
excludedBackendNodeIds: Set<number>;
}
export interface CaptureViewModelOptions {
conditionalSurfaceProbe?: boolean;
hoverProbeBypassOverlay?: (tabId: number, enabled: boolean) => Promise<void>;
signal?: AbortSignal;
}
@@ -368,7 +365,16 @@ function collectBackendIdsFromDomNode(node: CdpDomNode | undefined, out: Set<num
}
}
const MAX_HOVER_PROBE_MS = 2_000;
/**
* Ceiling for the whole hover-surface phase.
*
* The previous 2000 was only a floor: the loop checked elapsed time at the top,
* so a candidate could start with 1ms left and still run its two settle
* windows, pushing real cost to ~2.6s. This is that true ceiling, now actually
* enforced by an up-front affordability check, so the same number of candidates
* get probed under a bound that no longer lies.
*/
const MAX_HOVER_PROBE_MS = 2_600;
const MAX_HOVER_TRIGGERS = 6;
const MAX_HOVER_SURFACES = 3;
const HOVER_SETTLE_MS = 300;
@@ -614,13 +620,35 @@ function confidenceForHover(
return "low";
}
async function probeHoverSurfaces(
/**
* One candidate costs two settle windows (baseline + post-hover) plus a few
* CDP round trips. Used to decide whether the next candidate still fits the
* budget before paying for it.
*/
const HOVER_CANDIDATE_COST_MS = HOVER_SETTLE_MS * 2;
export interface HoverSurfaceProbeOptions {
signal?: AbortSignal;
}
/**
* Hovers a bounded set of likely menu triggers and reports the sub-items each
* one reveals.
*
* Runs after DOM *and* accessibility capture. Hovering can open menus and
* change layout, so probing between the two captures would leave the DOM half
* of an observation describing the page before the change and the AX half
* describing it after.
*
* The caller owns the overlay bypass span (see `withOverlayBypass`).
*/
export async function probeHoverSurfaces(
cdp: CdpRunner,
tabId: number,
nodes: CapturedNode[],
options: CaptureViewModelOptions,
options: HoverSurfaceProbeOptions = {},
): Promise<CapturedSurfaceProbe[]> {
const started = Date.now();
const budget = new ProbeBudget(MAX_HOVER_PROBE_MS);
try {
throwIfAborted(options.signal);
const cssScan = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
@@ -635,60 +663,54 @@ async function probeHoverSurfaces(
const results: CapturedSurfaceProbe[] = [];
const seen = new Set<number>();
throwIfAborted(options.signal);
await options.hoverProbeBypassOverlay?.(tabId, true).catch(() => undefined);
try {
for (const candidate of candidates.slice(0, MAX_HOVER_TRIGGERS)) {
throwIfAborted(options.signal);
for (const candidate of candidates.slice(0, MAX_HOVER_TRIGGERS)) {
if (!budget.canAfford(HOVER_CANDIDATE_COST_MS)) break;
if (results.length >= MAX_HOVER_SURFACES) break;
if (seen.has(candidate.backendNodeId)) continue;
try {
await clearHover(cdp, tabId);
throwIfAborted(options.signal);
if (Date.now() - started > MAX_HOVER_PROBE_MS) break;
if (results.length >= MAX_HOVER_SURFACES) break;
if (seen.has(candidate.backendNodeId)) continue;
try {
await clearHover(cdp, tabId);
throwIfAborted(options.signal);
await waitForHover(HOVER_SETTLE_MS, options.signal);
const baselineReply = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: hoverStateExpression(),
returnByValue: true,
});
throwIfAborted(options.signal);
const baselineItems = runtimeValue<HoverRuntimeItem[]>(baselineReply) ?? [];
await waitForHover(HOVER_SETTLE_MS, options.signal);
const baselineReply = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: hoverStateExpression(),
returnByValue: true,
});
throwIfAborted(options.signal);
const baselineItems = runtimeValue<HoverRuntimeItem[]>(baselineReply) ?? [];
throwIfAborted(options.signal);
await cdp.send(tabId, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: candidate.x,
y: candidate.y,
});
throwIfAborted(options.signal);
await waitForHover(HOVER_SETTLE_MS, options.signal);
const collected = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: hoverStateExpression(),
returnByValue: true,
});
throwIfAborted(options.signal);
const subItems = diffHoverItems(
baselineItems,
runtimeValue<HoverRuntimeItem[]>(collected) ?? [],
);
if (subItems.length === 0) continue;
seen.add(candidate.backendNodeId);
results.push({
triggerBackendNodeId: candidate.backendNodeId,
triggerPoint: { x: candidate.x, y: candidate.y },
triggerAction: "hover",
subItems,
confidence: confidenceForHover(candidate, subItems),
});
} catch (error) {
if (isAbortError(error)) throw error;
continue;
} finally {
await clearHover(cdp, tabId);
}
throwIfAborted(options.signal);
await cdp.send(tabId, "Input.dispatchMouseEvent", {
type: "mouseMoved",
x: candidate.x,
y: candidate.y,
});
throwIfAborted(options.signal);
await waitForHover(HOVER_SETTLE_MS, options.signal);
const collected = await cdp.send<RuntimeEvaluateReply>(tabId, "Runtime.evaluate", {
expression: hoverStateExpression(),
returnByValue: true,
});
throwIfAborted(options.signal);
const subItems = diffHoverItems(
baselineItems,
runtimeValue<HoverRuntimeItem[]>(collected) ?? [],
);
if (subItems.length === 0) continue;
seen.add(candidate.backendNodeId);
results.push({
triggerBackendNodeId: candidate.backendNodeId,
triggerPoint: { x: candidate.x, y: candidate.y },
triggerAction: "hover",
subItems,
confidence: confidenceForHover(candidate, subItems),
});
} catch (error) {
if (isAbortError(error)) throw error;
continue;
} finally {
await clearHover(cdp, tabId);
}
} finally {
await options.hoverProbeBypassOverlay?.(tabId, false).catch(() => undefined);
}
return results;
} catch (err) {
@@ -1042,7 +1064,6 @@ export async function captureViewModel(
nodes: [],
viewport,
iframeNodes: new Map(),
surfaceProbes: [],
excludedBackendNodeIds: new Set(),
};
}
@@ -1071,11 +1092,6 @@ export async function captureViewModel(
await enrichFormControlStates(cdp, tabId, [nodes, ...iframeNodes.values()], options.signal);
throwIfAborted(options.signal);
const surfaceProbes = options.conditionalSurfaceProbe
? await probeHoverSurfaces(cdp, tabId, nodes, options)
: [];
throwIfAborted(options.signal);
const frameNodes = new Map<string, CapturedNode[]>();
if (topContext.frameId) frameNodes.set(topContext.frameId, nodes);
for (const iframeFrameNodes of iframeNodes.values()) {
@@ -1096,7 +1112,6 @@ export async function captureViewModel(
frameOwnerBackendNodeIds: frameParsed.frameOwnerBackendNodeIds,
frameParentIds: frameParsed.frameParentIds,
...(topContext.frameId ? { rootFrameId: topContext.frameId } : {}),
surfaceProbes,
excludedBackendNodeIds,
};
}
@@ -35,3 +35,91 @@ export async function clearHover(cdp: CdpRunner, tabId: number): Promise<void> {
}),
);
}
/**
* Wall-clock budget for a probe phase.
*
* Callers ask whether the *next* candidate still fits before starting it,
* rather than checking elapsed time at the top of the loop. A top-of-loop
* check lets a candidate start with 1ms of budget left and then run to
* completion, overshooting by a full settle window.
*/
export class ProbeBudget {
private readonly startedAt = Date.now();
constructor(private readonly totalMs: number) {}
get elapsedMs(): number {
return Date.now() - this.startedAt;
}
/** True when a candidate costing `estimatedMs` still fits in the budget. */
canAfford(estimatedMs: number): boolean {
return this.elapsedMs + estimatedMs <= this.totalMs;
}
}
export type OverlayBypass = (tabId: number, enabled: boolean) => Promise<void>;
export interface OverlayBypassScopeOptions {
/**
* Called when the overlay could not be restored. Defaults to `console.error`
* so a stuck-bypassed overlay is never silent; tests inject a spy.
*/
onRestoreFailure?: (error: unknown) => void;
}
const overlayBypassDepth = new Map<number, number>();
/**
* Runs `body` with the agent overlay bypassed, restoring it afterwards.
*
* Reference counted per tab so nested probe phases share a single bypass span
* instead of toggling the overlay once per phase. Restore failures are
* reported rather than swallowed, and the depth is released even when restore
* throws so a failure cannot pin the overlay off for the tab's lifetime.
*/
export async function withOverlayBypass<T>(
bypass: OverlayBypass | undefined,
tabId: number,
body: () => Promise<T>,
options: OverlayBypassScopeOptions = {},
): Promise<T> {
if (!bypass) return body();
const depth = overlayBypassDepth.get(tabId) ?? 0;
overlayBypassDepth.set(tabId, depth + 1);
if (depth === 0) {
try {
await bypass(tabId, true);
} catch (error) {
// Failing to hide the overlay only risks probing our own UI, so continue.
console.debug("[bsk hover] overlay bypass could not be enabled", error);
}
}
try {
return await body();
} finally {
const current = overlayBypassDepth.get(tabId) ?? 1;
if (current <= 1) {
overlayBypassDepth.delete(tabId);
try {
await bypass(tabId, false);
} catch (error) {
(options.onRestoreFailure ?? defaultRestoreFailure)(error);
}
} else {
overlayBypassDepth.set(tabId, current - 1);
}
}
}
function defaultRestoreFailure(error: unknown): void {
console.error("[bsk hover] agent overlay could not be restored after probing", error);
}
/** Test seam: drops any bypass depth left behind by a crashed probe. */
export function resetOverlayBypassDepth(): void {
overlayBypassDepth.clear();
}
+26 -15
View File
@@ -1,7 +1,7 @@
import { isVomReferenceNode } from "@browser-skill/vom";
import { type CdpRunner, sendToCdpTarget } from "../shared";
import type { FrameDocument, FrameOwnedAxNode } from "./frame-document";
import { clearHover, waitForHover } from "./hover-perception";
import { clearHover, ProbeBudget, waitForHover } from "./hover-perception";
import { stableIdentifierName } from "./semantic-graph/name-evidence";
import { frameBackendKey, type ResolvedSemanticGraph } from "./semantic-graph/types";
@@ -9,11 +9,18 @@ const MAX_TOOLTIP_CANDIDATES = 8;
const MAX_TOOLTIP_PROBE_MS = 2_400;
const TOOLTIP_SETTLE_MS = 250;
const CACHE_TTL_MS = 5 * 60_000;
/**
* Misses expire sooner than hits: a control that gains a tooltip once the app
* finishes booting should still be discoverable, but not at the cost of
* re-probing every unlabelled control on every observation.
*/
const MISS_CACHE_TTL_MS = 60_000;
const MAX_CACHE_ENTRIES = 500;
const FORM_CONTROL_TAGS = new Set(["input", "select", "textarea"]);
interface TooltipCacheEntry {
name: string;
/** `null` records "probed, revealed no tooltip" so the miss is not re-paid. */
name: string | null;
expiresAt: number;
}
@@ -27,7 +34,6 @@ interface TooltipCandidate<T extends FrameOwnedAxNode> {
export interface TooltipNameProbeOptions {
signal?: AbortSignal;
hoverProbeBypassOverlay?: (tabId: number, enabled: boolean) => Promise<void>;
}
const tooltipCache = new Map<string, TooltipCacheEntry>();
@@ -39,10 +45,11 @@ function cacheKey<T extends FrameOwnedAxNode>(
return `${document.target.tabId}\u0000${document.frameId}\u0000${document.url ?? ""}\u0000${backendNodeId}`;
}
function cachedName<T extends FrameOwnedAxNode>(
/** Returns the cache entry, or `undefined` when this control was never probed. */
function cachedEntry<T extends FrameOwnedAxNode>(
document: FrameDocument<T>,
backendNodeId: number,
): string | undefined {
): TooltipCacheEntry | undefined {
const key = cacheKey(document, backendNodeId);
const entry = tooltipCache.get(key);
if (!entry) return undefined;
@@ -50,13 +57,13 @@ function cachedName<T extends FrameOwnedAxNode>(
tooltipCache.delete(key);
return undefined;
}
return entry.name;
return entry;
}
function storeName<T extends FrameOwnedAxNode>(
document: FrameDocument<T>,
backendNodeId: number,
name: string,
name: string | null,
): void {
if (tooltipCache.size >= MAX_CACHE_ENTRIES) {
const oldest = tooltipCache.keys().next().value;
@@ -64,7 +71,7 @@ function storeName<T extends FrameOwnedAxNode>(
}
tooltipCache.set(cacheKey(document, backendNodeId), {
name,
expiresAt: Date.now() + CACHE_TTL_MS,
expiresAt: Date.now() + (name === null ? MISS_CACHE_TTL_MS : CACHE_TTL_MS),
});
}
@@ -237,22 +244,27 @@ export async function probeTooltipNames<T extends FrameOwnedAxNode>(
const names = new Map<string, string>();
const pending: TooltipCandidate<T>[] = [];
for (const candidate of candidates(documents, graph)) {
const cached = cachedName(candidate.document, candidate.backendNodeId);
const cached = cachedEntry(candidate.document, candidate.backendNodeId);
if (cached) {
names.set(frameBackendKey(candidate.document.frameId, candidate.backendNodeId), cached);
// A cached miss is still a decision: skip the control instead of re-probing.
if (cached.name !== null) {
names.set(
frameBackendKey(candidate.document.frameId, candidate.backendNodeId),
cached.name,
);
}
} else if (pending.length < MAX_TOOLTIP_CANDIDATES) {
pending.push(candidate);
}
}
if (pending.length === 0) return names;
const started = Date.now();
const budget = new ProbeBudget(MAX_TOOLTIP_PROBE_MS);
const contexts = new Map<string, number>();
await options.hoverProbeBypassOverlay?.(tabId, true).catch(() => undefined);
try {
for (const candidate of pending) {
if (options.signal?.aborted) throw new DOMException("observation aborted", "AbortError");
if (Date.now() - started >= MAX_TOOLTIP_PROBE_MS) break;
if (!budget.canAfford(TOOLTIP_SETTLE_MS)) break;
let objectId: string | undefined;
try {
let contextId = contexts.get(candidate.document.frameId);
@@ -278,9 +290,9 @@ export async function probeTooltipNames<T extends FrameOwnedAxNode>(
await waitForHover(TOOLTIP_SETTLE_MS, options.signal);
const after = await tooltipTexts(cdp, candidate.document, objectId);
const name = cleanTooltipDiff(before, after);
storeName(candidate.document, candidate.backendNodeId, name ?? null);
if (!name) continue;
names.set(frameBackendKey(candidate.document.frameId, candidate.backendNodeId), name);
storeName(candidate.document, candidate.backendNodeId, name);
} catch (error) {
if (error instanceof Error && error.name === "AbortError") throw error;
} finally {
@@ -290,7 +302,6 @@ export async function probeTooltipNames<T extends FrameOwnedAxNode>(
}
} finally {
await clearHover(cdp, tabId);
await options.hoverProbeBypassOverlay?.(tabId, false).catch(() => undefined);
}
return names;
}
@@ -37,6 +37,17 @@ export interface CaptureVomObservationResult {
frames: CaptureVomFrame[];
matchNodes: CaptureVomMatchNode[];
surfaceProbes?: CapturedSurfaceProbe[];
/**
* Whether this observation actively hovered the page, and whether that
* revealed content the static tree does not contain. Lets callers judge how
* far the page may have drifted from the returned snapshot.
*/
hoverProbe?: HoverProbeReport;
}
export interface HoverProbeReport {
performed: boolean;
revealedContent: boolean;
}
function projectFrames(documents: CapturedFrameDocument<FrameAxNode>[]): CaptureVomFrame[] {
@@ -75,6 +86,7 @@ export function projectRecordSafeObservation(input: {
frameDocuments: CapturedFrameDocument<FrameAxNode>[];
rendered: VomResult;
surfaceProbes?: CapturedSurfaceProbe[];
hoverProbe?: HoverProbeReport;
}): CaptureVomObservationResult {
return {
text: input.rendered.text,
@@ -84,5 +96,6 @@ export function projectRecordSafeObservation(input: {
frames: projectFrames(input.frameDocuments),
matchNodes: projectMatchNodes(input.frameDocuments),
surfaceProbes: input.surfaceProbes,
...(input.hoverProbe ? { hoverProbe: input.hoverProbe } : {}),
};
}
+67
View File
@@ -33,10 +33,25 @@ export type RpcErrorReason =
| "restricted_tab_url"
| "borrow_conflict"
| "screenshot_capture_failed"
| "file_input_probe_failed"
| "file_input_not_activated"
| "set_file_input_failed"
| "upload_mechanism_unsupported"
| "file_drop_target_unavailable"
| "file_drop_failed"
| "download_capture_failed"
| "transfer_outcome_unknown"
| "transfer_timeout"
| "cleanup_failed";
export type TransferEffectState = "none" | "committed" | "unknown";
export type TransferCleanupState = "complete" | "failed";
export interface RpcErrorData {
reason?: RpcErrorReason;
effect_state?: TransferEffectState;
phase?: string;
cleanup_state?: TransferCleanupState;
[key: string]: unknown;
}
@@ -318,9 +333,14 @@ export interface SnapshotResult {
export interface ObserveParams extends SnapshotParams {
debug_surfaces?: boolean;
probe_hover?: boolean;
}
export interface ObserveResult extends SnapshotResult {
hover_probe?: {
performed: boolean;
revealed_content: boolean;
};
debug?: {
surface_probes?: Array<{
trigger_backend_node_id: number;
@@ -500,6 +520,53 @@ export interface SelectResult {
dialogs?: JavaScriptDialogInfo[];
}
export interface UploadFile {
transfer_id: string;
name: string;
staged_path?: string;
}
export type UploadMode = "input" | "drop";
export interface UploadParams {
session_id: string;
ref?: string;
selector?: string;
tab_id?: number;
files: UploadFile[];
mode?: UploadMode;
timeout_ms?: number;
}
export interface UploadResult {
tab_id: number;
used_ref?: string;
used_selector?: string;
file_names: string[];
}
export interface DownloadParams {
session_id: string;
ref?: string;
selector?: string;
tab_id?: number;
timeout_ms?: number;
browser_relative_dir?: string;
max_byte_size?: number;
}
export interface DownloadResult {
tab_id: number;
used_ref?: string;
used_selector?: string;
suggested_filename: string;
byte_size: number;
mime?: string;
danger?: string;
browser_path?: string;
transfer_id?: string;
}
// --------------------------------------------------------------------------
// M9 tool payloads — evaluate / wait_for_navigation / wait_ms
// --------------------------------------------------------------------------
+1
View File
@@ -30,6 +30,7 @@ export default defineConfig({
permissions: [
"alarms",
"debugger",
"downloads",
"idle",
"notifications",
"tabs",
+1
View File
@@ -44,6 +44,7 @@
"!packages/dsh-plugin-browserskill/src/client/bsk-ui.nomodule.css",
"!packages/dsh-plugin-browserskill/.bsk-ui.raw.css",
"!packages/dsh-plugin-browserskill/.bsk-ui.input.css",
"!evals/browser/results/**",
"!**/target/**",
"!**/.wxt/**",
"!**/.output/**",
+2
View File
@@ -53,6 +53,7 @@ flate2 = { workspace = true }
tar = { workspace = true }
zip = { workspace = true }
sha2 = { workspace = true }
base64 = { workspace = true }
[target.'cfg(unix)'.dependencies]
nix = { workspace = true }
@@ -63,6 +64,7 @@ windows-sys = { version = "0.59", features = [
"Win32_Foundation",
"Win32_System_Threading",
"Win32_Security",
"Win32_Storage_FileSystem",
] }
[dev-dependencies]
+128 -275
View File
@@ -1,327 +1,180 @@
---
name: browser-skill
description: |
Use when the user asks to perform browser automation tasks against their
logged-in browser: visit and read pages, fill forms, scrape data, click
through a flow, regression-test a PR's UI, validate a deployed page.
Requires the bsk CLI installed and the browser-skill extension loaded.
Use when the user asks to automate their logged-in Chromium browser: visit
and read pages, fill forms, scrape data, click through flows, regression-test
a PR's UI, validate a deployed page, or operate a tab they identify. Requires
the bsk CLI and browser extension.
---
# browser-skill
Drive the user's **real Chromium browser** (with their logins and cookies) through the `bsk` CLI. The extension opens an isolated **Agent Window** for automation; the user's normal windows stay protected unless you explicitly borrow a tab.
Drive the user's real Chromium browser through `bsk`. Automation runs in an isolated **Agent
Window** with the user's existing logins and cookies. User-window tabs remain protected unless they
are explicitly borrowed.
## When to use
Do not use this skill for tasks with no browser, for extension installation, or when the user only
wants instructions. Never extract credentials, cookies, tokens, or other secrets from pages.
- Open pages, read titles/text, scrape structured data from sites the user can already access
- Fill forms, click through multi-step flows, smoke-test a UI change
- Understand pages with `bsk snapshot` first; use `bsk get-html` or `bsk screenshot` only when the snapshot is insufficient
- Operate on a specific user tab they point you at (after `bsk tab borrow`)
## Required lifecycle
## When NOT to use
Every browser task owns a bounded session:
- Tasks with **no browser** involved (files, APIs, databases only)
- Installing or configuring the extension (point the user to setup docs instead)
- **Credential harvesting** — never run `bsk evaluate` on banking, SSO, or password-manager pages to extract tokens, cookies, or secrets
- Long-lived control of a user's personal login window — borrow only for the immediate step, then `bsk tab return` or end the session
- Replacing the user's manual browsing when they only wanted an explanation
## Prerequisites
1. `bsk` on `PATH` (Rust CLI from browser-skill)
2. browser-skill **extension** loaded in Chromium and connected (popup shows green)
3. Any `bsk` command auto-starts background services as needed; use `bsk doctor` if anything fails
## Mandatory workflow
Every automation task **must** follow this lifecycle. Do **not** rely on idle timeouts (default session idle is 5 minutes).
```
1. bsk session start → capture the 4-letter session id printed on stdout
2. … every tool command … → always pass --session <id>
3. bsk session stop <id> → REQUIRED when done (even on error paths)
```text
1. bsk session start # retain the printed 4-letter session id
2. bsk ... --session <id> # pass it to every session-scoped command
3. bsk session stop <id> # always run on success and error paths
```
Optional: `bsk session start --browser <instance-id-or-label>` when multiple browsers are connected (`bsk browsers` / error output lists them). Add `--no-focus` to open the Agent Window in the background without stealing focus from the user's current window.
Do not rely on the idle timeout for cleanup. Stop the session as soon as the goal is met unless the
user explicitly asks to keep it open. Stopping also returns borrowed tabs.
Emergency cleanup: `bsk session stop --all` or the Agent Window overlay **Stop all**.
Any `bsk` command auto-starts the background services it needs; never manage the daemon by hand.
When multiple browsers are connected, use `bsk browsers` and start with
`bsk session start --browser <id-or-label>`. Add `--no-focus` to that same start command when the
Agent Window does not need to interrupt the user's current work; it is not a flag on other commands.
Run `bsk doctor` when startup or transport problems persist after one retry.
## Stop when the goal is met
## Work toward one observable goal
Every task is a **bounded goal**, not open-ended browsing. The goal may come from the user's request, a recorded `trace.json`, or both.
- Derive a concrete success condition from the user's request or a supplied trace.
- Take the shortest purposeful path: observe, act, then make at most one observation to confirm an
ambiguous result.
- Once success is visible, do not click, refresh, navigate, switch tabs, or perform extra checks.
- If a human-only step appears or two attempts make no progress, request help instead of
brute-forcing.
1. **Define success first** — one concrete, observable condition derived from the user's words, `purpose`, or the last meaningful step in a trace (e.g. "form submitted", "item added to cart", "playback started").
2. **Take the shortest path** — snapshot → act → at most one check. Do not wander, re-try unrelated actions, or stack exploratory steps.
3. **Stop as soon as success is reached** — run `bsk session stop <id>` immediately unless the user explicitly asked to keep the session open (e.g. "don't close yet", "keep browsing").
4. **No post-success work** — once the goal is met, do not click, refresh, navigate, re-search, switch tabs, or "double-check" that it worked. Further verification is a new task.
5. **When blocked, pause — do not brute-force** — if the page requires human input (login, captcha, OTP, payment confirmation) or an action fails twice with no progress, call `bsk request-help` instead of retrying blindly. See **Ask the human for help** below.
6. **When unsure** — at most one extra `bsk snapshot`. If success looks met, stop. If not, ask the user; do not keep clicking.
With a trace, follow its semantic target information and values in order, but treat its refs as
record-local hints. Stop when its purpose or last meaningful effect is satisfied. A trace guides the
task; it does not expand the user's goal or authorize additional actions.
**With a trace:** replay steps in order using `target` role/name/tag and raw `value`/`selection` fields. After the last step (or when its `effect.navigated_to` / success hint is satisfied), apply rules 3–4 immediately. The trace guides execution; it does not extend control beyond the goal.
## Observe, act, observe
**Without a trace:** the user's request *is* the success condition. Satisfying it ends the task — same stop rules apply.
Use this default loop:
## Core interaction loop
Write operations only affect tabs in the **Agent Window** (or tabs you **borrowed** into it).
```
```text
bsk navigate <url> --session <id>
bsk observe --session <id> → primary semantic VOM view; reveals hover/focus surfaces
bsk snapshot --session <id> → static aria tree fallback when VOM is insufficient
bsk hover @e3 --session <id> → reveal hover-triggered menus before re-observing/clicking
bsk click @e4 --session <id> → or bsk fill, bsk select, bsk press
bsk observe --session <id> → again after navigation / DOM change
bsk observe --session <id>
bsk click|hover|fill|select|press ... --session <id>
bsk observe --session <id> # after navigation or a meaningful DOM change
```
**Refs invalidate after navigation** — always re-snapshot before clicking, filling, or selecting on a new page.
Prefer fresh `@eN` refs over CSS selectors. Navigation invalidates refs; large DOM changes may also
make them stale. Observe again before the next interaction.
Prefer `@eN` refs from the latest snapshot over raw CSS selectors. Use `--ref` / `--selector` when ambiguous (`bsk click --help`).
An observation marks a hover-only surface as `@e1 button "Products" [hover first: Shoes | Bags]`.
The listed items are labels, not usable refs: hover the trigger, observe again, then act on the
revealed item's own ref. Do not click the trigger itself unless the user wants the trigger's action.
`[has-submenu]` and `[expanded]` mark the same kind of trigger without listing what it hides.
When VOM renders `[hover first: …]` on an element, the listed items are not currently clickable refs. Run `bsk hover <that-ref> --session <id>`, then immediately run `bsk snapshot` or `bsk observe` again and click the newly visible menu item ref. Do not click the trigger itself unless the user explicitly wants the trigger action.
`bsk observe` does not hover the page on its own. Reach for `--probe-hover` when a control you have
good reason to expect is absent **and** no marker points at a trigger — that combination is what a
CSS-only hover menu looks like from here. It hovers a bounded set of likely triggers, so it costs a
few seconds and touches the live page; once you know which element hides the menu, `bsk hover <ref>`
is cheaper and more precise.
## Observation priority
Escalate page reading only as needed:
Start with `bsk observe` to understand page structure, text, controls, element refs, and conditional hover/focus surfaces. Use `bsk snapshot` only when you need the stricter static accessibility tree or VOM is insufficient. Only escalate to raw HTML or screenshots when the latest observation cannot answer the question:
1. `bsk observe` for normal semantic understanding, text, controls, and refs.
2. `bsk observe --probe-hover` once when an expected control is missing and no marker points at a
trigger.
3. `bsk snapshot` when a stricter static accessibility tree is more useful.
4. `bsk get-html` for exact markup or hidden metadata that semantic views cannot provide.
5. `bsk screenshot` for layout, styling, canvas, images, or requested visual evidence.
1. `bsk observe` — primary semantic VOM observation; may run bounded perception probes such as hover-surface discovery
2. `bsk snapshot` — strict static accessibility tree fallback
3. `bsk get-html` — when hidden DOM, metadata, or markup details are required
4. `bsk screenshot` — when visual layout, canvas/image content, or styling cannot be inferred from the observation. Use `--ref @eN` (from the latest snapshot/observe) to crop to one element; omit `--ref` for the full visible tab.
Do not start with raw HTML or screenshots merely to discover ordinary controls. When interaction is
needed, obtain a fresh observation before acting on screenshot or HTML findings.
Do **not** call `bsk get-html` or `bsk screenshot` first just to inspect a page.
## Respect the Agent Window boundary
## Sandbox rules
Normal page writes affect only Agent Window tabs. To operate a user tab, first list it with
`bsk tab list --scope user --session <id>`, then `bsk tab borrow <tab-id>`. Return it immediately
after the relevant step with `bsk tab return <tab-id>`; never invent a tab id or keep a personal tab
borrowed across unrelated work.
| Rule | Detail |
|------|--------|
| Agent Window | `bsk tab create`, `bsk navigate`, `bsk click`, etc. work on agent tabs by default |
| User tabs | Read-only until borrowed: `bsk tab list --session <id> --scope user` then `bsk tab borrow <tab-id> --session <id>` |
| Return borrowed tabs | Call `bsk tab return <tab-id> --session <id>` when finished; unreturned tabs are **auto-returned** on `bsk session stop` |
| Writes off-agent | Commands that mutate the page fail if the tab is not in the Agent Window — borrow or create a tab first |
## Ask the human when needed
## Global flags
Use `bsk request-help` for login, captcha, OTP, payment confirmation, consent, or another step the
user must complete. Give a precise prompt and pass fresh `--target` refs/selectors when concrete
controls can be highlighted. Use completion criteria only when the page has a clear stable success
signal.
| Flag | Purpose |
|------|---------|
| `--json` | Machine-readable JSON on stdout (errors too) |
| `--quiet` | Suppress informational stderr |
| `-v` / `-vv` | More verbose logging |
The result `outcome` is one of `continued`, `completed`, `cancelled`, `timed_out`, or `disabled`
(`navigated` is deprecated — never treat navigation as a completion signal). Resume only after
`continued` or `completed`. Treat `cancelled` as rejection, and `timed_out` or `disabled` as a
blocker rather than a reason to retry. After control returns, run a fresh `bsk observe` before
reasoning about the page or using refs.
Auto-update is **on by default** — the background daemon upgrades `bsk` itself when a new release is available (postponed while a session is active). Set `BSK_AUTO_UPDATE=off` to disable it and upgrade manually with `bsk update`.
## Command inventory
Command-specific flags (timeouts, `--tab-id`, `--wait-until`, …): **`bsk <cmd> --help`**
This list of names is complete. Never invent a command outside it; read
`bsk <command...> --help` for flags instead of guessing them.
## CLI command reference (one line each)
Details and flags: **`bsk <cmd> --help`**
### Diagnostics
| Command | Summary |
|---------|---------|
| `bsk status` | Connection health, connected browsers, active sessions |
| `bsk doctor` | Deep diagnostics and repair hints |
| `bsk browsers` | List connected browser instances (ids, labels, versions) |
### Session
| Command | Summary |
|---------|---------|
| `bsk session start` | Open Agent Window (`--width`/`--height` for initial size); prints **4-letter session id** |
| `bsk session start --no-focus` | Open Agent Window in the background without stealing focus |
| `bsk session stop <id>` | End session, close Agent Window, auto-return borrowed tabs |
| `bsk session stop --all` | Stop every active session |
| `bsk session list` | List active sessions |
### Window (require `--session <id>`)
| Command | Summary |
|---------|---------|
| `bsk window resize` | Resize the Agent Window (`--width`, `--height`; 100..=7680 CSS px) |
### Device emulation — `bsk emulate` (requires `--session <id>`)
Emulate a mobile device environment on the agent tab via CDP — viewport, User-Agent, and touch — to debug a page's mobile layout and behaviour:
```bash
bsk emulate --session <id> --device iphone-14
bsk emulate --session <id> --width 390 --height 844 --dpr 3 --mobile --ua "Mozilla/5.0 (iPhone…" --touch
bsk emulate --session <id> --off
```text
session start|stop|list browsers status doctor update logs
navigate navigate-back navigate-forward reload wait-for-navigation wait-ms
observe snapshot get-html screenshot console network
click hover fill select press evaluate
tab list|create|close|select|borrow|return window resize emulate
upload download request-help record start|stop
```
- Presets (`--device`): `iphone-14`, `iphone-14-pro-max`, `iphone-se`, `pixel-7`, `galaxy-s23`, `ipad-mini`, `galaxy-tab-s8`. Manual flags (`--width`/`--height`/`--dpr`/`--mobile`/`--ua`/`--accept-language`/`--touch`/`--max-touch-points`) also work without a preset, or override individual preset fields; `--no-mobile`/`--no-touch` turn a preset's mobile viewport / touch emulation back off.
- Repeated runs merge field by field onto the tab's current emulation state — only the flags you pass change. E.g. after `--device iphone-14`, `bsk emulate --session <id> --width 390 --height 844` keeps the preset's dpr (`3`) and mobile viewport. (The extension remembers the state per tab; after an extension reload the next run applies only the fields it carries.)
- `--off` clears every override (viewport, UA, touch) and the remembered state, restoring the tab's real environment.
- Scope: overrides apply to **one tab only** (CDP per-target) and are **not inherited by new tabs** — re-run `bsk emulate` after opening or switching to another tab (default target is the session's active tab; `--tab-id` overrides).
- Emulation covers viewport/UA/touch only; it does not throttle the network, fake geolocation, or synthesise real touch-event streams.
Required flags that are easy to get wrong:
### Tabs (require `--session <id>`)
| Command | Summary |
|---------|---------|
| `bsk tab list` | List tabs (`--scope user\|agent\|all`, default `all`) |
| `bsk tab create` | New tab in Agent Window (`--url`, `--no-active`, `--index`) |
| `bsk tab close <tab-id>` | Close an agent tab |
| `bsk tab select <tab-id>` | Focus an agent tab |
| `bsk tab borrow <tab-id>` | Move a user tab into the Agent Window |
| `bsk tab return <tab-id>` | Return a borrowed tab to its original window |
### Observation (require `--session` unless noted)
| Command | Summary |
|---------|---------|
| `bsk snapshot` | First-choice static page understanding: accessibility tree with `@eN` element refs |
| `bsk observe` | Semantic VOM observation with bounded perception probes for conditional surfaces |
| `bsk get-html` | Raw HTML dump after snapshot is insufficient (high token cost) |
| `bsk screenshot` | PNG capture after snapshot is insufficient: full visible tab, or `--ref @eN` to crop to one element (`--out` path optional) |
### Console & network debugging (read-only; require `--session`)
| Command | Summary |
|---------|---------|
| `bsk console` | Buffered page console messages, JS exceptions, and browser log entries (`--include-stack` for stack traces) |
| `bsk network` | Buffered network responses (status, method, URL, MIME/resource type) and failures (`net::ERR_*` reason) |
Both capture from the moment the tab is attached and read a bounded per-tab buffer: `--since <seq>` pages from a cursor (`next_since` in the result), `--limit` (default 50, max 200), `--max-text-chars` (default 1000, max 4096), `--tab-id` to target a non-active tab. Both are strictly read-only — they never intercept or modify traffic, and request/response headers, bodies, and timings are not captured.
### Navigation
| Command | Summary |
|---------|---------|
| `bsk navigate <url>` | Go to URL in agent tab (`--wait-until`, `--timeout`) |
| `bsk navigate-back` | History back one step |
| `bsk navigate-forward` | History forward one step |
| `bsk reload` | Reload current tab (`--hard` bypass cache) |
(`bsk navigate back` / `bsk navigate forward` are equivalent subcommands.)
### Interaction
| Command | Summary |
|---------|---------|
| `bsk click <ref-or-selector>` | Click element (`--button`, `--click-count`, `--modifiers`) |
| `bsk hover <ref-or-selector>` | Move the mouse to an element and wait for hover UI to settle (`--settle`, `--modifiers`) |
| `bsk fill <ref-or-selector> --value <text>` | Clear and type into input |
| `bsk select <ref-or-selector> --value <v>` | Set `<select>` option(s) by `value` (repeat `--value` for multi-select) |
| `bsk press <key>` | Key/combo (`Enter`, `Ctrl+A`, …; optional `--ref` to focus first) |
### Scripting & timing
| Command | Summary |
|---------|---------|
| `bsk evaluate <expression>` | Run JS in agent tab (see red lines); JS throw → stderr, **exit 0** |
| `bsk wait-for-navigation` | Block until load/DOM idle/etc. (`--wait-until`, `--timeout`) |
| `bsk wait-ms <duration>` | Sleep (`500ms`, `2s`, `1m`; **no** `--session`) |
### Ask the human for help — `bsk request-help`
When a step needs a human (captcha, login, OTP) or you want the user to
confirm an important action, pause and ask:
bsk request-help --session <id> --prompt "Solve the captcha, then click Done only after the site accepts it" \
--title "Captcha required" --target @e7 --target "#submit" --timeout 5m
- `--prompt` (required): what the user should do.
- `--title` (optional): custom title for the overlay panel. When omitted,
the extension shows its default localized title.
- `--target` (repeatable): a snapshot ref (`@e7`) or CSS selector
(`#submit`) to scroll to and flash-highlight. **Strongly recommended** —
whenever the prompt refers to a concrete element (a button to click, a
field to fill, a checkbox to toggle), pass its `@eN` ref / selector so the
user is guided straight to the right spot instead of hunting for it. For
interaction scenarios, always include the relevant target(s); reserve a
prompt with no `--target` for cases where there is genuinely no specific
element to point at (e.g. "wait for the page to finish loading").
- `--timeout` (default `5m`): how long to wait.
- `--completion-criteria` (optional): JSON success detector. Use it only
when there is a concrete post-help success signal, e.g.
`{"any":[{"url_contains":"/dashboard"},{"selector_exists":"[data-testid='account-menu']"}],"stable_for_ms":1000}`.
The target tab is brought to the foreground; the page stays interactive
while the agent control mask is hidden. The call blocks until the user
explicitly acts, the timeout expires, cancellation arrives, or explicit
completion criteria match. Page reloads, SPA route changes, and captcha
refreshes do not return control by themselves. The result `outcome` is one of:
- `continued` — the user finished and clicked Done / return control (treat as confirm).
- `cancelled` — the user clicked Cancel (treat as reject/abort).
- `timed_out` — nobody acted within the timeout.
- `completed` — the explicit `--completion-criteria` matched while the user had control.
- `navigated` — deprecated legacy outcome. Do not rely on navigation as a completion signal.
`note` carries any text the user typed back. `resolved_targets` reports
which refs/selectors matched a live element.
`request-help` does not refresh the page model after the user returns
control. After a `continued` or `completed` result, issue a separate
observation tool call (usually `bsk snapshot --session <id>`) before using
new refs or reasoning about the post-help page state.
#### Disabling request-help (unattended mode)
Set `BSK_REQUEST_HELP=off` on unattended servers: `bsk request-help` then
returns immediately with `outcome="disabled"` (no overlay, no waiting,
exit 0). Any other value keeps it enabled. If you get `disabled`, do not
retry — complete the task autonomously or stop gracefully.
### Recording — `bsk record`
Capture the user's own actions in the Agent Window to a **trace bundle**, for later LLM-driven automation. New CLI builds request **trace v3** (page observations + action chain); older extensions may still return **trace v2** (actions only), which the CLI exports as a single `trace.json` without `states/`.
```bash
bsk record start --browser <instance-id-or-label> \
[--url https://…] [--purpose "publish a wiki doc"] \
[--max-page-tokens 3000] [--redact-values] \
[--output trace]
# `--url` is optional; default https://example.com/ when omitted (must be http(s)).
# Blocks until the user clicks Finish in the recording panel, then writes:
# trace/trace.json — action chain (+ state index when v3)
# trace/states/ — v3 only: one `sN.txt` observe snapshot per settled page state
bsk record stop [--output trace] # terminal fallback if the browser panel is unavailable
```text
bsk fill <ref> --value <text> bsk select <ref> --value <option-value>
bsk screenshot --out <path> bsk emulate --device <preset-id>
bsk upload <ref> --file <path> bsk download <ref> --out <path>
```
- **v3 bundle (preferred):** `--output` is a directory (default `./trace`) containing `trace.json` and `states/`. `trace.json` lists `states[]` (page observation ids) and `steps[]` (each step binds `state` = observe snapshot *before* the action and `result.state` = snapshot *after* settle). Page bodies live in `states/sN.txt` using the same VOM format as `bsk observe`.
- **v2 fallback:** when the connected extension is older, export may contain only legacy `trace.json` with `pages[]` action context and no `states/` observation files. Update the extension for full v3 bundles.
- Each `states[]` entry is one **settled page observation** (captured after recording start, navigation landing, or action settle — not on a timer).
- `target.ref` values like `@e12` exist **only inside** the bundle for disambiguation; do **not** copy `@eN` refs into SKILL.md or agent runbooks — use visible names from the observation text instead.
- `--purpose` is optional context metadata; it does **not** change what gets captured.
- `--redact-values` masks all form values in page files as `[filled]` / `[empty]`.
- Do **not** record on banking/SSO/password-manager pages; passwords are redacted but traces may still contain sensitive text.
`select` matches an option's `value` attribute, not its visible label. Device preset ids are
lowercase and hyphenated, such as `iphone-14`.
## Error handling
- `console` and `network` provide bounded, read-only debugging evidence.
- `emulate` applies viewport, user-agent, and touch overrides to one tab; new tabs do not inherit
them. Use `--off` to restore the real environment.
- `evaluate` is a last resort when observe plus normal interactions cannot complete the task. With
`--json`, inspect `.ok`: a JavaScript exception may still have CLI exit code 0 because the RPC
succeeded. Never evaluate credential surfaces to read storage, cookies, or auth data.
- `record` captures a user's actions for later replay. Read `bsk record start --help` before use,
and never record banking, SSO, password-manager, or other sensitive pages.
### Exit codes (`echo $?` after `bsk …`)
## File transfer
| Code | Meaning | What to do |
|------|---------|------------|
| `0` | Success (including `evaluate` where JS threw but RPC succeeded) | Continue |
| `1` | User error — bad args, unknown session, tab not in Agent Window, stale ref | Fix args; `bsk session list`; re-snapshot |
| `2` | Protocol / transport — service unreachable, IPC failure | `bsk doctor`; check extension connected; retry the command |
| `3` | Browser / CDP execution failed | Retry; simplify selector; check tab still open |
| `4` | Timeout | Increase `--timeout`; try `--wait-until domcontentloaded` |
| `5` | Version skew (CLI vs extension) | Upgrade/reinstall matching versions |
`upload` and `download` stage files through the daemon; the agent never touches browser-internal
paths. Treat upload as disclosure to the website, download as accepting website-controlled bytes.
Human errors print `error:` + `hint:` on stderr; `--json` includes `code`, `message`, `hint`, `exit_code`.
Upload has two independent mechanisms — choose explicitly, never rely on automatic fallback:
### When to run diagnostics
- **Default (input mode):** for upload buttons, file-input labels, or "upload from computer"
actions. The command clicks the target and intercepts the native file chooser.
- **`--mode drop`:** for reliably identified attachment-receiving areas — an explicit drop zone,
chat composer, email editor, or form attachment area. Do not target page whitespace, generic
containers, or areas whose attachment ownership is ambiguous.
| Situation | Command |
|-----------|---------|
| Before first task in a session | `bsk status` — extension connected? |
| Any failure you cannot fix in one retry | `bsk doctor` |
| Multiple browsers / wrong target | `bsk browsers` then `bsk session start --browser <id>` |
Decision sequence when uploading:
Always **`bsk session stop <id>`** in a `finally`-style path so the Agent Window closes and borrowed tabs return.
1. Try input mode (the default).
2. If it returns `reason=file_input_not_activated` with `effect_state=none`, re-observe. When a
reliable attachment target exists, try `--mode drop` once against that target.
3. Otherwise fall back to `request-help`.
4. **Never** switch mechanisms or repeat when `effect_state` is `unknown` or `committed` — the
browser may already have applied the file.
## Red lines
A successful drop means Chrome dispatched the native file-drop event; it does not prove the site
accepted the attachment. Observe the page once after the command.
1. **No token theft** — do not `bsk evaluate` on sensitive sites to read `localStorage`, cookies, or auth headers for exfiltration.
2. **No long borrow** — do not leave a user's personal tab in the Agent Window across unrelated tasks.
3. **No skip stop** — always `bsk session stop <id>`; never assume idle timeout will clean up.
4. **No post-success control** — once the user’s goal (or last trace step) is met, do not keep operating the page; stop the session unless they asked to keep it open.
5. **No raw observe escalation before snapshot/observe** — use `bsk snapshot` first; use `bsk observe` when VOM semantics or conditional surfaces help. Only use `bsk get-html` or `bsk screenshot` when snapshot/observe is insufficient. Element screenshots (`--ref @eN`) still require a fresh snapshot/observe ref — never skip observation just to grab a visual.
6. **`evaluate` is powerful and risky** — use only when snapshot + click/fill/select cannot suffice; never on credential surfaces.
Download default-refuses to overwrite; pass `--overwrite` when replacing an existing file is
intended. Read `bsk upload --help` and `bsk download --help` for all flags and error details.
---
## Recover without wandering
**More detail for any command:** `bsk <cmd> --help`
- Stale ref: observe again and retry the intended action once.
- Unknown tab or session: list current tabs/sessions; never guess identifiers.
- Timeout: inspect current page state before deciding whether one longer purposeful wait is useful.
- Unsupported command: continue with available capabilities; suggest updating only when the missing
command is necessary.
- Unrecoverable failure: report the blocker and stop the session in a finally-style path.
The CLI's current help and error hints are authoritative for flags, parameters, and recovery
details.
+75
View File
@@ -0,0 +1,75 @@
//! Atomic visibility boundary for files received into a same-directory temp.
use std::path::Path;
pub fn commit(temp: &Path, out: &Path, overwrite: bool) -> std::io::Result<()> {
if !overwrite {
std::fs::hard_link(temp, out)?;
std::fs::remove_file(temp)?;
return Ok(());
}
replace(temp, out)
}
#[cfg(unix)]
fn replace(temp: &Path, out: &Path) -> std::io::Result<()> {
// POSIX rename replaces an existing non-directory destination atomically.
std::fs::rename(temp, out)
}
#[cfg(windows)]
fn replace(temp: &Path, out: &Path) -> std::io::Result<()> {
use std::os::windows::ffi::OsStrExt;
use windows_sys::Win32::Storage::FileSystem::{
MOVEFILE_REPLACE_EXISTING, MOVEFILE_WRITE_THROUGH, MoveFileExW,
};
let from: Vec<u16> = temp.as_os_str().encode_wide().chain(Some(0)).collect();
let to: Vec<u16> = out.as_os_str().encode_wide().chain(Some(0)).collect();
// SAFETY: both buffers are NUL-terminated and remain alive for the call.
let ok = unsafe {
MoveFileExW(
from.as_ptr(),
to.as_ptr(),
MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH,
)
};
if ok == 0 {
Err(std::io::Error::last_os_error())
} else {
Ok(())
}
}
#[cfg(not(any(unix, windows)))]
fn replace(temp: &Path, out: &Path) -> std::io::Result<()> {
std::fs::rename(temp, out)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn no_replace_never_overwrites_existing_output() {
let dir = tempfile::tempdir().unwrap();
let temp = dir.path().join("new.part");
let out = dir.path().join("out.bin");
std::fs::write(&temp, b"new").unwrap();
std::fs::write(&out, b"old").unwrap();
assert!(commit(&temp, &out, false).is_err());
assert_eq!(std::fs::read(&out).unwrap(), b"old");
}
#[test]
fn overwrite_replaces_existing_output_without_predelete() {
let dir = tempfile::tempdir().unwrap();
let temp = dir.path().join("new.part");
let out = dir.path().join("out.bin");
std::fs::write(&temp, b"new").unwrap();
std::fs::write(&out, b"old").unwrap();
commit(&temp, &out, true).unwrap();
assert_eq!(std::fs::read(&out).unwrap(), b"new");
assert!(!temp.exists());
}
}
+18 -11
View File
@@ -351,10 +351,10 @@ fn check_version_compatible(status: Option<&StatusResult>) -> CheckResult {
}
}
/// `bsk doctor` check: every connected browser should use the same
/// protocol version as the daemon. Minor protocol drift is accepted by
/// the daemon but flagged here so the user can see who needs updating
/// (M10.4).
/// `bsk doctor` check: connected browsers should speak a protocol the
/// daemon accepts. A different protocol string is still a live
/// connection — report Ok so agents keep working. The detail is an
/// upgrade reminder, not a blocker.
///
/// Review M2 (round-1 minor): when no browsers are connected, the
/// check has nothing to compare against, so it now reports
@@ -396,13 +396,12 @@ fn check_browsers_protocol_compatible(status: Option<&StatusResult>) -> CheckRes
})
.collect::<Vec<_>>()
.join(", ");
CheckResult::fail(
CheckResult::ok(
name,
format!(
"{} browser(s) have protocol minor drift from the daemon: {stale}",
"{} browser(s) report a different protocol version (still usable — continue, and upgrade soon): {stale}",
status.version_skew_browsers.len()
),
"upgrade the browser-skill extension or bsk CLI so both sides use the same protocol version",
)
}
@@ -581,7 +580,7 @@ mod m2_tests {
}
#[test]
fn browsers_check_reports_fail_when_skew_present() {
fn browsers_check_reports_ok_when_skew_present() {
let status = fake_status(
vec![BrowserStatusEntry {
instance_id: "alpha".into(),
@@ -605,9 +604,17 @@ mod m2_tests {
}],
);
let check = check_browsers_protocol_compatible(Some(&status));
assert_eq!(check.status, CheckStatus::Fail);
assert_eq!(check.status, CheckStatus::Ok);
assert!(check.detail.contains("alpha"));
assert!(check.hint.is_some());
assert!(
check
.detail
.contains("still usable — continue, and upgrade soon")
);
assert!(
check.hint.is_none(),
"a protocol-version note must not hint to stop"
);
}
#[test]
@@ -635,7 +642,7 @@ mod m2_tests {
}],
);
let check = check_browsers_protocol_compatible(Some(&status));
assert_eq!(check.status, CheckStatus::Fail);
assert_eq!(check.status, CheckStatus::Ok);
assert!(
check
.detail
+147
View File
@@ -0,0 +1,147 @@
//! `bsk download` — capture one browser download and atomically commit it.
use std::fs::OpenOptions;
use std::io::Write;
use std::path::{Path, PathBuf};
use std::time::Duration;
use anyhow::Context;
use base64::Engine;
use bsk_protocol::Method;
use bsk_protocol::tools::{
DownloadParams, DownloadResult, TransferChunkParams, TransferChunkResult, TransferIdParams,
TransferReleaseResult,
};
use clap::Args;
use uuid::Uuid;
use crate::cli::atomic_output;
use crate::cli::ensure_daemon::ensure_daemon;
use crate::cli::error::{CliError, Format};
use crate::cli::interaction::split_target;
use crate::cli::navigate::parse_timeout_ms;
#[derive(Debug, Clone, Args)]
pub struct DownloadArgs {
/// Snapshot ref (`@e3`) or CSS selector for the download trigger.
pub target: Option<String>,
#[arg(long = "ref")]
pub ref_: Option<String>,
#[arg(long = "selector")]
pub selector: Option<String>,
#[arg(long)]
pub out: PathBuf,
#[arg(long)]
pub session: String,
#[arg(long = "tab-id")]
pub tab_id: Option<i64>,
#[arg(long, default_value = "2m", value_parser = parse_timeout_ms)]
pub timeout: u32,
#[arg(long)]
pub overwrite: bool,
}
pub fn dispatch(args: DownloadArgs, format: Format) -> Result<(), CliError> {
if args.out.exists() && !args.overwrite {
return Err(CliError::Local(anyhow::anyhow!(
"output already exists (pass --overwrite to replace it): {}",
args.out.display()
)));
}
let info = ensure_daemon().context("ensure daemon is running")?;
let (ref_, selector) = split_target(args.target, args.ref_, args.selector)?;
let params = DownloadParams {
session_id: args.session,
ref_,
selector,
tab_id: args.tab_id,
timeout_ms: Some(args.timeout),
browser_relative_dir: None,
max_byte_size: None,
};
let reply: DownloadResult = crate::cli::business_rpc::call(
info.sock_path.clone(),
"download",
Method::ToolDownload,
Some(params),
ipc_timeout(args.timeout),
)?;
let transfer_id = reply.transfer_id.clone().ok_or_else(|| {
CliError::Local(anyhow::anyhow!("daemon returned no download transfer id"))
})?;
let write_result = write_transfer(&info.sock_path, &transfer_id, &args.out, args.overwrite);
let _: Result<TransferReleaseResult, CliError> = crate::cli::business_rpc::call(
info.sock_path,
"transfer-release",
Method::TransferRelease,
Some(TransferIdParams { transfer_id }),
Duration::from_secs(5),
);
write_result?;
match format {
Format::Json => {
let mut value = serde_json::to_value(&reply).unwrap();
value["path"] = serde_json::json!(args.out.to_string_lossy());
if let Some(obj) = value.as_object_mut() {
obj.remove("transfer_id");
obj.remove("browser_path");
}
println!("{}", serde_json::to_string_pretty(&value).unwrap());
}
Format::Human => println!("{}", args.out.display()),
}
Ok(())
}
fn write_transfer(sock: &Path, id: &str, out: &Path, overwrite: bool) -> Result<(), CliError> {
let parent = out
.parent()
.filter(|p| !p.as_os_str().is_empty())
.unwrap_or(Path::new("."));
let temp = parent.join(format!(".bsk-download-{}.part", Uuid::new_v4().simple()));
let result = (|| {
let mut file = OpenOptions::new()
.write(true)
.create_new(true)
.open(&temp)
.with_context(|| format!("create temporary download {}", temp.display()))
.map_err(CliError::Local)?;
let mut offset = 0u64;
loop {
let chunk: TransferChunkResult = crate::cli::business_rpc::call(
sock.to_path_buf(),
"transfer-read",
Method::TransferRead,
Some(TransferChunkParams {
transfer_id: id.to_string(),
offset,
data_base64: String::new(),
}),
Duration::from_secs(30),
)?;
let encoded = chunk.data_base64.as_deref().unwrap_or("");
let bytes = base64::engine::general_purpose::STANDARD
.decode(encoded)
.map_err(|e| CliError::Local(anyhow::anyhow!("decode download chunk: {e}")))?;
file.write_all(&bytes)
.map_err(|e| CliError::Local(e.into()))?;
offset = chunk.next_offset;
if chunk.eof {
break;
}
}
file.sync_all().map_err(|e| CliError::Local(e.into()))?;
drop(file);
atomic_output::commit(&temp, out, overwrite)
.with_context(|| format!("atomically commit download to {}", out.display()))
.map_err(CliError::Local)
})();
if result.is_err() {
let _ = std::fs::remove_file(&temp);
}
result
}
fn ipc_timeout(timeout_ms: u32) -> Duration {
Duration::from_millis(u64::from(timeout_ms) + 5_000)
}
+1 -1
View File
@@ -71,7 +71,7 @@ pub(crate) fn parse_modifiers(input: &str) -> Result<Vec<KeyModifier>, String> {
Ok(out)
}
fn split_target(
pub(crate) fn split_target(
positional: Option<String>,
explicit_ref: Option<String>,
explicit_selector: Option<String>,
+11
View File
@@ -2,6 +2,7 @@
use std::time::Duration;
mod atomic_output;
pub mod browser_wait;
pub mod browsers;
pub mod business_rpc;
@@ -9,6 +10,7 @@ pub mod console;
pub mod daemon;
pub mod dialogs;
pub mod doctor;
pub mod download;
pub mod emulate;
pub mod ensure_daemon;
pub mod error;
@@ -31,6 +33,7 @@ pub mod snapshot;
pub mod status;
pub mod tab;
pub mod update;
pub mod upload;
pub mod waits;
pub mod window;
@@ -38,6 +41,7 @@ use clap::{Args, Parser, Subcommand};
use crate::cli::console::ConsoleArgs;
use crate::cli::daemon::DaemonCmd;
use crate::cli::download::DownloadArgs;
use crate::cli::emulate::EmulateArgs;
use crate::cli::evaluate::EvaluateArgs;
use crate::cli::get_html::GetHtmlArgs;
@@ -53,6 +57,7 @@ use crate::cli::session::SessionCmd;
use crate::cli::snapshot::SnapshotArgs;
use crate::cli::tab::TabCmd;
use crate::cli::update::UpdateArgs;
use crate::cli::upload::UploadArgs;
use crate::cli::waits::{WaitForNavigationArgs, WaitMsArgs};
use crate::cli::window::WindowCmd;
@@ -177,6 +182,12 @@ pub enum Command {
/// Set `<select>` option values by `value` attribute.
Select(SelectArgs),
/// Upload files through a page file input or explicit drop target.
Upload(UploadArgs),
/// Capture one browser download and write it to a local path.
Download(DownloadArgs),
/// Evaluate a JavaScript expression inside the Agent Window.
Evaluate(EvaluateArgs),
+7
View File
@@ -33,6 +33,12 @@ pub struct ObserveArgs {
/// Include conditional surface probe diagnostics in JSON output.
#[arg(long = "debug-surfaces")]
pub debug_surfaces: bool,
/// Actively hover page controls to reveal hover-only menus and tooltips.
/// Costs seconds of wall clock and touches the live page, so it is off
/// unless a static observation looks like it is missing hover content.
#[arg(long = "probe-hover")]
pub probe_hover: bool,
}
pub fn dispatch(args: ObserveArgs, format: Format) -> Result<(), CliError> {
@@ -47,6 +53,7 @@ fn run(sock: PathBuf, args: ObserveArgs, format: Format) -> Result<(), CliError>
max_depth: args.max_depth,
max_tokens: args.max_tokens,
debug_surfaces: args.debug_surfaces,
probe_hover: args.probe_hover,
};
let reply: ObserveResult = call(sock, params)?;
match format {
+135
View File
@@ -49,6 +49,15 @@ pub mod reason {
pub const TAB_NOT_ACTIVE: &str = "tab_not_active";
pub const BORROW_CONFLICT: &str = "borrow_conflict";
pub const SCREENSHOT_CAPTURE_FAILED: &str = "screenshot_capture_failed";
pub const FILE_INPUT_PROBE_FAILED: &str = "file_input_probe_failed";
pub const FILE_INPUT_NOT_ACTIVATED: &str = "file_input_not_activated";
pub const SET_FILE_INPUT_FAILED: &str = "set_file_input_failed";
pub const UPLOAD_MECHANISM_UNSUPPORTED: &str = "upload_mechanism_unsupported";
pub const FILE_DROP_TARGET_UNAVAILABLE: &str = "file_drop_target_unavailable";
pub const FILE_DROP_FAILED: &str = "file_drop_failed";
pub const DOWNLOAD_CAPTURE_FAILED: &str = "download_capture_failed";
pub const TRANSFER_OUTCOME_UNKNOWN: &str = "transfer_outcome_unknown";
pub const TRANSFER_TIMEOUT: &str = "transfer_timeout";
pub const SESSION_BUSY: &str = crate::rpc_reason::SESSION_BUSY;
pub const RECORD_START_PAGE_UNREACHABLE: &str = "record_start_page_unreachable";
}
@@ -197,6 +206,20 @@ pub fn info_for_error(code: ErrorCode, data: Option<&serde_json::Value>) -> Rend
return base;
};
match (code, reason) {
(_, reason::TRANSFER_OUTCOME_UNKNOWN) => RenderInfo {
summary: "the file transfer outcome could not be confirmed",
hint: Some(
"do not retry the transfer: the browser may already have applied it; inspect the page or stop safely",
),
exit_code: base.exit_code,
},
(ErrorCode::Timeout, reason::TRANSFER_TIMEOUT) => RenderInfo {
summary: "the file transfer timed out after browser dispatch",
hint: Some(
"do not retry when effect_state is unknown or committed; inspect the page before taking another action",
),
exit_code: base.exit_code,
},
(ErrorCode::PermissionDenied, reason::ELEMENT_NOT_VISIBLE) => RenderInfo {
summary: "target element has no visible geometry",
hint: Some(
@@ -280,6 +303,60 @@ pub fn info_for_error(code: ErrorCode, data: Option<&serde_json::Value>) -> Rend
),
exit_code: base.exit_code,
},
(ErrorCode::Unsupported, reason::FILE_INPUT_NOT_ACTIVATED) => RenderInfo {
summary: "the upload trigger did not activate a file input",
hint: Some(
"the page may use a non-input picker such as showOpenFilePicker(); when effect_state is none, re-observe and use `--mode drop` once only for a reliably identified attachment-receiving area — otherwise use `bsk request-help` with the exact original local path",
),
exit_code: base.exit_code,
},
(ErrorCode::Timeout | ErrorCode::CdpFailed, reason::FILE_INPUT_PROBE_FAILED) => {
RenderInfo {
summary: "the browser upload transaction could not be established",
hint: Some(
"do not retry the same upload blindly; use `bsk request-help` when human help is available",
),
exit_code: base.exit_code,
}
}
(ErrorCode::Timeout | ErrorCode::CdpFailed, reason::SET_FILE_INPUT_FAILED) => RenderInfo {
summary: "the browser could not attach the staged file to the input",
hint: Some(
"check that BrowserSkill has Chrome's 'Allow access to file URLs' permission; otherwise use `bsk request-help`",
),
exit_code: base.exit_code,
},
(ErrorCode::Unsupported, reason::UPLOAD_MECHANISM_UNSUPPORTED) => RenderInfo {
summary: "the browser does not support native file-drop upload",
hint: Some(
"do not retry the same drop; use a standard file-input target or `bsk request-help`",
),
exit_code: base.exit_code,
},
(
ErrorCode::PermissionDenied | ErrorCode::Timeout | ErrorCode::CdpFailed,
reason::FILE_DROP_TARGET_UNAVAILABLE,
) => RenderInfo {
summary: "the file-drop target could not be safely resolved",
hint: Some(
"rerun observe and select the attachment-receiving editor or drop zone itself; do not guess another target — no file was delivered when effect_state is none",
),
exit_code: base.exit_code,
},
(ErrorCode::Timeout | ErrorCode::CdpFailed, reason::FILE_DROP_FAILED) => RenderInfo {
summary: "the browser could not complete the native file drop",
hint: Some(
"do not retry when effect_state is unknown; observe the page first, otherwise use `bsk request-help`",
),
exit_code: base.exit_code,
},
(ErrorCode::CdpFailed, reason::DOWNLOAD_CAPTURE_FAILED) => RenderInfo {
summary: "the browser download could not be attributed or completed",
hint: Some(
"do not retry blindly; the target browser may not expose a reliable download intent signal",
),
exit_code: base.exit_code,
},
_ => base,
}
}
@@ -419,6 +496,64 @@ mod tests {
assert_eq!(info.exit_code, 3);
}
#[test]
fn file_transfer_reasons_render_actionable_fallbacks() {
let unsupported = serde_json::json!({ "reason": reason::FILE_INPUT_NOT_ACTIVATED });
let info = info_for_error(ErrorCode::Unsupported, Some(&unsupported));
assert_eq!(
info.summary,
"the upload trigger did not activate a file input"
);
let hint = info.hint.unwrap();
assert!(hint.contains("--mode drop"));
assert!(hint.contains("reliably identified attachment-receiving area"));
assert!(hint.contains("exact original local path"));
let control = serde_json::json!({ "reason": reason::FILE_INPUT_PROBE_FAILED });
let info = info_for_error(ErrorCode::Timeout, Some(&control));
assert!(
info.summary
.contains("transaction could not be established")
);
assert!(info.hint.unwrap().contains("do not retry"));
let unsupported_drop =
serde_json::json!({ "reason": reason::UPLOAD_MECHANISM_UNSUPPORTED });
let info = info_for_error(ErrorCode::Unsupported, Some(&unsupported_drop));
assert_eq!(
info.summary,
"the browser does not support native file-drop upload"
);
assert!(info.hint.unwrap().contains("standard file-input"));
let unavailable_target =
serde_json::json!({ "reason": reason::FILE_DROP_TARGET_UNAVAILABLE });
let info = info_for_error(ErrorCode::PermissionDenied, Some(&unavailable_target));
assert_eq!(
info.summary,
"the file-drop target could not be safely resolved"
);
assert!(info.hint.unwrap().contains("effect_state is none"));
let failed_drop = serde_json::json!({ "reason": reason::FILE_DROP_FAILED });
let info = info_for_error(ErrorCode::Timeout, Some(&failed_drop));
assert!(info.summary.contains("native file drop"));
assert!(info.hint.unwrap().contains("effect_state is unknown"));
let download = serde_json::json!({ "reason": reason::DOWNLOAD_CAPTURE_FAILED });
let info = info_for_error(ErrorCode::CdpFailed, Some(&download));
assert!(info.summary.contains("download could not be attributed"));
let unknown = serde_json::json!({ "reason": reason::TRANSFER_OUTCOME_UNKNOWN });
let info = info_for_error(ErrorCode::ProtocolError, Some(&unknown));
assert!(info.summary.contains("outcome could not be confirmed"));
assert!(info.hint.unwrap().contains("do not retry"));
let timeout = serde_json::json!({ "reason": reason::TRANSFER_TIMEOUT });
let info = info_for_error(ErrorCode::Timeout, Some(&timeout));
assert!(info.summary.contains("timed out after browser dispatch"));
}
#[test]
fn ref_not_found_overrides_not_found_copy() {
let data = serde_json::json!({ "reason": reason::REF_NOT_FOUND });
+2 -2
View File
@@ -65,7 +65,7 @@ fn render_human(s: &StatusResult) {
let bold = "\x1b[1m";
let reset = "\x1b[0m";
eprintln!(
"{bold}{yellow}warning:{reset} {} browser(s) have protocol version drift from the daemon{reset}",
"{bold}{yellow}note:{reset} {} browser(s) report a different protocol version; still usable — continue, and run `bsk update` soon{reset}",
s.version_skew_browsers.len()
);
for skew in &s.version_skew_browsers {
@@ -75,7 +75,7 @@ fn render_human(s: &StatusResult) {
skew.label.clone()
};
eprintln!(
" {} ({}) — protocol ext {} vs daemon {} (app ext v{}, daemon v{}) — please align protocol versions",
" {} ({}) — protocol ext {} vs CLI {} (app ext v{}, CLI v{}) — still usable; please upgrade",
skew.instance_id,
label,
display_protocol(&skew.client_protocol_version),
+5 -2
View File
@@ -761,6 +761,7 @@ fn stage_windows_replacement(
.with_context(|| format!("write {}", paths.script_path.display()))?;
const CREATE_NO_WINDOW: u32 = 0x0800_0000;
const CREATE_NEW_PROCESS_GROUP: u32 = 0x0000_0200;
let command = format!("\"{}\"", paths.script_path.display());
if let Err(err) = Command::new("cmd.exe")
.args(["/D", "/S", "/C"])
@@ -768,7 +769,7 @@ fn stage_windows_replacement(
.stdin(Stdio::null())
.stdout(Stdio::null())
.stderr(Stdio::null())
.creation_flags(CREATE_NO_WINDOW)
.creation_flags(CREATE_NO_WINDOW | CREATE_NEW_PROCESS_GROUP)
.spawn()
{
let _ = std::fs::remove_file(&paths.binary_path);
@@ -806,7 +807,7 @@ fn windows_replacement_script(target: &Path, staged_binary: &Path, restart_daemo
:retry\r\n\
move /Y \"{staged_binary}\" \"{target}\" >nul 2>nul\r\n\
if errorlevel 1 (\r\n\
timeout /t 1 /nobreak >nul\r\n\
ping -n 2 127.0.0.1 >nul\r\n\
goto retry\r\n\
)\r\n\
{restart}\
@@ -1007,6 +1008,8 @@ mod tests {
assert!(script.contains(":retry"));
assert!(script.contains("move /Y"));
assert!(script.contains("ping -n 2 127.0.0.1"));
assert!(!script.contains("timeout"));
assert!(script.contains(r#"start "" /B "C:\Program Files\bsk.exe" daemon start"#));
assert!(script.contains(r#"del /F /Q "%~f0""#));
}
+205
View File
@@ -0,0 +1,205 @@
//! `bsk upload` — stage caller-readable files and attach them to a page.
use std::fs::File;
use std::io::Read;
use std::path::{Path, PathBuf};
use std::time::Duration;
use anyhow::Context;
use base64::Engine;
use bsk_protocol::Method;
use bsk_protocol::tools::{
TransferBeginParams, TransferBeginResult, TransferChunkParams, TransferChunkResult,
TransferIdParams, TransferReadyResult, TransferReleaseResult, UploadFile, UploadMode,
UploadParams, UploadResult,
};
use clap::{Args, ValueEnum};
use crate::cli::ensure_daemon::ensure_daemon;
use crate::cli::error::{CliError, Format};
use crate::cli::interaction::split_target;
use crate::cli::navigate::parse_timeout_ms;
#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, ValueEnum)]
pub enum UploadModeArg {
#[default]
Input,
Drop,
}
impl From<UploadModeArg> for UploadMode {
fn from(value: UploadModeArg) -> Self {
match value {
UploadModeArg::Input => Self::Input,
UploadModeArg::Drop => Self::Drop,
}
}
}
#[derive(Debug, Clone, Args)]
pub struct UploadArgs {
/// Snapshot ref (`@e3`) or CSS selector for the chooser trigger / drop target.
pub target: Option<String>,
#[arg(long = "ref")]
pub ref_: Option<String>,
#[arg(long = "selector")]
pub selector: Option<String>,
/// Local file to upload. Repeat for a multiple-file input.
#[arg(long = "file", required = true)]
pub files: Vec<PathBuf>,
/// Browser mechanism used to deliver the staged files.
#[arg(long, value_enum, default_value_t)]
pub mode: UploadModeArg,
#[arg(long)]
pub session: String,
#[arg(long = "tab-id")]
pub tab_id: Option<i64>,
#[arg(long, default_value = "2m", value_parser = parse_timeout_ms)]
pub timeout: u32,
}
pub fn dispatch(args: UploadArgs, format: Format) -> Result<(), CliError> {
if args.files.len() > crate::daemon::file_transfer::MAX_UPLOAD_FILES {
return Err(CliError::Local(anyhow::anyhow!(
"upload accepts at most {} files",
crate::daemon::file_transfer::MAX_UPLOAD_FILES
)));
}
let info = ensure_daemon().context("ensure daemon is running")?;
let (ref_, selector) = split_target(args.target, args.ref_, args.selector)?;
let mut staged = Vec::new();
for path in &args.files {
match stage_file(&info.sock_path, &args.session, path) {
Ok(file) => staged.push(file),
Err(err) => {
for (id, _) in &staged {
let _ = release(&info.sock_path, id);
}
return Err(err);
}
}
}
let params = UploadParams {
session_id: args.session,
ref_,
selector,
tab_id: args.tab_id,
files: staged
.iter()
.map(|(id, name)| UploadFile {
transfer_id: id.clone(),
name: name.clone(),
staged_path: None,
})
.collect(),
mode: args.mode.into(),
timeout_ms: Some(args.timeout),
};
let result = crate::cli::business_rpc::call::<_, UploadResult>(
info.sock_path.clone(),
"upload",
Method::ToolUpload,
Some(params),
ipc_timeout(args.timeout),
);
// Once tool.upload is dispatched, staging ownership belongs to the
// session. A transport timeout cannot prove that Chrome did not attach
// the file, so releasing here could invalidate a late successful attach.
// Session teardown remains the single cleanup boundary after dispatch.
let reply = result?;
match format {
Format::Json => println!("{}", serde_json::to_string_pretty(&reply).unwrap()),
Format::Human => println!(
"upload ok tab={} files={}",
reply.tab_id,
reply.file_names.join(", ")
),
}
Ok(())
}
fn stage_file(sock: &Path, session: &str, path: &PathBuf) -> Result<(String, String), CliError> {
let mut file = File::open(path)
.with_context(|| format!("open upload file {}", path.display()))
.map_err(CliError::Local)?;
let meta = file
.metadata()
.with_context(|| format!("read upload file metadata {}", path.display()))
.map_err(CliError::Local)?;
if !meta.is_file() {
return Err(CliError::Local(anyhow::anyhow!(
"upload source is not a regular file: {}",
path.display()
)));
}
let name = path
.file_name()
.and_then(|v| v.to_str())
.filter(|v| !v.is_empty())
.ok_or_else(|| CliError::Local(anyhow::anyhow!("upload file has no valid basename")))?
.to_string();
let begin: TransferBeginResult = crate::cli::business_rpc::call(
sock.to_path_buf(),
"transfer-begin",
Method::TransferBegin,
Some(TransferBeginParams {
session_id: session.to_string(),
name: name.clone(),
byte_size: meta.len(),
}),
Duration::from_secs(10),
)?;
let staged = (|| {
let mut offset = 0u64;
let mut buf = vec![0u8; begin.chunk_size as usize];
loop {
let n = file.read(&mut buf).map_err(|e| CliError::Local(e.into()))?;
if n == 0 {
break;
}
let reply: TransferChunkResult = crate::cli::business_rpc::call(
sock.to_path_buf(),
"transfer-chunk",
Method::TransferChunk,
Some(TransferChunkParams {
transfer_id: begin.transfer_id.clone(),
offset,
data_base64: base64::engine::general_purpose::STANDARD.encode(&buf[..n]),
}),
Duration::from_secs(30),
)?;
offset = reply.next_offset;
}
let _: TransferReadyResult = crate::cli::business_rpc::call(
sock.to_path_buf(),
"transfer-finish",
Method::TransferFinish,
Some(TransferIdParams {
transfer_id: begin.transfer_id.clone(),
}),
Duration::from_secs(10),
)?;
Ok::<_, CliError>(())
})();
if let Err(err) = staged {
let _ = release(sock, &begin.transfer_id);
return Err(err);
}
Ok((begin.transfer_id, name))
}
fn release(sock: &Path, id: &str) -> Result<TransferReleaseResult, CliError> {
crate::cli::business_rpc::call(
sock.to_path_buf(),
"transfer-release",
Method::TransferRelease,
Some(TransferIdParams {
transfer_id: id.to_string(),
}),
Duration::from_secs(5),
)
}
fn ipc_timeout(timeout_ms: u32) -> Duration {
Duration::from_millis(u64::from(timeout_ms) + 5_000)
}
+728
View File
@@ -0,0 +1,728 @@
//! Session-scoped, content-transparent staging for upload/download tools.
//!
//! The registry deliberately exposes no arbitrary-path operations. Upload
//! bytes arrive from the invoking CLI in bounded chunks; browser downloads
//! land in a daemon-minted directory and are read back by the invoking CLI.
use std::collections::HashMap;
use std::fs::{self, File, OpenOptions};
use std::io::{Read, Seek, SeekFrom, Write};
use std::path::{Component, Path, PathBuf};
use std::sync::Mutex;
use base64::Engine;
use bsk_protocol::tools::{
TransferBeginParams, TransferBeginResult, TransferChunkParams, TransferChunkResult,
TransferIdParams, TransferReadyResult, TransferReleaseResult,
};
use bsk_protocol::{ErrorCode, RpcError};
use uuid::Uuid;
use super::paths;
pub const TRANSFER_CHUNK_SIZE: u32 = 512 * 1024;
pub const MAX_TRANSFER_BYTES: u64 = 512 * 1024 * 1024;
pub const MAX_UPLOAD_FILES: usize = 20;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum Direction {
Upload,
Download,
}
#[derive(Debug)]
struct Entry {
session_id: String,
direction: Direction,
path: PathBuf,
expected_size: Option<u64>,
written: u64,
ready: bool,
}
/// Browser download path after the daemon has verified that it belongs to
/// the capability minted for this transfer. From this point onward the daemon
/// owns cleanup on every error path; unvalidated arbitrary paths are never
/// removed.
#[derive(Debug)]
struct ValidatedBrowserFile {
path: PathBuf,
parent: PathBuf,
cleanup_armed: bool,
}
impl ValidatedBrowserFile {
fn validate(reported_path: &Path, transfer_id: &str) -> Result<Self, RpcError> {
if !reported_path.is_absolute() {
return Err(permission(
"download path is outside its browser capability",
));
}
let expected_parent = Path::new("BrowserSkill").join(transfer_id);
let parent = reported_path
.parent()
.ok_or_else(|| permission("download path has no parent directory"))?;
if !parent.ends_with(&expected_parent) {
return Err(permission(
"download escaped its browser capability directory",
));
}
for path in [reported_path, parent] {
if fs::symlink_metadata(path)
.map_err(io_error)?
.file_type()
.is_symlink()
{
return Err(permission("download capability path contains a symlink"));
}
}
Ok(Self {
path: reported_path.to_path_buf(),
parent: parent.to_path_buf(),
cleanup_armed: true,
})
}
fn remove_source(mut self) -> Result<(), RpcError> {
remove_if_present(&self.path).map_err(io_error)?;
remove_dir_if_present(&self.parent).map_err(io_error)?;
self.cleanup_armed = false;
Ok(())
}
}
impl Drop for ValidatedBrowserFile {
fn drop(&mut self) {
if !self.cleanup_armed {
return;
}
let _ = remove_if_present(&self.path);
let _ = remove_dir_if_present(&self.parent);
}
}
#[derive(Debug)]
pub struct DownloadStaging {
pub transfer_id: String,
pub browser_relative_dir: String,
}
#[derive(Debug)]
pub struct TransferRegistry {
root: PathBuf,
initialized: Mutex<bool>,
entries: Mutex<HashMap<String, Entry>>,
}
impl TransferRegistry {
pub fn new() -> anyhow::Result<Self> {
Ok(Self {
root: paths::bsk_home()?.join("run").join("transfers"),
initialized: Mutex::new(false),
entries: Mutex::new(HashMap::new()),
})
}
#[cfg(test)]
fn at_root(root: PathBuf) -> anyhow::Result<Self> {
let registry = Self {
root,
initialized: Mutex::new(false),
entries: Mutex::new(HashMap::new()),
};
registry.ensure_root()?;
Ok(registry)
}
pub fn initialize(&self) -> anyhow::Result<()> {
self.ensure_root()
}
fn ensure_root(&self) -> anyhow::Result<()> {
let mut initialized = self.initialized.lock().unwrap();
if *initialized {
return Ok(());
}
if self.root.exists() {
// Transfers never survive a daemon; removing stale directories on
// startup gives crash cleanup without a second persistence model.
let _ = fs::remove_dir_all(&self.root);
}
fs::create_dir_all(&self.root)?;
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
fs::set_permissions(&self.root, fs::Permissions::from_mode(0o700))?;
}
*initialized = true;
Ok(())
}
pub fn begin_upload(&self, p: TransferBeginParams) -> Result<TransferBeginResult, RpcError> {
self.ensure_root()
.map_err(|err| protocol_error(err.to_string()))?;
let basename = safe_upload_basename(&p.name)?;
if p.byte_size > MAX_TRANSFER_BYTES {
return Err(invalid(format!(
"file size {} exceeds transfer limit {}",
p.byte_size, MAX_TRANSFER_BYTES
)));
}
let mut entries = self.entries.lock().unwrap();
let session_bytes: u64 = entries
.values()
.filter(|entry| entry.session_id == p.session_id)
.map(|entry| entry.expected_size.unwrap_or(entry.written))
.sum();
if session_bytes.saturating_add(p.byte_size) > MAX_TRANSFER_BYTES {
return Err(invalid(format!(
"session transfer staging exceeds limit {}",
MAX_TRANSFER_BYTES
)));
}
let id = new_id();
let dir = self.root.join(&id);
fs::create_dir(&dir).map_err(io_error)?;
set_private_dir(&dir).map_err(io_error)?;
let path = dir.join(basename);
let file = OpenOptions::new()
.write(true)
.create_new(true)
.open(&path)
.map_err(io_error)?;
set_private_file(&file).map_err(io_error)?;
entries.insert(
id.clone(),
Entry {
session_id: p.session_id,
direction: Direction::Upload,
path,
expected_size: Some(p.byte_size),
written: 0,
ready: false,
},
);
Ok(TransferBeginResult {
transfer_id: id,
chunk_size: TRANSFER_CHUNK_SIZE,
})
}
pub fn write_chunk(&self, p: TransferChunkParams) -> Result<TransferChunkResult, RpcError> {
let bytes = base64::engine::general_purpose::STANDARD
.decode(p.data_base64)
.map_err(|e| invalid(format!("invalid transfer chunk: {e}")))?;
if bytes.len() > TRANSFER_CHUNK_SIZE as usize {
return Err(invalid("transfer chunk exceeds negotiated size"));
}
let mut entries = self.entries.lock().unwrap();
let entry = entries
.get_mut(&p.transfer_id)
.ok_or_else(|| not_found("transfer not found"))?;
if entry.direction != Direction::Upload || entry.ready {
return Err(invalid("transfer is not writable"));
}
if p.offset != entry.written {
return Err(invalid(format!(
"non-sequential transfer offset: expected {}, got {}",
entry.written, p.offset
)));
}
let next = entry.written.saturating_add(bytes.len() as u64);
if next > entry.expected_size.unwrap_or(MAX_TRANSFER_BYTES) {
return Err(invalid("transfer exceeds declared byte size"));
}
let mut file = OpenOptions::new()
.append(true)
.open(&entry.path)
.map_err(io_error)?;
file.write_all(&bytes).map_err(io_error)?;
entry.written = next;
Ok(TransferChunkResult {
next_offset: next,
eof: false,
data_base64: None,
})
}
pub fn finish_upload(&self, p: TransferIdParams) -> Result<TransferReadyResult, RpcError> {
let mut entries = self.entries.lock().unwrap();
let entry = entries
.get_mut(&p.transfer_id)
.ok_or_else(|| not_found("transfer not found"))?;
if entry.direction != Direction::Upload {
return Err(invalid("transfer is not an upload"));
}
if entry.written != entry.expected_size.unwrap_or(entry.written) {
return Err(invalid(format!(
"incomplete transfer: expected {} bytes, received {}",
entry.expected_size.unwrap_or(0),
entry.written
)));
}
entry.ready = true;
Ok(TransferReadyResult {
transfer_id: p.transfer_id,
byte_size: entry.written,
})
}
pub fn resolve_uploads(
&self,
session_id: &str,
ids: &[String],
) -> Result<Vec<PathBuf>, RpcError> {
let entries = self.entries.lock().unwrap();
ids.iter()
.map(|id| {
let entry = entries
.get(id)
.ok_or_else(|| not_found("upload transfer not found"))?;
if entry.session_id != session_id
|| entry.direction != Direction::Upload
|| !entry.ready
{
return Err(permission(
"upload transfer is outside this session or not ready",
));
}
Ok(entry.path.clone())
})
.collect()
}
pub fn begin_download(&self, session_id: &str) -> Result<DownloadStaging, RpcError> {
self.ensure_root()
.map_err(|err| protocol_error(err.to_string()))?;
let id = new_id();
let dir = self.root.join(&id);
fs::create_dir(&dir).map_err(io_error)?;
set_private_dir(&dir).map_err(io_error)?;
self.entries.lock().unwrap().insert(
id.clone(),
Entry {
session_id: session_id.to_string(),
direction: Direction::Download,
path: dir.clone(),
expected_size: None,
written: 0,
ready: false,
},
);
Ok(DownloadStaging {
browser_relative_dir: format!("BrowserSkill/{id}"),
transfer_id: id,
})
}
pub fn import_download(&self, id: &str, reported_path: &Path) -> Result<u64, RpcError> {
let mut entries = self.entries.lock().unwrap();
let entry = entries
.get_mut(id)
.ok_or_else(|| not_found("download transfer not found"))?;
if entry.direction != Direction::Download {
return Err(permission("transfer is not a download capability"));
}
let browser_file = ValidatedBrowserFile::validate(reported_path, id)?;
let canonical = browser_file.path.canonicalize().map_err(io_error)?;
let meta = fs::metadata(&canonical).map_err(io_error)?;
if !meta.is_file() || meta.len() > MAX_TRANSFER_BYTES {
return Err(invalid(
"download is not a regular file or exceeds the transfer limit",
));
}
let destination = entry.path.join("payload");
let source = File::open(&canonical).map_err(io_error)?;
let mut destination_file = OpenOptions::new()
.write(true)
.create_new(true)
.open(&destination)
.map_err(io_error)?;
let copied = std::io::copy(
&mut source.take(MAX_TRANSFER_BYTES.saturating_add(1)),
&mut destination_file,
)
.map_err(io_error)?;
if copied > MAX_TRANSFER_BYTES {
let _ = fs::remove_file(&destination);
return Err(invalid("download exceeds the transfer limit"));
}
entry.path = destination;
entry.written = copied;
entry.expected_size = Some(copied);
entry.ready = true;
browser_file.remove_source()?;
Ok(copied)
}
pub fn read_chunk(&self, p: TransferChunkParams) -> Result<TransferChunkResult, RpcError> {
let entries = self.entries.lock().unwrap();
let entry = entries
.get(&p.transfer_id)
.ok_or_else(|| not_found("transfer not found"))?;
if entry.direction != Direction::Download || !entry.ready {
return Err(invalid("transfer is not readable"));
}
if p.offset > entry.written {
return Err(invalid("read offset is beyond transfer length"));
}
let mut file = File::open(&entry.path).map_err(io_error)?;
file.seek(SeekFrom::Start(p.offset)).map_err(io_error)?;
let mut buf = vec![0u8; TRANSFER_CHUNK_SIZE as usize];
let count = file.read(&mut buf).map_err(io_error)?;
buf.truncate(count);
let next = p.offset + count as u64;
Ok(TransferChunkResult {
next_offset: next,
eof: next >= entry.written,
data_base64: Some(base64::engine::general_purpose::STANDARD.encode(buf)),
})
}
pub fn release(&self, p: TransferIdParams) -> TransferReleaseResult {
let entry = self.entries.lock().unwrap().remove(&p.transfer_id);
if let Some(entry) = entry {
let dir = if entry.path.is_dir() {
entry.path
} else {
entry.path.parent().unwrap_or(&self.root).to_path_buf()
};
let _ = fs::remove_dir_all(dir);
TransferReleaseResult { released: true }
} else {
TransferReleaseResult { released: false }
}
}
pub fn release_session(&self, session_id: &str) {
let ids: Vec<String> = self
.entries
.lock()
.unwrap()
.iter()
.filter(|(_, entry)| entry.session_id == session_id)
.map(|(id, _)| id.clone())
.collect();
for id in ids {
self.release(TransferIdParams { transfer_id: id });
}
}
}
fn new_id() -> String {
format!("tr_{}", Uuid::new_v4().simple())
}
fn set_private_dir(path: &Path) -> std::io::Result<()> {
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
fs::set_permissions(path, fs::Permissions::from_mode(0o700))?;
}
Ok(())
}
fn set_private_file(file: &File) -> std::io::Result<()> {
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
file.set_permissions(fs::Permissions::from_mode(0o600))?;
}
Ok(())
}
fn remove_if_present(path: &Path) -> std::io::Result<()> {
match fs::remove_file(path) {
Ok(()) => Ok(()),
Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(()),
Err(err) => Err(err),
}
}
fn remove_dir_if_present(path: &Path) -> std::io::Result<()> {
match fs::remove_dir(path) {
Ok(()) => Ok(()),
Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(()),
Err(err) => Err(err),
}
}
fn safe_upload_basename(name: &str) -> Result<&str, RpcError> {
if name.is_empty() || name.contains(['/', '\\', '\0']) {
return Err(invalid("upload name must be a safe basename"));
}
let mut components = Path::new(name).components();
match (components.next(), components.next()) {
(Some(Component::Normal(_)), None) => Ok(name),
_ => Err(invalid("upload name must be a safe basename")),
}
}
fn invalid(message: impl Into<String>) -> RpcError {
RpcError {
code: ErrorCode::InvalidParams,
message: message.into(),
data: None,
}
}
fn not_found(message: impl Into<String>) -> RpcError {
RpcError {
code: ErrorCode::NotFound,
message: message.into(),
data: None,
}
}
fn permission(message: impl Into<String>) -> RpcError {
RpcError {
code: ErrorCode::PermissionDenied,
message: message.into(),
data: None,
}
}
fn io_error(err: std::io::Error) -> RpcError {
RpcError {
code: ErrorCode::ProtocolError,
message: err.to_string(),
data: None,
}
}
fn protocol_error(message: impl Into<String>) -> RpcError {
RpcError {
code: ErrorCode::ProtocolError,
message: message.into(),
data: None,
}
}
#[cfg(test)]
mod tests {
use super::*;
fn registry() -> (tempfile::TempDir, TransferRegistry) {
let temp = tempfile::tempdir().unwrap();
let registry = TransferRegistry::at_root(temp.path().join("transfers")).unwrap();
(temp, registry)
}
#[test]
fn upload_is_not_resolvable_until_complete_or_from_another_session() {
let (_temp, registry) = registry();
let begin = registry
.begin_upload(TransferBeginParams {
session_id: "s1".into(),
name: "image.png".into(),
byte_size: 3,
})
.unwrap();
assert!(
registry
.resolve_uploads("s1", std::slice::from_ref(&begin.transfer_id))
.is_err()
);
registry
.write_chunk(TransferChunkParams {
transfer_id: begin.transfer_id.clone(),
offset: 0,
data_base64: base64::engine::general_purpose::STANDARD.encode(b"abc"),
})
.unwrap();
registry
.finish_upload(TransferIdParams {
transfer_id: begin.transfer_id.clone(),
})
.unwrap();
assert!(
registry
.resolve_uploads("s2", std::slice::from_ref(&begin.transfer_id))
.is_err()
);
let [path] = registry
.resolve_uploads("s1", &[begin.transfer_id])
.unwrap()
.try_into()
.unwrap();
assert_eq!(path.file_name().unwrap(), "image.png");
assert_eq!(fs::read(path).unwrap(), b"abc");
}
#[test]
fn upload_name_must_be_one_safe_path_component() {
let (_temp, registry) = registry();
for name in ["", ".", "..", "../secret", "folder/file", "folder\\file"] {
let result = registry.begin_upload(TransferBeginParams {
session_id: "s1".into(),
name: name.into(),
byte_size: 0,
});
assert!(result.is_err(), "accepted unsafe upload name");
}
assert!(registry.entries.lock().unwrap().is_empty());
assert!(fs::read_dir(&registry.root).unwrap().next().is_none());
}
#[test]
fn same_upload_names_are_isolated_by_transfer_directory() {
let (_temp, registry) = registry();
let first = registry
.begin_upload(TransferBeginParams {
session_id: "s1".into(),
name: "image.png".into(),
byte_size: 0,
})
.unwrap();
let second = registry
.begin_upload(TransferBeginParams {
session_id: "s1".into(),
name: "image.png".into(),
byte_size: 0,
})
.unwrap();
let entries = registry.entries.lock().unwrap();
let first_path = &entries[&first.transfer_id].path;
let second_path = &entries[&second.transfer_id].path;
assert_ne!(first_path.parent(), second_path.parent());
assert_eq!(first_path.file_name().unwrap(), "image.png");
assert_eq!(second_path.file_name().unwrap(), "image.png");
}
#[cfg(unix)]
#[test]
fn upload_staging_is_private() {
use std::os::unix::fs::PermissionsExt;
let (_temp, registry) = registry();
let begin = registry
.begin_upload(TransferBeginParams {
session_id: "s1".into(),
name: "private.png".into(),
byte_size: 0,
})
.unwrap();
let entries = registry.entries.lock().unwrap();
let path = &entries[&begin.transfer_id].path;
assert_eq!(
fs::metadata(path.parent().unwrap())
.unwrap()
.permissions()
.mode()
& 0o777,
0o700
);
assert_eq!(
fs::metadata(path).unwrap().permissions().mode() & 0o777,
0o600
);
}
#[test]
fn download_must_import_from_its_browser_capability_directory() {
let (temp, registry) = registry();
let staging = registry.begin_download("s1").unwrap();
let outside = temp.path().join("outside.bin");
fs::write(&outside, b"secret").unwrap();
assert!(
registry
.import_download(&staging.transfer_id, &outside)
.is_err()
);
assert_eq!(fs::read(&outside).unwrap(), b"secret");
let browser_dir = temp
.path()
.join("Downloads")
.join("BrowserSkill")
.join(&staging.transfer_id);
fs::create_dir_all(&browser_dir).unwrap();
let inside = browser_dir.join("result.bin");
fs::write(&inside, b"result").unwrap();
assert_eq!(
registry
.import_download(&staging.transfer_id, &inside)
.unwrap(),
6
);
assert!(!inside.exists());
let chunk = registry
.read_chunk(TransferChunkParams {
transfer_id: staging.transfer_id,
offset: 0,
data_base64: String::new(),
})
.unwrap();
assert_eq!(
base64::engine::general_purpose::STANDARD
.decode(chunk.data_base64.unwrap())
.unwrap(),
b"result"
);
}
#[test]
fn validated_oversized_browser_download_is_removed_on_rejection() {
let (temp, registry) = registry();
let staging = registry.begin_download("s1").unwrap();
let browser_dir = temp
.path()
.join("Downloads")
.join("BrowserSkill")
.join(&staging.transfer_id);
fs::create_dir_all(&browser_dir).unwrap();
let inside = browser_dir.join("oversized.bin");
File::create(&inside)
.unwrap()
.set_len(MAX_TRANSFER_BYTES + 1)
.unwrap();
assert!(
registry
.import_download(&staging.transfer_id, &inside)
.is_err()
);
assert!(!inside.exists());
assert!(!browser_dir.exists());
}
#[test]
fn releasing_a_session_removes_all_staging() {
let (_temp, registry) = registry();
let first = registry.begin_download("s1").unwrap();
let second = registry.begin_download("s2").unwrap();
let first_directory = registry.root.join(&first.transfer_id);
let second_directory = registry.root.join(&second.transfer_id);
registry.release_session("s1");
assert!(!first_directory.exists());
assert!(second_directory.exists());
}
#[test]
fn upload_staging_is_bounded_across_a_session() {
let (_temp, registry) = registry();
registry
.begin_upload(TransferBeginParams {
session_id: "s1".into(),
name: "large.bin".into(),
byte_size: MAX_TRANSFER_BYTES,
})
.unwrap();
assert!(
registry
.begin_upload(TransferBeginParams {
session_id: "s1".into(),
name: "one-more-byte.bin".into(),
byte_size: 1,
})
.is_err()
);
assert!(
registry
.begin_upload(TransferBeginParams {
session_id: "s2".into(),
name: "other-session.bin".into(),
byte_size: 1,
})
.is_ok()
);
}
}
+209 -5
View File
@@ -27,7 +27,8 @@ use bsk_protocol::system::{
VersionSkewEntry,
};
use bsk_protocol::tools::{
HelpOutcome, RequestHelpResult, ReturnFailure, WaitMsParams, WaitMsResult,
DownloadParams, DownloadResult, HelpOutcome, RequestHelpResult, ReturnFailure,
TransferBeginParams, TransferIdParams, UploadParams, WaitMsParams, WaitMsResult,
};
use bsk_protocol::{
CancelParams, CancelResult, ErrorCode, Method, PingResult, ResponseBody, RpcError, RpcId,
@@ -61,6 +62,10 @@ pub type RpcHandler = Arc<
>;
const DEFAULT_RPC_TIMEOUT: Duration = Duration::from_secs(15);
// The upload transaction owns its operation deadline and may need a bounded
// cleanup before it can return a useful structured error. Keep only that
// transport alive slightly longer so it does not replace the result.
const EXTENSION_RESPONSE_GRACE: Duration = Duration::from_secs(2);
/// Upper bound on `wait_for_browser_ms` accepted over IPC.
const MAX_BROWSER_WAIT: Duration = Duration::from_secs(60);
// `session.stop` fast-fails while another tool is active; this budget
@@ -223,6 +228,11 @@ pub fn full_handler(status: DaemonStatus, state: Arc<DaemonState>) -> RpcHandler
Ok(v) => ResponseBody::Ok(v),
Err(e) => ResponseBody::Err(e),
},
Method::TransferBegin => handle_transfer_begin(&state, params),
Method::TransferChunk => handle_transfer_chunk(&state, params),
Method::TransferFinish => handle_transfer_finish(&state, params),
Method::TransferRead => handle_transfer_read(&state, params),
Method::TransferRelease => handle_transfer_release(&state, params),
Method::ToolTabList
| Method::ToolTabCreate
| Method::ToolTabClose
@@ -246,6 +256,8 @@ pub fn full_handler(status: DaemonStatus, state: Arc<DaemonState>) -> RpcHandler
| Method::ToolFill
| Method::ToolPress
| Method::ToolSelect
| Method::ToolUpload
| Method::ToolDownload
| Method::ToolEvaluate
| Method::ToolWaitForNavigation
| Method::ToolRequestHelp
@@ -324,7 +336,7 @@ async fn handle_tool_dispatch(
data: None,
});
}
let timeout = match tool_dispatch_timeout(&params) {
let timeout = match tool_dispatch_transport_timeout(&method, &params) {
Ok(timeout) => timeout,
Err(err) => return ResponseBody::Err(err),
};
@@ -338,24 +350,180 @@ async fn handle_tool_dispatch(
});
}
};
let mut params = params;
let mut download_transfer_id: Option<String> = None;
if method == Method::ToolUpload {
let mut upload: UploadParams = match serde_json::from_value(params) {
Ok(v) => v,
Err(err) => return ResponseBody::Err(invalid_params(err.to_string())),
};
if upload.files.is_empty() || upload.files.len() > super::file_transfer::MAX_UPLOAD_FILES {
return ResponseBody::Err(invalid_params(format!(
"upload requires 1..={} files",
super::file_transfer::MAX_UPLOAD_FILES
)));
}
let ids: Vec<String> = upload.files.iter().map(|f| f.transfer_id.clone()).collect();
let paths = match state.transfers.resolve_uploads(&session_id.0, &ids) {
Ok(v) => v,
Err(err) => return ResponseBody::Err(err),
};
for (file, path) in upload.files.iter_mut().zip(paths) {
file.staged_path = Some(path.to_string_lossy().into_owned());
}
params = serde_json::to_value(upload).unwrap_or(Value::Null);
} else if method == Method::ToolDownload {
let mut download: DownloadParams = match serde_json::from_value(params) {
Ok(v) => v,
Err(err) => return ResponseBody::Err(invalid_params(err.to_string())),
};
let staging = match state.transfers.begin_download(&session_id.0) {
Ok(v) => v,
Err(err) => return ResponseBody::Err(err),
};
download.browser_relative_dir = Some(staging.browser_relative_dir);
download.max_byte_size = Some(super::file_transfer::MAX_TRANSFER_BYTES);
download_transfer_id = Some(staging.transfer_id);
params = serde_json::to_value(download).unwrap_or(Value::Null);
}
let entry = inflight_guard.entry();
// `record_stop` must reach the extension while `record_await` holds the
// serial busy lock — finishing the recording unblocks await.
let outcome = if method == Method::ToolRecordStop {
state
.tool_queues
.dispatch_unlocked(&session_id, method, params, timeout, Some(entry))
.dispatch_unlocked(&session_id, method.clone(), params, timeout, Some(entry))
.await
} else {
state
.tool_queues
.dispatch(&session_id, method, params, timeout, Some(entry))
.dispatch(&session_id, method.clone(), params, timeout, Some(entry))
.await
};
drop(inflight_guard);
match outcome {
Ok(v) if method == Method::ToolDownload => {
let id = download_transfer_id.expect("download transfer allocated");
let mut result: DownloadResult = match serde_json::from_value(v) {
Ok(v) => v,
Err(err) => {
state
.transfers
.release(TransferIdParams { transfer_id: id });
return ResponseBody::Err(RpcError {
code: ErrorCode::ProtocolError,
message: format!("invalid tool.download result: {err}"),
data: None,
});
}
};
let Some(path) = result.browser_path.take() else {
state
.transfers
.release(TransferIdParams { transfer_id: id });
return ResponseBody::Err(RpcError {
code: ErrorCode::ProtocolError,
message: "tool.download returned no browser_path".into(),
data: None,
});
};
match state
.transfers
.import_download(&id, std::path::Path::new(&path))
{
Ok(size) => {
result.byte_size = size;
result.transfer_id = Some(id);
ResponseBody::Ok(serde_json::to_value(result).unwrap_or(Value::Null))
}
Err(err) => {
state
.transfers
.release(TransferIdParams { transfer_id: id });
ResponseBody::Err(err)
}
}
}
Ok(v) => ResponseBody::Ok(v),
Err(err) => ResponseBody::Err(err.into_rpc()),
Err(err) => {
if let Some(id) = download_transfer_id {
state
.transfers
.release(TransferIdParams { transfer_id: id });
}
ResponseBody::Err(err.into_rpc())
}
}
}
fn handle_transfer_begin(state: &Arc<DaemonState>, params: Value) -> ResponseBody {
let p: TransferBeginParams = match serde_json::from_value(params) {
Ok(v) => v,
Err(err) => return ResponseBody::Err(invalid_params(err.to_string())),
};
if state
.sessions
.get(&SessionId(p.session_id.clone()))
.is_none()
{
return ResponseBody::Err(RpcError {
code: ErrorCode::NotFound,
message: format!("session {} unknown", p.session_id),
data: None,
});
}
match state.transfers.begin_upload(p) {
Ok(v) => ResponseBody::Ok(serde_json::to_value(v).unwrap_or(Value::Null)),
Err(err) => ResponseBody::Err(err),
}
}
fn handle_transfer_chunk(state: &Arc<DaemonState>, params: Value) -> ResponseBody {
let p = match serde_json::from_value(params) {
Ok(v) => v,
Err(err) => return ResponseBody::Err(invalid_params(err.to_string())),
};
match state.transfers.write_chunk(p) {
Ok(v) => ResponseBody::Ok(serde_json::to_value(v).unwrap_or(Value::Null)),
Err(err) => ResponseBody::Err(err),
}
}
fn handle_transfer_finish(state: &Arc<DaemonState>, params: Value) -> ResponseBody {
let p = match serde_json::from_value(params) {
Ok(v) => v,
Err(err) => return ResponseBody::Err(invalid_params(err.to_string())),
};
match state.transfers.finish_upload(p) {
Ok(v) => ResponseBody::Ok(serde_json::to_value(v).unwrap_or(Value::Null)),
Err(err) => ResponseBody::Err(err),
}
}
fn handle_transfer_read(state: &Arc<DaemonState>, params: Value) -> ResponseBody {
let p = match serde_json::from_value(params) {
Ok(v) => v,
Err(err) => return ResponseBody::Err(invalid_params(err.to_string())),
};
match state.transfers.read_chunk(p) {
Ok(v) => ResponseBody::Ok(serde_json::to_value(v).unwrap_or(Value::Null)),
Err(err) => ResponseBody::Err(err),
}
}
fn handle_transfer_release(state: &Arc<DaemonState>, params: Value) -> ResponseBody {
let p: TransferIdParams = match serde_json::from_value(params) {
Ok(v) => v,
Err(err) => return ResponseBody::Err(invalid_params(err.to_string())),
};
ResponseBody::Ok(serde_json::to_value(state.transfers.release(p)).unwrap_or(Value::Null))
}
fn invalid_params(message: impl Into<String>) -> RpcError {
RpcError {
code: ErrorCode::InvalidParams,
message: message.into(),
data: None,
}
}
@@ -533,6 +701,16 @@ fn tool_dispatch_timeout(params: &Value) -> Result<Duration, RpcError> {
Ok(Duration::from_millis(u64::from(ms)))
}
fn tool_dispatch_transport_timeout(method: &Method, params: &Value) -> Result<Duration, RpcError> {
tool_dispatch_timeout(params).map(|timeout| {
if matches!(method, Method::ToolUpload | Method::ToolDownload) {
timeout.saturating_add(EXTENSION_RESPONSE_GRACE)
} else {
timeout
}
})
}
// Local CLI-facing shapes. Intentionally distinct from
// `bsk_protocol::tools::SessionStart*` (which describes the WS-facing
// `tool.session_*` round-trip with the extension) so the two sides can
@@ -798,6 +976,7 @@ async fn handle_session_stop(
.await
{
Ok(stop) => {
state.transfers.release_session(&session_id.0);
let result = CliSessionStopResult {
stopped: vec![session_id.0],
failed: Vec::new(),
@@ -897,6 +1076,7 @@ async fn handle_session_stop_all(
.await
{
Ok(stop) => {
state.transfers.release_session(&id.0);
stopped.push(id.0);
returned_tab_ids.extend(stop.returned_tab_ids);
return_failures.extend(stop.return_failures);
@@ -1329,6 +1509,30 @@ mod tests {
assert_eq!(got, std::time::Duration::from_millis(300_000));
}
#[test]
fn extension_transport_outlives_the_operation_deadline() {
let params = serde_json::json!({
"session_id": "abcd",
"timeout_ms": 60_000,
});
let dispatch_timeout =
tool_dispatch_transport_timeout(&Method::ToolUpload, &params).unwrap();
assert_eq!(dispatch_timeout, Duration::from_secs(62));
}
#[test]
fn extension_response_grace_does_not_change_other_tools() {
let params = serde_json::json!({
"session_id": "abcd",
"timeout_ms": 60_000,
});
let dispatch_timeout =
tool_dispatch_transport_timeout(&Method::ToolClick, &params).unwrap();
assert_eq!(dispatch_timeout, Duration::from_secs(60));
}
#[test]
fn tool_dispatch_timeout_rejects_zero_timeout_ms() {
let params = serde_json::json!({
+1
View File
@@ -3,6 +3,7 @@
pub mod abort;
pub mod browsers;
mod cancel_forward;
pub mod file_transfer;
pub mod inflight;
pub mod info;
pub mod ipc;
+226 -27
View File
@@ -438,6 +438,7 @@ async fn dispatch_with_sender(
lifecycle_cancel: Option<AbortToken>,
cancel_cleanup_timeout: Duration,
) -> Result<Value, DispatchError> {
let effect_aware_transfer = is_effect_aware_transfer(&method);
let (respond_tx, respond_rx) = oneshot::channel();
let job = ToolJob {
method,
@@ -474,15 +475,21 @@ async fn dispatch_with_sender(
mpsc::error::TrySendError::Closed(_) => DispatchError::QueueClosed,
});
}
let mut response_timeout = timeout.saturating_add(Duration::from_secs(1));
if lifecycle_cancellable {
// A lifecycle cancel can arrive at the original RPC deadline and
// intentionally keeps the worker alive while extension cleanup
// settles. Do not let this outer waiter abandon that reconciliation
// early, which could reopen a daemon session whose window is gone.
response_timeout = response_timeout.saturating_add(cancel_cleanup_timeout);
}
let waited = tokio::time::timeout(response_timeout, respond_rx).await;
// Lifecycle teardown and effect-aware transfers both keep the worker busy
// for bounded compensation after their original deadline. Keep the outer
// waiter alive for the same grace period so it cannot abandon reconciliation.
let response_grace = if lifecycle_cancellable || effect_aware_transfer {
cancel_cleanup_timeout
} else {
Duration::ZERO
};
let waited = tokio::time::timeout(
timeout
.saturating_add(response_grace)
.saturating_add(Duration::from_secs(1)),
respond_rx,
)
.await;
match waited {
Ok(Ok(Ok(v))) => Ok(v),
Ok(Ok(Err(rpc))) => Err(DispatchError::Rpc(rpc)),
@@ -636,44 +643,85 @@ async fn forward_one(
.lifecycle_cancel
.as_ref()
.map(|_| &forward_lifecycle_cancel as &(dyn Fn() + Sync));
let send_deadline_cancel = || {
client
.sink
.send(Frame::Request(RequestFrame {
id: format!("deadline-cancel-{rpc_id}"),
method: Method::Cancel,
params: Some(serde_json::json!({ "rpc_id": rpc_id })),
}))
.is_ok()
};
let deadline_cancel: Option<&(dyn Fn() -> bool + Sync)> =
is_effect_aware_transfer(&job.method).then_some(&send_deadline_cancel);
let waited = await_with_optional_cancel(
job.timeout,
job.cancel_cleanup_timeout,
waiter,
cancel_token.as_ref(),
on_abort,
deadline_cancel,
)
.await;
let response = match waited {
WaitOutcome::Response(resp) => resp,
WaitOutcome::CancelledAfterResponse(resp) => {
if job.method == Method::ToolSessionStop
&& let ResponseBody::Ok(value) = resp.body
&& let ResponseBody::Ok(value) = &resp.body
{
// Teardown crossed its final irreversible boundary before
// the cancel landed. Commit the real extension result so the
// daemon does not retain a session whose window is gone.
return Ok(value);
return Ok(value.clone());
}
// Cancellation wins the external verdict, but only after the
// extension's original RPC has settled. Preserve any non-cancel
// error so compensation failures remain explicit instead of
// being hidden behind a generic cancelled result.
if let ResponseBody::Err(err) = resp.body
&& !matches!(err.code, ErrorCode::Cancelled | ErrorCode::UserAborted)
{
return Err(err);
// File transfer commits are irreversible. A late cancel cannot
// overwrite a confirmed success, and an unknown transfer effect
// must remain explicit so callers do not retry or release upload
// staging as though nothing happened.
if is_effect_aware_transfer(&job.method) {
match &resp.body {
ResponseBody::Ok(_) => resp,
ResponseBody::Err(err)
if transfer_effect(err) == Some("unknown")
|| transfer_effect(err) == Some("committed") =>
{
return Err(err.clone());
}
ResponseBody::Err(err)
if !matches!(err.code, ErrorCode::Cancelled | ErrorCode::UserAborted) =>
{
return Err(err.clone());
}
ResponseBody::Err(_) => {
return Err(cancelled_error(
job.inflight.as_deref(),
"tool dispatch cancelled after extension cleanup",
));
}
}
} else {
// For ordinary tools cancellation keeps the existing verdict,
// while non-cancel errors still expose compensation failures.
if let ResponseBody::Err(err) = resp.body
&& !matches!(err.code, ErrorCode::Cancelled | ErrorCode::UserAborted)
{
return Err(err);
}
return Err(cancelled_error(
job.inflight.as_deref(),
"tool dispatch cancelled after extension cleanup",
));
}
return Err(cancelled_error(
job.inflight.as_deref(),
"tool dispatch cancelled after extension cleanup",
));
}
WaitOutcome::TimedOutAfterResponse(resp) => match resp.body {
ResponseBody::Ok(value) if job.method == Method::ToolSessionStop => {
ResponseBody::Ok(value)
if job.method == Method::ToolSessionStop
|| is_effect_aware_transfer(&job.method) =>
{
// The close crossed its irreversible boundary during the
// timeout cleanup grace. Reconcile daemon state instead of
// reopening a session whose window is already gone.
// timeout cleanup grace, or the transfer committed before its
// deadline cancel settled. Preserve the irreversible result.
return Ok(value);
}
ResponseBody::Err(err)
@@ -681,6 +729,9 @@ async fn forward_one(
{
return Err(err);
}
ResponseBody::Err(err) if is_effect_aware_transfer(&job.method) => {
return Err(timed_out_transfer_error(&err));
}
_ => {
return Err(RpcError {
code: ErrorCode::Timeout,
@@ -691,6 +742,14 @@ async fn forward_one(
},
WaitOutcome::CleanupTimeout => {
client.pending.lock().unwrap().cancel(&rpc_id);
if is_effect_aware_transfer(&job.method) {
return Err(unknown_transfer_error(
ErrorCode::Timeout,
"cancelled file transfer did not confirm its outcome before cleanup timed out",
"cleanup",
true,
));
}
return Err(RpcError {
code: ErrorCode::Timeout,
message: format!(
@@ -700,8 +759,25 @@ async fn forward_one(
data: Some(serde_json::json!({ "reason": "cancel_cleanup_timeout" })),
});
}
WaitOutcome::TimeoutCleanupFailed => {
client.pending.lock().unwrap().cancel(&rpc_id);
return Err(unknown_transfer_error(
ErrorCode::Timeout,
"file transfer timed out and cleanup could not be confirmed",
"cleanup",
true,
));
}
WaitOutcome::WaiterClosed => {
client.pending.lock().unwrap().cancel(&rpc_id);
if is_effect_aware_transfer(&job.method) {
return Err(unknown_transfer_error(
ErrorCode::ProtocolError,
"file transfer transport closed after dispatch; outcome is unknown",
"transport",
false,
));
}
return Err(RpcError {
code: ErrorCode::ProtocolError,
message: "transport closed mid-call".into(),
@@ -710,6 +786,14 @@ async fn forward_one(
}
WaitOutcome::Timeout => {
client.pending.lock().unwrap().cancel(&rpc_id);
if is_effect_aware_transfer(&job.method) {
return Err(unknown_transfer_error(
ErrorCode::Timeout,
"file transfer timed out after dispatch; outcome is unknown",
"transport",
false,
));
}
return Err(RpcError {
code: ErrorCode::Timeout,
message: format!("tool RPC timed out after {:?}", job.timeout),
@@ -723,12 +807,65 @@ async fn forward_one(
}
}
fn is_effect_aware_transfer(method: &Method) -> bool {
matches!(method, Method::ToolUpload | Method::ToolDownload)
}
fn transfer_effect(err: &RpcError) -> Option<&str> {
err.data.as_ref()?.get("effect_state")?.as_str()
}
fn unknown_transfer_error(
code: ErrorCode,
message: impl Into<String>,
phase: &str,
cleanup_failed: bool,
) -> RpcError {
let mut data = serde_json::json!({
"reason": "transfer_outcome_unknown",
"effect_state": "unknown",
"phase": phase,
});
if cleanup_failed {
data["cleanup_state"] = serde_json::json!("failed");
}
RpcError {
code,
message: message.into(),
data: Some(data),
}
}
fn timed_out_transfer_error(err: &RpcError) -> RpcError {
let data = err.data.as_ref();
RpcError {
code: ErrorCode::Timeout,
message: "file transfer timed out after dispatch".into(),
data: Some(serde_json::json!({
"reason": "transfer_timeout",
"effect_state": data
.and_then(|value| value.get("effect_state"))
.and_then(Value::as_str)
.unwrap_or("unknown"),
"phase": data
.and_then(|value| value.get("phase"))
.and_then(Value::as_str)
.unwrap_or("cleanup"),
"cleanup_state": data
.and_then(|value| value.get("cleanup_state"))
.and_then(Value::as_str)
.unwrap_or("complete"),
})),
}
}
#[derive(Debug)]
enum WaitOutcome {
Response(bsk_protocol::ResponseFrame),
CancelledAfterResponse(bsk_protocol::ResponseFrame),
TimedOutAfterResponse(bsk_protocol::ResponseFrame),
CleanupTimeout,
TimeoutCleanupFailed,
WaiterClosed,
Timeout,
}
@@ -739,6 +876,7 @@ async fn await_with_optional_cancel(
mut waiter: oneshot::Receiver<bsk_protocol::ResponseFrame>,
cancel: Option<&super::abort::AbortToken>,
on_abort: Option<&(dyn Fn() + Sync)>,
on_deadline: Option<&(dyn Fn() -> bool + Sync)>,
) -> WaitOutcome {
match cancel {
Some(token) => {
@@ -764,6 +902,16 @@ async fn await_with_optional_cancel(
Err(_) => WaitOutcome::WaiterClosed,
},
_ = &mut deadline => {
if let Some(send_cancel) = on_deadline {
return if send_cancel() {
match tokio::time::timeout(cleanup_timeout, &mut waiter).await {
Ok(Ok(resp)) => WaitOutcome::TimedOutAfterResponse(resp),
Ok(Err(_)) | Err(_) => WaitOutcome::TimeoutCleanupFailed,
}
} else {
WaitOutcome::TimeoutCleanupFailed
};
}
let Some(on_abort) = on_abort else {
return WaitOutcome::Timeout;
};
@@ -779,7 +927,16 @@ async fn await_with_optional_cancel(
None => match tokio::time::timeout(timeout, &mut waiter).await {
Ok(Ok(resp)) => WaitOutcome::Response(resp),
Ok(Err(_)) => WaitOutcome::WaiterClosed,
Err(_) => WaitOutcome::Timeout,
Err(_) => match on_deadline {
Some(send_cancel) if send_cancel() => {
match tokio::time::timeout(cleanup_timeout, &mut waiter).await {
Ok(Ok(resp)) => WaitOutcome::TimedOutAfterResponse(resp),
Ok(Err(_)) | Err(_) => WaitOutcome::TimeoutCleanupFailed,
}
}
Some(_) => WaitOutcome::TimeoutCleanupFailed,
None => WaitOutcome::Timeout,
},
},
}
}
@@ -864,6 +1021,7 @@ mod await_with_optional_cancel_tests {
rx,
Some(&token),
None,
None,
)
.await;
assert!(
@@ -887,6 +1045,7 @@ mod await_with_optional_cancel_tests {
rx,
Some(&token),
None,
None,
)
.await;
match outcome {
@@ -907,6 +1066,7 @@ mod await_with_optional_cancel_tests {
rx,
None,
None,
None,
)
.await;
assert!(matches!(outcome, WaitOutcome::Response(_)));
@@ -924,6 +1084,7 @@ mod await_with_optional_cancel_tests {
rx,
Some(&token),
None,
None,
)
.await
});
@@ -948,6 +1109,7 @@ mod await_with_optional_cancel_tests {
rx,
Some(&token),
None,
None,
)
.await;
assert!(matches!(outcome, WaitOutcome::CleanupTimeout));
@@ -972,12 +1134,49 @@ mod await_with_optional_cancel_tests {
rx,
Some(&token),
Some(&forward_abort),
None,
)
.await;
assert!(abort_forwarded.load(Ordering::SeqCst));
assert!(matches!(outcome, WaitOutcome::TimedOutAfterResponse(_)));
}
#[tokio::test]
async fn transfer_deadline_sends_cancel_and_waits_for_cleanup_response() {
let (tx, rx) = oneshot::channel();
tokio::spawn(async move {
tokio::time::sleep(Duration::from_millis(5)).await;
tx.send(dummy_response()).unwrap();
});
let send_cancel = || true;
let outcome = await_with_optional_cancel(
Duration::from_millis(1),
Duration::from_secs(1),
rx,
None,
None,
Some(&send_cancel),
)
.await;
assert!(matches!(outcome, WaitOutcome::TimedOutAfterResponse(_)));
}
#[tokio::test]
async fn transfer_deadline_is_unknown_when_cancel_cannot_be_sent() {
let (_tx, rx) = oneshot::channel();
let send_cancel = || false;
let outcome = await_with_optional_cancel(
Duration::from_millis(1),
Duration::from_secs(1),
rx,
None,
None,
Some(&send_cancel),
)
.await;
assert!(matches!(outcome, WaitOutcome::TimeoutCleanupFailed));
}
}
#[cfg(test)]
+10 -1
View File
@@ -225,6 +225,10 @@ pub fn run_foreground(cfg: DaemonConfig) -> Result<()> {
.with_context(|| format!("bind IPC endpoint {}", sock_path.display()))?;
let state = Arc::new(DaemonState::new(cfg.clone()));
state
.transfers
.initialize()
.context("initialize transfer staging")?;
let session_idle_task = spawn_session_idle_reaper(Arc::clone(&state));
let browser_liveness_task = spawn_browser_liveness_reaper(Arc::clone(&state));
// Fired by the update check task after a successful auto-update:
@@ -448,6 +452,7 @@ pub(crate) fn spawn_browser_liveness_reaper(
for s in state.sessions.purge_browser(&client.id) {
state.tool_queues.remove(&s.id);
state.session_interrupts.drop_session(&s.id);
state.transfers.release_session(&s.id.0);
debug!(session = %s.id, "purged session on browser liveness timeout");
}
}
@@ -485,7 +490,10 @@ pub(crate) fn spawn_session_idle_reaper(state: Arc<DaemonState>) -> tokio::task:
)
.await
{
Ok(_) => info!(session = %session_id, "idle session stopped"),
Ok(_) => {
state.transfers.release_session(&session_id.0);
info!(session = %session_id, "idle session stopped");
}
Err(StopSessionError::SessionBusy | StopSessionError::Stopping) => {
debug!(session = %session_id, "idle session still active; retrying later");
}
@@ -496,6 +504,7 @@ pub(crate) fn spawn_session_idle_reaper(state: Arc<DaemonState>) -> tokio::task:
&state.session_interrupts,
&session_id,
);
state.transfers.release_session(&session_id.0);
}
Err(err) => {
warn!(session = %session_id, error = %err, "failed to stop idle session");
+6
View File
@@ -7,6 +7,7 @@ use tokio::task::JoinHandle;
use super::abort::AbortRegistry;
use super::browsers::BrowserRegistry;
use super::file_transfer::TransferRegistry;
use super::inflight::ToolInflightRegistry;
use super::ipc::IpcHandle;
use super::queue::ToolQueueRegistry;
@@ -51,6 +52,9 @@ pub struct DaemonState {
/// `SessionRegistry` because the signal is a transient runtime
/// control state.
pub session_interrupts: Arc<SessionInterruptRegistry>,
/// Operation-scoped local file staging. The extension only sees paths
/// minted here; agent-facing RPCs use opaque transfer ids.
pub transfers: Arc<TransferRegistry>,
}
impl DaemonState {
@@ -64,6 +68,7 @@ impl DaemonState {
));
let abort_registry = Arc::new(AbortRegistry::new());
let session_interrupts = Arc::new(SessionInterruptRegistry::new());
let transfers = Arc::new(TransferRegistry::new().expect("initialise transfer staging"));
Self {
config,
browsers,
@@ -72,6 +77,7 @@ impl DaemonState {
abort_registry,
tool_inflight,
session_interrupts,
transfers,
}
}
}
+3
View File
@@ -295,6 +295,7 @@ async fn drive_connection(
for session in state.sessions.purge_browser(&browser_id) {
state.tool_queues.remove(&session.id);
state.session_interrupts.drop_session(&session.id);
state.transfers.release_session(&session.id.0);
debug!(session = %session.id, "purged stale session before browser reconnect");
}
let (tx, mut rx) = mpsc::unbounded_channel::<Frame>();
@@ -407,6 +408,7 @@ async fn drive_connection(
for s in state.sessions.purge_browser(&browser_id) {
state.tool_queues.remove(&s.id);
state.session_interrupts.drop_session(&s.id);
state.transfers.release_session(&s.id.0);
debug!(session = %s.id, "purged session on browser disconnect");
}
} else {
@@ -503,6 +505,7 @@ fn handle_session_window_closed(
&state.session_interrupts,
&session_id,
) {
state.transfers.release_session(&session_id.0);
info!(session = %session_id, "session removed: user closed Agent Window");
} else {
debug!(session = %session_id, "session.window_closed for unknown session id");
+2
View File
@@ -97,6 +97,8 @@ fn dispatch(cli: Cli, format: Format) -> Result<(), CliError> {
Command::Fill(args) => cli::interaction::dispatch_fill(args, format),
Command::Press(args) => cli::interaction::dispatch_press(args, format),
Command::Select(args) => cli::interaction::dispatch_select(args, format),
Command::Upload(args) => cli::upload::dispatch(args, format),
Command::Download(args) => cli::download::dispatch(args, format),
Command::Evaluate(args) => cli::evaluate::dispatch(args, format),
Command::WaitForNavigation(args) => cli::waits::dispatch_wait_for_navigation(args, format),
Command::WaitMs(args) => cli::waits::dispatch_wait_ms(args, format),
+88
View File
@@ -6,6 +6,7 @@ use bsk::cli::daemon::{DaemonCmd, parse_duration};
use bsk::cli::navigate::NavigateCmd;
use bsk::cli::record::{RecordCmd, RecordSub};
use bsk::cli::session::{SessionCmd, SessionSub};
use bsk::cli::upload::UploadModeArg;
use bsk::{Cli, Command};
use clap::Parser;
@@ -13,6 +14,54 @@ fn parse(args: &[&str]) -> Cli {
Cli::try_parse_from(args).expect("clap parse should succeed")
}
#[test]
fn parses_upload_modes() {
let cli = parse(&[
"bsk",
"upload",
"@e1",
"--file",
"image.png",
"--session",
"s1",
]);
let Command::Upload(default_args) = cli.command else {
panic!("expected upload subcommand");
};
assert_eq!(default_args.mode, UploadModeArg::Input);
let cli = parse(&[
"bsk",
"upload",
"@e1",
"--file",
"image.png",
"--session",
"s1",
"--mode",
"drop",
]);
let Command::Upload(drop_args) = cli.command else {
panic!("expected upload subcommand");
};
assert_eq!(drop_args.mode, UploadModeArg::Drop);
assert!(
Cli::try_parse_from([
"bsk",
"upload",
"@e1",
"--file",
"image.png",
"--session",
"s1",
"--mode",
"auto",
])
.is_err()
);
}
#[test]
fn parses_daemon_start_with_defaults() {
let cli = parse(&["bsk", "daemon", "start"]);
@@ -213,6 +262,45 @@ fn rejects_zero_click_count() {
);
}
#[test]
fn parses_upload_with_repeated_files() {
let cli = parse(&[
"bsk",
"upload",
"@e3",
"--file",
"one.png",
"--file",
"two.png",
"--session",
"s1",
]);
let Command::Upload(args) = cli.command else {
panic!("expected upload command");
};
assert_eq!(args.target.as_deref(), Some("@e3"));
assert_eq!(args.files.len(), 2);
}
#[test]
fn parses_download_with_exact_output_policy() {
let cli = parse(&[
"bsk",
"download",
"#export",
"--out",
"result.zip",
"--session",
"s1",
"--overwrite",
]);
let Command::Download(args) = cli.command else {
panic!("expected download command");
};
assert_eq!(args.target.as_deref(), Some("#export"));
assert!(args.overwrite);
}
#[test]
fn parses_record_start_with_browser_and_url() {
let cli = parse(&[
+2
View File
@@ -478,6 +478,7 @@ async fn observe_returns_semantic_text_and_ref_count() {
tab_id: 13,
truncated: false,
dialogs: vec![],
hover_probe: None,
debug: None,
})
.unwrap(),
@@ -494,6 +495,7 @@ async fn observe_returns_semantic_text_and_ref_count() {
max_depth: None,
max_tokens: None,
debug_surfaces: false,
probe_hover: false,
},
)
.await
@@ -0,0 +1,56 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "DownloadParams",
"type": "object",
"required": [
"session_id"
],
"properties": {
"browser_relative_dir": {
"description": "Daemon-injected relative directory beneath Chrome's Downloads root.",
"type": [
"string",
"null"
]
},
"max_byte_size": {
"description": "Daemon-injected authoritative transfer size limit. The extension uses it only for early cancellation; daemon import remains the final check.",
"type": [
"integer",
"null"
],
"format": "uint64",
"minimum": 0.0
},
"ref": {
"type": [
"string",
"null"
]
},
"selector": {
"type": [
"string",
"null"
]
},
"session_id": {
"type": "string"
},
"tab_id": {
"type": [
"integer",
"null"
],
"format": "int64"
},
"timeout_ms": {
"type": [
"integer",
"null"
],
"format": "uint32",
"minimum": 0.0
}
}
}
@@ -0,0 +1,62 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "DownloadResult",
"type": "object",
"required": [
"byte_size",
"suggested_filename",
"tab_id"
],
"properties": {
"browser_path": {
"description": "Extension-internal completed Chrome download path, stripped by the daemon.",
"type": [
"string",
"null"
]
},
"byte_size": {
"type": "integer",
"format": "uint64",
"minimum": 0.0
},
"danger": {
"type": [
"string",
"null"
]
},
"mime": {
"type": [
"string",
"null"
]
},
"suggested_filename": {
"type": "string"
},
"tab_id": {
"type": "integer",
"format": "int64"
},
"transfer_id": {
"description": "Opaque id returned by the daemon to the CLI.",
"type": [
"string",
"null"
]
},
"used_ref": {
"type": [
"string",
"null"
]
},
"used_selector": {
"type": [
"string",
"null"
]
}
}
}
@@ -1,7 +1,7 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "ObserveParams",
"description": "Parameters for `tool.observe` — produce a semantic VOM observation for one tab plus a fresh `@e<N>` ref-store on the extension side.\n\nUnlike `tool.snapshot`, this path may run bounded perception probes such as hover-surface discovery. It must not submit input, click, navigate, or otherwise commit page state.",
"description": "Parameters for `tool.observe` — produce a semantic VOM observation for one tab plus a fresh `@e<N>` ref-store on the extension side.\n\nUnlike `tool.snapshot`, this path may run bounded perception probes such as hover-surface discovery, but only when `probe_hover` asks for them. It must not submit input, click, navigate, or otherwise commit page state.",
"type": "object",
"required": [
"session_id"
@@ -30,6 +30,11 @@
"format": "uint32",
"minimum": 0.0
},
"probe_hover": {
"description": "Opt in to active hover probing: the extension moves the real cursor over a bounded set of controls to discover hover-only menus and tooltips.\n\nOff by default. Probing costs seconds of wall clock, mutates the live page, and pays off only on pages whose content is genuinely hidden behind hover. Request it when a static observation looks like it is missing hover-revealed structure.",
"default": false,
"type": "boolean"
},
"session_id": {
"type": "string"
},
@@ -24,6 +24,17 @@
"$ref": "#/definitions/JavaScriptDialogInfo"
}
},
"hover_probe": {
"description": "Present only when `probe_hover` was requested and probing actually ran.",
"anyOf": [
{
"$ref": "#/definitions/HoverProbeReport"
},
{
"type": "null"
}
]
},
"ref_count": {
"description": "Number of `@e<N>` refs registered for this session by this observation. Equivalent to the size of the new ref-store map.",
"type": "integer",
@@ -46,6 +57,24 @@
}
},
"definitions": {
"HoverProbeReport": {
"description": "Reports what active hover probing did during an observation, so the agent can judge how far the live page may have drifted from the returned text.",
"type": "object",
"required": [
"performed",
"revealed_content"
],
"properties": {
"performed": {
"description": "The real cursor was moved over page content during this observation.",
"type": "boolean"
},
"revealed_content": {
"description": "Hovering surfaced content the static tree does not contain, so the page reacts to the cursor and may not have fully reverted.",
"type": "boolean"
}
}
},
"JavaScriptDialogHandledAction": {
"description": "How the extension resolved the dialog so CDP could continue.",
"type": "string",
@@ -0,0 +1,80 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "UploadParams",
"type": "object",
"required": [
"files",
"session_id"
],
"properties": {
"files": {
"type": "array",
"items": {
"$ref": "#/definitions/UploadFile"
}
},
"mode": {
"$ref": "#/definitions/UploadMode"
},
"ref": {
"type": [
"string",
"null"
]
},
"selector": {
"type": [
"string",
"null"
]
},
"session_id": {
"type": "string"
},
"tab_id": {
"type": [
"integer",
"null"
],
"format": "int64"
},
"timeout_ms": {
"type": [
"integer",
"null"
],
"format": "uint32",
"minimum": 0.0
}
},
"definitions": {
"UploadFile": {
"type": "object",
"required": [
"name",
"transfer_id"
],
"properties": {
"name": {
"type": "string"
},
"staged_path": {
"type": [
"string",
"null"
]
},
"transfer_id": {
"type": "string"
}
}
},
"UploadMode": {
"type": "string",
"enum": [
"input",
"drop"
]
}
}
}
@@ -0,0 +1,33 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "UploadResult",
"type": "object",
"required": [
"file_names",
"tab_id"
],
"properties": {
"file_names": {
"type": "array",
"items": {
"type": "string"
}
},
"tab_id": {
"type": "integer",
"format": "int64"
},
"used_ref": {
"type": [
"string",
"null"
]
},
"used_selector": {
"type": [
"string",
"null"
]
}
}
}
@@ -85,6 +85,10 @@ fn main() {
dump!(PressResult, "tool_press_result");
dump!(SelectParams, "tool_select_params");
dump!(SelectResult, "tool_select_result");
dump!(UploadParams, "tool_upload_params");
dump!(UploadResult, "tool_upload_result");
dump!(DownloadParams, "tool_download_params");
dump!(DownloadResult, "tool_download_result");
dump!(SnapshotParams, "tool_snapshot_params");
dump!(SnapshotResult, "tool_snapshot_result");
+22
View File
@@ -78,6 +78,10 @@ pub enum Method {
ToolPress,
#[serde(rename = "tool.select")]
ToolSelect,
#[serde(rename = "tool.upload")]
ToolUpload,
#[serde(rename = "tool.download")]
ToolDownload,
#[serde(rename = "tool.snapshot")]
ToolSnapshot,
#[serde(rename = "tool.observe")]
@@ -105,6 +109,17 @@ pub enum Method {
#[serde(rename = "tool.record_await")]
ToolRecordAwait,
#[serde(rename = "transfer.begin")]
TransferBegin,
#[serde(rename = "transfer.chunk")]
TransferChunk,
#[serde(rename = "transfer.finish")]
TransferFinish,
#[serde(rename = "transfer.read")]
TransferRead,
#[serde(rename = "transfer.release")]
TransferRelease,
#[serde(rename = "cancel")]
Cancel,
}
@@ -156,6 +171,8 @@ impl Method {
| Method::ToolFill
| Method::ToolPress
| Method::ToolSelect
| Method::ToolUpload
| Method::ToolDownload
| Method::ToolEvaluate
// May navigate via optional `url` and changes Agent Window
// chrome; gate behind pending-interrupt like other writes.
@@ -194,6 +211,11 @@ impl Method {
| Method::SystemPing
| Method::SystemStatus
| Method::BrowserList
| Method::TransferBegin
| Method::TransferChunk
| Method::TransferFinish
| Method::TransferRead
| Method::TransferRelease
| Method::Cancel => MethodEffect::ControlPlane,
}
}

Some files were not shown because too many files have changed in this diff Show More